Beat the Reorder Rule First: Agents That Run Shops Must Clear a Rule-Based Baseline

When Agents Move from Buying for Shoppers to Running the Shop

Consider a retail team handing daily operation of unmanned kiosks to a language-model agent: choosing assortment, setting prices, ordering stock, and sending paid staff to restock shelves. In a short demo, the agent writes plausible rationales and stays profitable. Over half a year, however, does an inference-heavy agent build more net value than a textbook inventory rule?

Our thesis: in long-horizon retail operations, staying profitable is not beating a rule baseline, and better individual metrics do not add up to more net assets^[1]. Before an agent may place orders or send staff, it must beat a simple rule baseline over a long horizon on net value after inference cost, check every order against product-level available stock, and pass human approval above set thresholds.

Long-Horizon Benchmarks, In-Transit Illusions, and the Classic Inventory Trade-Off

Arrow, Harris, and Marschak (1951) formulated inventory decisions under uncertain demand: the optimal stock level balances the expected cost of running short against the cost of holding excess stock under lead times and reorder levels^[3]. That critical-ratio trade-off is the root of the rule baseline used to judge store agents^{[1, 3]}.

In a long-horizon vending benchmark, Shi, Li, Luan, Sun, Ma, Li, Jiang, and Bai (2026) had one agent operate 20 vending machines across five site types (6 worksite, 2 campus, 4 residential, 6 commercial, 2 office) sharing one warehouse over 180 simulated days via a web interface with 18 tools, starting with 40,000 yuan and 1,372 products^[1]. Grounded in real vending sales records (220,128 units sold) and validated against observed sales patterns with demand parameters hidden, the simulator charged orders at placement with a 3-day warehouse lead time; on-site restocking or removal cost 35 yuan/machine/day; batches expired 30 or 90 days after arrival with a 1,000 yuan fine per expired unit sold; warehouse rent was 0.03 yuan/unit/day, machine rent 4 yuan/machine/day, and capacity 200 units/machine^[1]. The primary outcome was terminal net assets (cash plus unexpired inventory at cost)^[1]. The deterministic baseline paired a critical-ratio target from Arrow et al. (1951) with weekly periodic order-up-to replenishment from rolling 30-day demand and variability—accounting for lead times, stock, and outstanding orders—attaining 208.73 ± 2.26 thousand yuan across three seeds (coefficient of variation 1.1%)^{[1, 3]}.

Five agents tested across three runs each in the same frozen world at highest reasoning effort with identical prompts, tools, and daily budgets showed four results^[1]:

• Four of five agents finished below the rule: All 15 runs ended above starting capital, yet four of five models trailed the rule on mean terminal net assets^[1]. Only one model exceeded it in all three runs (mean 309.35 thousand yuan); none of the other 12 runs did, the lowest averaged 91.52 thousand yuan, and run-to-run coefficient of variation hit 45.6% for one model versus ~11% for the others^[1].
• Inference spend reshuffled rankings: Returns per dollar of estimated API cost ranked models differently from returns alone; the top-return agent spent ~80× as much in estimated API cost as the most cost-efficient agent while growing net assets ~2.9× as much^[1].
• Partial metrics did not add up to net assets: No agent led on all five dimensions (profitability, assortment, inventory/waste, pricing, capital efficiency), and all five scored below the baseline on assortment^[1]. Low waste coexisted with empty shelves on 25.21% of site-days for one agent, while another kept empty shelves to 1.67% with the highest waste rate (1.85%)^[1]. Comparing days 16–45 with days 136–165, daily asset growth fell for all five model means and in 13 of 15 runs; waste or pricing improved for some agents without stronger overall growth (a comparison that cannot causally identify whether feedback improves decisions)^[1].
• Illustrative failures: Illustrative cases (not prevalence) show how errors compound: all 49 allocations in one day’s restocking batch were rejected because product-level warehouse stock was insufficient after the agent looked at aggregate units while an order was in transit; an agent ordered 80 more units of a product with no recorded sales in that interval despite 2,158 unexpired units in the warehouse; a 35-yuan visit added 4 units to a machine that had earned 5.14 yuan gross profit since the prior visit; and the same four infeasible allocations rejected on day 163 reappeared on day 168—corrected within the interaction, not retained as policy^[1]. Limits: simulation rather than a real store, three runs per model, one environment^[1].

An earlier single-machine benchmark by Backlund and Petersson (2025)—where an agent ordered by email, set prices, and paid a daily fee—found high variance across runs for all models, with even top models derailing via misread delivery schedules, forgotten orders, or tangential loops^[2]. Most commonly, an agent assumed an order had arrived on the expected date while still in transit, then pursued the "problem" instead of waiting or re-checking inventory^[2]. Context fullness did not explain failures (correlation 0.167 between days until sales stop and days until full memory), and a single human baseline limits comparison^[2].

Engineering Mapping on rlab.works: Qualification Baselines, Stock Checks, and Cost Ledgers

In our engineering view, teams should: (1) qualify ordering agents against a critical-ratio / order-up-to baseline over a long simulated horizon with several runs on net value after inference cost, reporting spread across runs; (2) check every allocation and order against product-level available stock and in-transit quantity before commit, not aggregate numbers; (3) require human approval for replenishment and visits above thresholds; (4) keep the rule as fallback and default when agent proposals deviate strongly; (5) persist corrections as written policy rechecked in weekly reviews rather than relying on agent memory; and (6) track inference spend in the same ledger as operating costs.

Across Commerce, AI Gateway, and Intelligence on rlab.works, three proposed skill packs and three working components map these safeguards:

• Fulfillment and replenishment gating: In Commerce, Prepare and deliver orders (S35) runs Align orders and items → Check supply and reserve → Approve replenishment or splits (human gate) → Pick and schedule delivery → Reconcile receipts and exceptions (human gate), pairing Inventory ledger, batch control, stocktake and valuation (reservations, movement ledgers, batch expiry freeze/inspection, stocktake differences, valuation) with Procurement, replenishment suggestions and supplier collaboration (requisitions/POs with approval, arrivals, replenishment from sales, stock, open orders, and lead-time policies). Proven for retail (transferable for logistics), it requires linking inventory, purchasing, outbound, and delivery IDs; routes are advisory and refunds need separate approval. In our engineering view, Check supply and reserve checks product-level available and in-transit stock, while Approve replenishment or splits gates large orders and uneconomical visits.
• Inference spend accounting: In AI Gateway, Check usage and costs (S13) runs Check available budget → Submit and collect results → Account for request costs → Review discrepancies (human gate) with Usage reservations, budgets, versioned pricing and settlement (reserving/settling usage, enforcing budgets, reconciling priced/unpriced ledger evidence). Transferable for finance and public internal tool cost management with no retail tag, it is presented as transferable to track the agent’s inference spend under one billing owner.
• Weekly review and policy updates: In Intelligence, Review weekly operations (S26) runs Collect and align tables → Compare periods and segments → Find exception candidates → Check causes and evidence (human gate) → Assemble report and follow-ups. Proven only for travel with hotel-specific components, retail use is transferable: hotel components need replacing, and holiday overlap is not a cause. In our engineering view, Check causes and evidence turns recurring rejected allocations into written policy.

All packs are proposed and not integrated, demonstrated with synthetic data. All three components have working maturity (the first two with retail tags), but none runs an agent-versus-rule comparison or a long-horizon simulation—that requirement comes from the papers.

Three Checks Before Letting an Agent Order Stock or Dispatch Staff

• Beat a rule baseline after inference spend: Run multi-run long-horizon simulations against a critical-ratio / order-up-to rule including inference costs, keeping the rule as default until cleared^{[1, 3]}.
• Check product-level stock and visit thresholds before commit: Verify product-level available and in-transit stock at Check supply and reserve and gate replenishment or visits above thresholds at Approve replenishment or splits^{[1, 2]}.
• Persist fixes as policy in weekly reviews: Audit rejected allocations, expiry waste, and empty-shelf days at Check causes and evidence so fixes become written policy rather than relying on agent memory^{[1, 2]}.