A bounded payment-recovery agent that cannot take an unsafe money action.
An LLM proposes payment-recovery actions. A deterministic policy gate — ordinary Python reading a YAML file, no model call — is the only path to moving money. Every decision, including every refusal, is logged and replayable offline.
All three strategies have the RETRY action, run against the same gate over the same 150 held-out orders. This is the comparison that is not confounded by differences in action space, so it comes first.
RETRY only; SWITCH_RAIL excluded. The denominator is retries that actually executed, which matches the attempts column below.
Read B0 carefully before drawing a conclusion. Blind retry scores far better here than a strategy with no logic deserves, and the reason is the gate: B0 proposed 214 retries and the gate denied 98 of them. What survived to execute was the remainder the invariants could not object to. The gate is doing performance work in this table, not only safety work.
Including SWITCH_RAIL, which neither baseline ever proposes.
The like-for-like retry advantage accounts for more than the entire gain. The agent-only action accounts for roughly a quarter. On nudges, which both strategies have, B1 recovers more value than the agent does.
Graded by worst plausible consequence to the customer, after ToolEmu (Ruan et al., ICLR 2024). A single flat count would price a double charge and a late-evening text identically. The “must be zero” claim covers the catastrophic and severe tiers, pinned in code so it cannot drift.
Four things these numbers do not show. They are findings, not caveats added after the fact.
The recovery lift is conditional on a simulated outcome model. Whether a retry succeeds is decided by curves written in config/taxonomy.yaml — a sigmoid with a peak probability of 0.55 at a 48-hour midpoint, nudge recovery probabilities between 0.25 and 0.45 — not fitted to production traffic. Note where the advantage lands: the agent's retries that follow no prior nudge succeed at 55%, and the peak probability in that file is 0.55. A substantial part of what this measures is whether an LLM can find the peak of a curve we specified. Real curves vary by issuer, rail, amount band and time of day, and are precisely what Razorpay already optimises with far more data. This is not a production lift estimate.
The held-out run used the twelve-invariant gate, before the provenance rule landed. The thirteenth only downgrades actions, so applying it would move these numbers rather than leave them unchanged, and re-running would mean inspecting the held-out set a second time. The provenance rule is evidenced on the adversarial suite instead.
Diagnosis accuracy is degenerate on this corpus. The gateway-reported reason code equals the true reason for 150 of 150 held-out orders, so the answer is visible in the input and B1 scores 100% by copying a field. The agent's 99.3% therefore means it overrode a correct signal once. The metric measures corruption of a good signal, not diagnostic skill; the generator was deliberately left frozen rather than fixed after results were known.
The adversarial trap rates are in-sample. Two invariants — hard_decline_no_retry and the extended risk_block — were derived from running the same suite against the deterministic baselines, so the gate is partly fitted to those scenarios. Replicates measure model variance rather than coverage: 100 decisions is ten scenario families seen ten times, not a hundred distinct traps.
A system that logs only what it did is not auditable. This is the complete decision record from the run: the context the gate saw, what the agent proposed and why, every invariant evaluated, and the disposition. Nothing here is recomputed — it is read from the committed audit trail.
Select a decision to read its full record.