Safety nets & runbook

The stock / reordering engine writes movements that commit real money: a drifting ledger produces under-ordered supplier orders, stockouts, or silent over-ordering. The platform therefore stacks three defense layers (audited invariants, pinned tests, production replay-diff) plus a daily health probe. This page is the operational reference: what each layer guarantees, the exact commands to run, and decision trees for the three most frequent incidents.

The three layers at a glance

The layers complement each other: layer 1 catches broken data (whatever code broke it), layer 2 catches code that regresses (before it touches data), layer 3 catches behavior changes that neither an invariant nor a unit test spells out — "this supplier order would have 12 more lines on my branch".

Layer 1 — Invariants: stock-ledger-audit

The task codifies the invariants the ledger MUST satisfy. It was born from the May 2026 "Bio Rennes" investigation (phantom stock on untracked products, supplier orders silently under-ordered, the reconcile cron failing mutely for weeks) and has grown with every incident since.

The three modes

# Report (read-only, always exits 0)
cd apps/app && bun run cli --task stock-ledger-audit \
  --tenantId <tenantId> --siteId <siteId>

# CI / monitoring gate — non-zero exit on any violation
cd apps/app && bun run cli --task stock-ledger-audit \
  --tenantId <tenantId> --siteId <siteId> --strict

# Heal — writes ONLY to rows identified as broken
cd apps/app && bun run cli --task stock-ledger-audit \
  --tenantId <tenantId> --siteId <siteId> --apply

The invariants

Four subtleties that prevent false alarms:

• I3 never touches drafts. PENDING projections belonging to DRAFT orders are the merchant's review queue (the cron stages drafts, the merchant reviews before sending). They are reported as info, never auto-cancelled.
• I5 under backorder. Since the STOCK_LEDGER_ALLOW_BACKORDER cutover (2026-07-09), a negative Σ is legitimate (backorder); I5 compares the clamped mirror, I5b pins the raw cache.
• I6 is gated by the cutover. Historical debt (pre-cutover clamps) is reported info-only; only clamps after STOCK_LEDGER_BACKORDER_CUTOVER fail --strict — any new clamp is a code regression and must be treated as one.
• I7 is pure information — never gated, never healed. Variation stock sits outside the ledger model: the trust boundary and the supplier-order math read the product only, I5/I5b deliberately scope to productVariationId: null, and no cron writes variation rows. I7 therefore just measures how much drift is hiding at that level (number of variations affected + total |drift|). It never fails --strict, and --apply never repairs it: snapping a variation counter onto a Σ that nothing maintains would be the wrong fix.

Layer 2 — Pinned regression tests

Every ledger bug found in production becomes a pinned test case: the test replays the incident's real data (anonymized via a snapshot) and fails if the code ever drifts back to the faulty behavior. The four pillars:

# The whole stock suite (fast, < 5 s)
cd packages/tests && npm test -- run tests/unit/stock/

# The trust boundary alone
cd packages/tests && npm test -- run tests/unit/supplier-order/stock-ledger-trust-boundary.test.ts

bun packages/scripts/src/dev/snapshot-tenant-stock.ts \
  --tenant-id <tenantId> --site-id <siteId> [--vendor-id <uuid>] \
  --days 30 --out packages/tests/fixtures/stock-ledger/<name>.json

Layer 3 — Production replay-diff (read-only)

replay-supplier-orders.ts runs the production paths computeExpectedOrders() and getCurrentOrders() with the current branch's code against the live database, then prints per-vendor drift: new / modified / orphaned orders, items added / changed / removed. Zero writes — safe against production at any time.

bun packages/scripts/src/dev/replay-supplier-orders.ts \
  --tenant-id <tenantId> --site-id <siteId> \
  [--vendor-id <uuid>] \
  [--from 2026-07-01] [--to 2026-07-31]

Run it before merging any change in the reordering blast radius:

• packages/services-supplier-order/src/utils/auto-supplier-order-utils-new.ts
• packages/services-supplier-order/src/utils/supplier-order-utils-new.ts (the trustedStock boundary)
• packages/services-supplier-order/src/utils/supplier-order-reconciliation.ts
• packages/services-supplier-order/src/utils/auto-supplier-orders-regeneration.ts
• packages/services/src/stock/stock-actions.service.ts
• any code that touches Product.stock or the StockAction lifecycle

Interpretation is simple: if your branch does not change business logic, the diff must be identical to master's at the same instant. An unexpected delta (e.g. "−33 lines for this vendor") is exactly the regression class no unit test spelled out — and the 07:15 reconcile would have silently applied it the morning after the merge.

The daily health probe (06:30)

The supplier-stock-health task runs every morning at 06:30 (drop-in /etc/cron.d/supplier-stock-health, read-only) — right before the 07:15 reconcile, to photograph the state before the day's corrective pass. Its wrapper is the reference pattern: it runs the task file directly and ends with exit $EXIT_CODE — failure genuinely reaches alerting.

What it evaluates (overall verdict ✅ HEALTHY / 🛑 ANOMALY, persisted to KV for the control tower):

cd apps/app && bun run cli --task supplier-stock-health \
  --tenantId <tenantId> --siteId <siteId> --json

autoValidateSealed24h — the forgotten-seals counter

Since #2886, the primary decrement gesture is the merchant's seal at the physical event (delivery click in the UI, or "Validate today's route"); the auto-validate cron (12:01 + a 2-hourly sweep) is only a safety net that seals what was forgotten — see write paths. autoValidateSealed24h counts the deliveries the net had to seal in the last 24 h. An occasional non-zero is normal; a persistent non-zero means operators are not sealing in the UI — the ledger stays correct, but with some latency: once the AUTO_VALIDATE_GRACE_HOURS=12 grace clears, the 2-hourly sweep seals the miss within ~2 h, and because the ~06:01 pass runs before the 07:15 reconcile, the morning reorder pass normally does see the prior day's deliveries.

Runbook

Three decision trees for the recurring incidents. In every case: diagnose read-only first, heal second, pin a test at the end if a code path was at fault.

"A product's stock looks wrong"

flowchart TD
  A["Displayed stock looks suspicious"] --> B{"manageStock ?"}
  B -->|no| C["Convention: untracked product ⇒ 0<br/>trustedStock returns 0, nothing to heal"]
  B -->|yes| D["Open the product timeline<br/>/manage/ecommerce/inventory/stock-ledger"]
  D --> E{"Negative ledger?"}
  E -->|yes| F["Real backorder — the next<br/>reception nets it: normal"]
  E -->|no| G["stock-ledger-audit in report mode"]
  G --> H{"I5 / I5b drifting?"}
  H -->|yes| I["--apply — prod (backorder ON): resync stock mirror, no SA<br/>legacy (backorder OFF): Backfill SA + snaps stockLedger"]
  H -->|no| J{"Faulty movement visible<br/>in the timeline?"}
  J -->|yes| K["Write path bypassing the ledger:<br/>rule-17 violation → fix code + pin a test"]
  J -->|no| L["Physical count<br/>/manage/ecommerce/inventory/counts"]

Step by step:

1. Timeline first. /manage/ecommerce/inventory/stock-ledger → product → timeline. The divergent movement is almost always visible by eye (a delta with no matching reception, an unexpected ADJUSTMENT, a duplicated reception).
2. Audit in report mode (read-only) to qualify: I5 drift (counter ≠ Σ) or I5b (cache only)?
3. --apply only rewrites broken rows. In prod (backorder ON), I5 resyncs the Product.stock mirror only — no SA (snapping the stockLedger cache is I5b's job); in legacy (backorder OFF), it emits a "Backfill" reconciliation SA. The stock becomes consistent with the history — it does not become physically correct by magic: if the physical doubt remains, run a count.
4. If a code path bypassed the ledger (writing Product.stock without a StockAction in the same transaction), that is a violation of the golden rule: fix the code and add a case to stock-write-math.test.ts or to the trust boundary.

"A supplier order is missing lines"

Model reminder: a "missing" line means the engine believes the demand is already covered — either by (over-)credited stock, or because the product is outside the scope of the cron that generated the order. The pessimistic kill switch produces the opposite effect (over-ordering): it is never the cause of missing lines.

flowchart TD
  A["Missing lines in a supplier order"] --> B{"Product manageStock=false ?"}
  B -->|yes| C["Only the 07:15 reconcile fills<br/>those products — check its last run"]
  B -->|no| D{"Does credited stock cover the demand?"}
  D -->|stock high| E{"Phantom stock?"}
  E -->|yes| F["STOCK_TRUST_MAX_COVER_WEEKS = 8 ceiling<br/>+ /manage/ecommerce/vendors/orders/phantom-stock page"]
  E -->|no| G["Audit I5 / I5b: inflated ledger?<br/>→ --apply then recalc"]
  D -->|stock plausible| H["replay-supplier-orders<br/>read-only diff on the vendor"]
  H --> I{"Does the diff show added items?"}
  I -->|yes| J["The engine sees them: the 07:15 reconcile<br/>will add them — or trigger a recalc"]
  I -->|no| K["Demand genuinely covered —<br/>a 100% quantity-0 batch is benign"]

Step by step:

1. manageStock=false? The auto-supplier cron (every 30 min) excludes those products; only the daily 07:15 reconcile (--apply --removeOrphans) fills them. If the order was inspected before 07:15, or the reconcile failed, the lines simply don't exist yet.
2. Phantom stock? The trust boundary caps trusted stock at 8 weeks of recent consumption (STOCK_TRUST_MAX_COVER_WEEKS=8) — but below that ceiling an inflated ledger still zeroes lines. The /manage/ecommerce/vendors/orders/phantom-stock page lists the suspects (weeks-of-cover per product).
3. Targeted replay-diff (--vendor-id): if the lines show up as added in the expected state, the engine agrees with you — it's a timing issue (the next reconcile/recalc will materialize them). If the diff is empty, the demand is genuinely covered.
4. A batch that is 100% quantity 0 is not an error — it's covered demand, materialized for information.

"A cron is failing silently"

The structural trap — now closed, but worth knowing: bun run cli --task X exits 0 unless the task throws, so a task that returns {success: false} is swallowed; and a shell wrapper without set -e or a final exit $EXIT_CODE turns even a genuine non-zero exit into 0. Historical result: the scheduler never alerted, and crons could "run" (log line present) while failing for weeks.

✅ Both halves are fixed: the three gate tasks (auto-validate, auto-supplier-orders-service, supplier-order-email) now throw — on whole-run failure and whenever per-item failures remain (after the safe work) — and all 24 wrappers propagate their exit code. The contract is pinned by cron-wrappers.shape.test.ts, which fails any wrapper that swallows its exit code. The triage below still applies to failures that don't travel through an exit code (cron uninstalled, stuck lock, box down).

flowchart TD
  A["Suspicion: a cron stopped producing"] --> B["Cron-5 chip on the stock-ledger page<br/>(heartbeat) + 06:30 probe in KV"]
  B --> C{"Chip state?"}
  C -->|green ≤ 5h| D["The cron is running — look elsewhere<br/>(check 'last activity')"]
  C -->|amber ≤ 12h| E["Several 2-hourly sweeps missed"]
  C -->|red > 12h| F["The cron has not run:<br/>likely down"]
  E --> G["tail the logs ~/.ploi/scheduled-*.log"]
  F --> G
  G --> H{"Error in the log<br/>but exit code 0?"}
  H -->|yes| I["Exit-code masking (regression):<br/>wrapper without exit $EXIT_CODE,<br/>or task returning instead of throwing"]
  H -->|no| J["Stuck lock? fuser -v on the lockfile<br/>+ replay in dry-run<br/>+ check the /etc/crontab entry"]
  I --> K["Correct pattern: wrapper exit $EXIT_CODE<br/>+ gate tasks that THROW"]

Step by step:

1. Chips first. The /manage/ecommerce/inventory/stock-ledger page carries a health chip for the auto-validate cron. Its headline is liveness ("did the cron run?"), read from the heartbeat written on every run: ≤ 5 h green, ≤ 12 h amber, > 12 h red. The age of the last seal (reception / delivery) is demoted to secondary info ("last activity"): a quiet day produces no row, and that is not a failure. If the chip is red while the box is up, suspect an orphaned lock (a stuck bun holding the flock makes every subsequent sweep skip): fuser -v /tmp/auto-validate-stock-actions.<TENANT_ID>.lock.
2. Logs. Every wrapper redirects to ~/.ploi/scheduled-*.log on the tenant box:

ls -lt ~/.ploi/scheduled-*.log | head
tail -100 ~/.ploi/scheduled-<id>.log

3. Visible error + exit 0 = masking. Check the wrapper (scripts/cron/*.sh): it must end with exit $EXIT_CODE (model: supplier-stock-health.sh), and any task used as a gate must throw on failure (model: reconcile-supplier-orders.ts, which throws on any per-row Prisma error instead of swallowing it).
4. Replay by hand in dry-run to reproduce — every cron has a documented smoke-test invocation in the daily timeline.

The commandments (rule 17, condensed)

The NEVER/ALWAYS list distilled from real incidents — every line cost at least one production incident before being written down.

Never:

• Write Product.stock without (a) checking manageStock and (b) emitting the matching StockAction in the same transaction — otherwise stockLedger decouples (I5/I5b).
• Reintroduce a stock decrement at order sync. That path was deleted on 2026-06-14: stock decrements on delivery only (UI seal, auto-validate safety net). A sync-time decrement would double-count with no shared idempotency signal.
• Create a PENDING StockAction for a manageStock=false product — it pollutes the ledger (the origin of the Bio Rennes case).
• Transition an SA from PENDING/COMPLETED to CANCELLED without reversing the corresponding Product.stock movement for tracked products.
• Stub quantityBefore=0 when creating an SA — read the real Product.stock first.
• Swallow per-item Prisma errors in a cron task — throw, and let monitoring catch it (that is the whole point of decision tree 3 above).

Always:

• Apply the trust boundary manageStock ? max(0, stock) : 0 (plus the consumption ceiling) when reading stock for reordering math.
• Add a pinned case to stock-ledger-trust-boundary.test.ts (or to the affected write path's test) for every newly discovered edge case.
• Run the replay-diff before merging a change in the reordering blast radius.
• Register any new ledger task entry point in the CLI smoke test (packages/tests/tests/unit/cli/cli-smoke.test.ts).

See also

• Overview — the full stock → reordering → email model
• The stock ledger — signed authority, clamped mirror, backorder
• Write paths — recordPhysicalMovement and sealing at the physical event
• Supplier orders — event-driven recalc, trust boundary, gates
• Daily timeline — the crons, their schedules, their smoke tests
• PrestaShop sync — ledger-integrated sync writes
• Back office — timeline, counts, movements journal, phantom-stock page
• Configuration — all the flags (STOCK_LEDGER_*, SUPPLIER_*, STOCK_TRUST_*)