Runbook: dead-letter storm

A dead letter is a terminal failure kept for replay (dead_letters, kinds invocation, outbox, inbox). Input errors (4xx-class) are never dead-lettered; everything that is can be replayed once, atomically, from the admin API or the dashboard.

Symptoms

• deadLetters.pending in GET /api/admin/overview climbing; the dashboard’s Dead letters page filling up.
• Invocations ending dead after their retry policy (stayfn.events.failed{handler} rising for handlers).
• Usually one cause: an OHIP outage (5xx), a bad deploy of a function, a malformed stream of events (kind inbox), or a missing variable.

Diagnose

1. Group them: GET /api/admin/dead-letters?tenantId=<id>&replayed=false&page=1&pageSize=200 → kind, reason, refId, createdAt. One reason shared by all letters means one cause.
2. For kind invocation: GET /api/admin/invocations/{refId} → error code and the OHIP call timeline (5xx from OHIP, timeouts, or no call at all).
3. For kind inbox/outbox: GET /api/admin/events?tenantId=<id>&status=dead and GET /api/admin/events/{id} (masked payload, handler invocations).
4. Check the upstream: the OHIP status page, stayfn.ohip_calls{status} and stayfn.ohip.call_ms in the metrics, WireMock/sandbox reachability.
5. Before replaying: make sure the cause is gone — a replay of a still-failing call just dead-letters again.

Remediate

1. Fix the cause (wait for OHIP, roll back the function, correct the variable, fix the event source).
2. Replay: one letter POST /api/admin/dead-letters/{id}/replay (operator) — or stayfn replay --dead-letter <id> — creates a new invocation (correlation replay:<letter id>) through the outbox; a letter replays once (409 already_replayed).
3. Many letters: loop over the list from step 1 of Diagnose, oldest first, a few per second (the replays share the OHIP budget).
4. Letters that should not be replayed simply age out (Retention:DeadLetters, 90 days by default; backup-restore.md).

Drill

WireMock answered the reservation search with 503; three invocations died after their in-process retries and left three dead letters; after the mapping reset they were listed, one was inspected and all were replayed successfully.