Runbooks

Operational procedures for a deployed StayFn. Each has the same shape — Symptoms, Diagnose, Remediate, Drill — and was exercised once against the compose stack with WireMock standing in for OHIP.

Runbook	When
Token failures	OHIP calls fail with 401/403, OhipAuthenticationException, after a credential rotation
Rate-limit exhaustion	429 rate_limited, invocations waiting on the governor, OHIP answering 429
Dead-letter storm	dead letters piling up (upstream outage, a bad deploy, poison events)
Stream stalled	streaming checkpoints stop moving, inbox or outbox lag grows
Backup and restore	backups, restore drills, retention windows

Common tools

BASE=https://stayfn.example.invalid          # or http://127.0.0.1:8080 on the compose host
TOKEN=<operator or owner JWT>                # from your identity provider; on a drill stack with an HS256 key:
#   ASPNETCORE_ENVIRONMENT=Development STAYFN__Auth__SigningKey=… STAYFN__Auth__Issuer=… STAYFN__Auth__Audience=… stayfn dev token --role owner
curl -fsS "$BASE/health/ready" | jq .                                         # readiness: postgres, schema, stream, timezones
curl -fsS "$BASE/metrics" | grep '^stayfn_'                                   # Prometheus metrics (dots become underscores)
curl -fsS -H "Authorization: Bearer $TOKEN" "$BASE/api/admin/overview?tenantId=<id>&minutes=60" | jq .
docker compose -f docker-compose.prod.yml logs host --since 15m | grep '"@l":"\(Warning\|Error\)"'

A system-scope owner passes ?tenantId=<id> on every admin call; operators and viewers are bound to their tenant. Logs are JSON (CLEF) with PII masked; search them by CorrelationId, InvocationId, Function or EventName.