Runbook: token failures

OHIP access tokens come from POST {gateway}/oauth/v1/tokens, are cached per (tenant, environment) and refreshed before they expire; the credentials are secret references resolved through the secret provider.

Symptoms

• Invocations fail with 502 upstream_error, upstreamStatus 401 (or 400) and the detail “Token request to ‘<environment>’ returned 401 (invalid_client: …)”; the invocation ends failed with error code invalid_client and is dead-lettered (an authentication failure is permanent).
• Logs: Token request to … failed, Invalidated OHIP token for environment …, repeated Obtained OHIP token (a refresh loop).
• SecretNotFoundException (“Secret reference ‘…’ was not found by the ‘<provider>’ secret provider”) — the reference no longer resolves.
• Right after rotating an app key / client secret in the secret store.

Diagnose

1. Which environment and since when: GET /api/admin/invocations?tenantId=<id>&status=failed → errorCode/errorMessage of the newest rows; open one with GET /api/admin/invocations/{id}: the OHIP call it attempted is recorded with status 401 and a few milliseconds, and the error text names the token request (not the operation).
2. Logs of the host: grep -E 'Token request|SecretNotFound|Invalidated OHIP token' (the failed invocation’s Warning line carries the token error). The environment is named, the secret value never is.
3. Is it the credential or the reference?
   • SecretNotFoundException → the reference is wrong or the provider cannot read it (AWS: role/IRSA permissions; Vault: policy, token or Kubernetes role; env: the variable is missing or not under an allowed prefix OHIP_/STAYFN_SECRET_).
   • 401/400 from the token endpoint → the value is wrong or was rotated at Oracle but not in the store (or the reverse).
4. GET /api/admin/environments/{id} shows which references the environment uses (never values) and its OAuth flow.
5. secret.rotated rows in GET /api/admin/audit?tenantId=<id>&action=secret.rotated tell whether StayFn already saw a new version.

Remediate

1. Put the correct value in the secret store under the same reference (or PUT /api/admin/environments/{id} to point at a new reference).
2. Apply it on every replica now instead of waiting for Secrets:CacheTtl / Secrets:RotationCheckInterval (5 min each): POST /api/admin/environments/{id}/rotate-credentials (operator). It drops the cached values and the environment’s tokens on this replica, relays the same to the others through pg_notify('secrets_rotated'), and writes environment.rotate.
3. Re-run a cheap call (stayfn ohip ping --env <name> --hotel <code> from an operator machine, or one function invocation) and confirm a new Obtained OHIP token log line and a successful OHIP call.
4. Invocations that died meanwhile are dead letters — replay them once the token works (dead-letter-storm.md).

Drill

WireMock’s token endpoint was switched to 401 invalid_client, a function was invoked (502, token failure in the invocation), the mapping was restored, rotate-credentials was called and the next invocation succeeded.