Deployment

StayFn ships as one Linux container image (ghcr.io/<owner>/stayfn:<version>, built by the release workflow from src/StayFn.Host/Dockerfile) plus PostgreSQL 16. Two supported shapes:

1. Single machine: docker-compose.prod.yml (this page, first half). Proven on a clean directory.
2. Kubernetes: the sample manifests in k8s/ (second half), validated by kubeconform in CI.

The image runs as the non-root app user (UID 1654) on a read-only root file system; it listens on port 8080 and serves /health/live, /health/ready, /metrics, the APIs, the dashboard at /ui and MCP at /mcp.

Single machine with docker compose

A clean machine needs Docker with Compose v2.24 or later, and three files: docker-compose.prod.yml, docker/postgres/init-prod.sh (keep the relative path) and an env file.

cp deploy/prod.env.example prod.env        # then replace every REPLACE_ME
openssl rand -base64 48 | tr -d '\n/+='    # one per secret
docker compose -f docker-compose.prod.yml --env-file prod.env up -d
curl -fsS http://127.0.0.1:8080/health/ready

On the first start PostgreSQL runs init-prod.sh, which creates the two non-superuser roles with your passwords (stayfn_migrator owns the schema, stayfn_app has DML only, neither bypasses row-level security). The host then applies the migrations as the migrator and serves traffic as the app role. The database port is never published; the host port binds to 127.0.0.1 unless you change it.

If the first start fails (for example a password shorter than 16 characters), PostgreSQL has already initialised the data volume and will not run init-prod.sh again: fix prod.env, then docker compose -f docker-compose.prod.yml --env-file prod.env down -v and start again. The host logs three ASP.NET Data Protection warnings (ephemeral key ring — StayFn protects no data with it) and, with an HS256 signing key, one warning recommending an OIDC issuer; both are expected.

Variables

Every variable docker-compose.prod.yml reads. Required ones stop docker compose with a message when they are missing; the host and init-prod.sh also refuse the REPLACE_ME placeholders and values that are too short.

Variable	Required	Meaning
STAYFN_IMAGE	yes	Image to run, e.g. ghcr.io/<owner>/stayfn:0.8.1 (or a locally built tag).
POSTGRES_ADMIN_PASSWORD	yes	Password of the postgres superuser inside the database container (bootstrap only; the host never uses it).
STAYFN_APP_DB_PASSWORD	yes	Password of stayfn_app, the role the host serves traffic with (at least 16 characters).
STAYFN_MIGRATOR_DB_PASSWORD	yes	Password of stayfn_migrator, the schema owner that runs migrations (at least 16 characters).
STAYFN_AUTH_ISSUER	yes	iss of admin JWTs.
STAYFN_AUTH_AUDIENCE	yes	aud of admin JWTs.
STAYFN_AUTH_SIGNING_KEY	yes	HS256 key for admin JWTs, at least 32 bytes. Development values (dev-only-…) are refused outside Development. With STAYFN_AUTH_JWKS_URL set it is ignored for validation but must still be a random value.
STAYFN_AUTH_JWKS_URL	no	OIDC metadata / JWKS URL (https) for RS256 admin tokens from your identity provider (preferred in production).
STAYFN_API_KEY_PEPPER	yes	Pepper of tenant API key digests, at least 16 bytes. Changing it invalidates every tenant API key (see “Rotating the pepper”).
STAYFN_OIDC_AUTHORITY	no	Dashboard OIDC issuer (authorization code + PKCE, public client). Empty: the dashboard offers paste-a-token login.
STAYFN_OIDC_CLIENT_ID	no	Dashboard OIDC public client id.
STAYFN_SECRETS_PROVIDER	no	Default provider for secret references without a prefix: env (default), azure-keyvault, aws-secrets-manager, hashicorp-vault.
STAYFN_SECRETS_ENV_FILE	no	Env file whose OHIP_* / STAYFN_SECRET_* variables back env: references (default ./stayfn-secrets.env, optional).
STAYFN_SECRETS_AWS_REGION	no	AWS region for aws: references; credentials come from the AWS default chain.
STAYFN_SECRETS_VAULT_ADDRESS	no	HashiCorp Vault address for vault: references.
STAYFN_SECRETS_VAULT_TOKEN	no	Vault token (single machine; Kubernetes uses the Kubernetes auth method instead).
OTEL_EXPORTER_OTLP_ENDPOINT	no	OTLP collector for traces and metrics.
STAYFN_RATE_LIMIT_CAPACITY	no	Default OHIP token-bucket burst (40; per app key).
STAYFN_RATE_LIMIT_REFILL_PER_SECOND	no	Default OHIP token-bucket rate (40/s).
STAYFN_BIND	no	Interface the host port binds to (127.0.0.1 default; 0.0.0.0 to publish).
STAYFN_PORT	no	Host port (8080).

Any other setting of the host can be added to the environment: block as STAYFN__<Section>__<Key> (for example STAYFN__Retention__Invocations=90.00:00:00, see docs/runbooks/backup-restore.md).

Secrets for OHIP environments

Tenants’ OHIP environments store secret references, never values. Pick one store:

• env: references read host environment variables, only those named OHIP_* or STAYFN_SECRET_* (never STAYFN__*). Put them in the file named by STAYFN_SECRETS_ENV_FILE.
• aws:<secret id or ARN>[#<json key>] reads AWS Secrets Manager (AWSCURRENT).
• vault:<path>[#<field>] reads HashiCorp Vault KV v2 under Secrets:Vault:MountPath (default secret).
• kv:<name>[@<version>] reads Azure Key Vault (STAYFN__Secrets__AzureKeyVault__VaultUri).

Remote values are cached for Secrets:CacheTtl (5 min) and re-read every Secrets:RotationCheckInterval (5 min). A new provider version is a rotation: the OHIP tokens that used the reference are dropped and an audit row secret.rotated is written. After rotating a credential by hand, POST /api/admin/environments/{id}/rotate-credentials (operator) applies it on every replica at once (runbook: token failures).

Streaming (OHIP business events)

The streaming ingest (EventIngestWorker) is off by default (Events:Streaming:Enabled=false in every appsettings file). Oracle allows one consumer per chain and app key: enable it on exactly one deployment per app key (replicas of that deployment share it through a database lease); a second consumer is refused with close code 4409 and retried after Events:Streaming:ConflictBackoff (60 s).

Setting	Default	Meaning
STAYFN__Events__Streaming__Enabled	false	Run the ingest.
STAYFN__Events__Streaming__WebSocketUrl	empty	A URL or an env: reference to an OHIP_*/STAYFN_SECRET_* variable; empty (or an unset variable) means wss://<gateway host>/subscriptions. The client always adds ?key=<sha256 hex of the app key>, which the gateway requires.
STAYFN__Events__Streaming__StartFromLatest / InitialOffset	false / empty	Where a stream without checkpoint starts (offsetType: highest, or an offset; empty = the server default).
STAYFN__Events__Streaming__PingInterval	00:00:10	Client ping (Oracle: at least every 15 s; the gateway drops a socket idle for ~60 s).
STAYFN__Events__Streaming__ResubscribeGap	00:00:10	Minimum wait between complete and the next subscribe (Oracle limit).
STAYFN__Events__Streaming__MaxSessionDuration	01:00:00	The session is completed and reopened (from the checkpoint) this often and before the token goes stale.

Sandbox dev loop. With the sandbox variables sourced (set -a; source .env; set +a) and the development tenant seeded (stayfn dev seed --env sandbox --tenant "Sandbox (dev)"), run the host with the sandbox-streaming launch profile:

dotnet run --project src/StayFn.Host --launch-profile sandbox-streaming

It sets ASPNETCORE_ENVIRONMENT=Development, STAYFN__Events__Streaming__Enabled=true and STAYFN__Events__Streaming__WebSocketUrl=env:OHIP_SANDBOX_WEBSOCKET_URL (the first, default http profile is neutral — it sets nothing, so a plain dotnet run behaves as before — and the tests keep streaming off). Stop any other consumer of the same app key first (another host, stayfn ohip stream-probe). To check the connection without the host, run stayfn ohip stream-probe --env sandbox --offset-type highest --seconds 60 (structure only, never values; docs/cli.md).

TLS, headers and limits

Terminate TLS in a reverse proxy (or ingress). Outside Development the host sends Strict-Transport-Security, and always a CSP suited to the dashboard, X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer and frame-ancestors 'none'. Request bodies are capped at 1 MB (functions, webhook, admin) and at Kestrel:Limits:MaxRequestBodySize (1 MB) globally.

Capacity and load shedding

One replica served ≈ 1,000 req/s of a one-OHIP-call function on a 4-vCPU VM (docs/limits.md); the OHIP per-app-key limit (≈ 50 req/s) is reached long before. Beyond Security:MaxConcurrentRequests (64) in flight and Security:MaxQueuedRequests (128) queued the host answers 503 urn:stayfn:problem:overloaded with Retry-After: 1; probes, /metrics and the long-lived streams are exempt. docker-compose.prod.yml raises nofile to 65536 — keep that if you run the image elsewhere.

Upgrades

Pull the new image tag into STAYFN_IMAGE and run docker compose … up -d. The host migrates on start (one replica); take a backup first (docs/runbooks/backup-restore.md).

Rotating the pepper

Auth:ApiKeyPepper is mixed into every stored API key digest, so a new pepper makes every existing key unknown (401). Rotation is therefore a planned re-issue: create new keys for each tenant (POST /api/admin/api-keys), hand them out, change the pepper, restart, revoke the old rows. Dual-pepper verification was considered and not built.

Kubernetes

k8s/ holds a Kustomize base: ServiceAccount, ConfigMap, the migration Job, Deployment (2 replicas, non-root, read-only root, probes), Service, HPA (2–6 on CPU), PodDisruptionBudget (min 1) and a NetworkPolicy (ingress 8080; egress DNS, 5432, 443, OTLP). Validate with tools/ci/validate-k8s.sh (kubeconform).

1. Create the namespace and the Secret the manifests reference (keys below). Use your secret operator (External Secrets, Vault Agent, Sealed Secrets) rather than a literal manifest:

   kubectl create namespace stayfn
   kubectl -n stayfn create secret generic stayfn-secrets \
     --from-literal=postgres-app-connection='Host=<pg>;Database=stayfn;Username=stayfn_app;Password=<…>;SSL Mode=Require' \
     --from-literal=postgres-migrator-connection='Host=<pg>;Database=stayfn;Username=stayfn_migrator;Password=<…>;SSL Mode=Require' \
     --from-literal=auth-issuer='https://<issuer>' --from-literal=auth-audience='stayfn-admin' \
     --from-literal=auth-jwks-url='https://<issuer>/.well-known/openid-configuration' \
     --from-literal=api-key-pepper='<random>'

   The database needs the two roles of docker/postgres/init-prod.sh (run its SQL once as an administrator of a managed PostgreSQL).

2. Pin the image: cd k8s && kustomize edit set image ghcr.io/OWNER/stayfn=ghcr.io/<owner>/stayfn:<version>.

3. kubectl apply -k k8s/. The Job stayfn-migrate runs --migrate-only as the migrator; the Deployment’s pods report not ready (schema check) until it finished, then join the Service. For the next release delete the finished Job (or use a Helm/Argo hook) and apply again.

Readiness (/health/ready) gates on postgres, schema (no pending migration), stream (always healthy while streaming is disabled — Phase 3b — and unhealthy only when an enabled ingest loop stopped) and timezones. Liveness (/health/live) has no dependencies.

AWS (EKS): set STAYFN__Secrets__Provider=aws-secrets-manager and …__Aws__Region in the ConfigMap and annotate the ServiceAccount for IRSA; the image carries the web-identity credential support. Vault: set …__Vault__Address and …__Vault__KubernetesRole, bind the ServiceAccount in Vault, and set automountServiceAccountToken: true on the pod template (it is off by default).

Every replica runs all workers; outbox leases, schedule claims and stream leases make that safe. Rate limits are per OHIP app key, so more replicas do not raise upstream throughput (docs/limits.md).