Deployment — from a bare VM to a running stack

A from-scratch deploy of §9.2’s shape: Docker Compose on one VM, Caddy terminating TLS, Postgres as the only coordination mechanism. Each step proves something works before the next builds on it, the same convention getting-started.adoc uses — if a step fails, stop there rather than pushing through.

This is the automatable half. It does not decide where that VM is — #3 (cross-border data transfer under Law No. 2024/017) blocks that decision, not this runbook.

Known seams — read this before you start

All three gaps this section used to name are closed, and this file was left saying otherwise for longer than it should have — the same "documentation asserts something the code does not do" shape AGENTS.md keeps rediscovering, found here while writing the backend-only section below:

• sms-gateway register-client exists (backends/apps/sms-gateway/src/main.rs, Command::RegisterClient) and registers an integrator’s own Ed25519 public key as an AppClient — step 5 below uses it for the console composer’s key. The gateway never generates, sees or stores a private key; there is no token to mint unless the deployment serves machine_tokens (provision-client was deleted in VS-4 and is back behind op:, VSA-3a).

• sms-gateway create-app exists (Command::CreateApp, production-readiness audit N1) and closes what used to be a real chicken-and-egg gap: creating the very first App a real production deployment sends through — App.create's own @@allow is hasRole('owner') || hasRole('admin'), and nothing this deployment can mint over HTTP carries either role until a human has paired a console (VS-5b), which itself needs sms-gateway bootstrap/provision-user to have already run. create-app writes through the same hand-built owner-role context every other bootstrap subcommand in this file uses — see step 5’s own note on the composer’s key. vsms-demo-seed (ghcr.io/vaam-apps/vsms/demo, a separate binary/image — see that crate’s own module doc) is explicitly not a substitute for this in a real deployment: it’s demo-only, seeds fixture data no production App should carry, and isn’t referenced anywhere below.

• backends/apps/sms-fake-orange exists as a real binary/image, and compose.demo.yaml (repo root, not this file’s own deploy/docker-compose.yml) wires it in for a GHCR-only showcase — see that file’s own header for how it differs from this runbook’s production-shaped stack. deploy/docker-compose.yml itself still has no fake-orange profile; every step below still assumes real Orange Cameroon client_credentials, except the backend-only verification section, which points the orange_cm entry’s host.url (config/application-prod.yml) at a manually-run sms-fake-orange container instead — see that section for exactly how.

Prerequisites

• A VM with Docker Engine and the Compose plugin (docker compose version should print v2.20+ — service_completed_successfully depends-on conditions, used below, need it).

• Two DNS records (A/AAAA) pointed at the VM — one for the gateway’s public origin, one for the admin console. A real domain, not just an IP: Caddy’s automatic HTTPS needs one. (Testing without public DNS: see deploy/Caddyfile's own comment for the tls internal swap — everything else below still applies.)

• Real Orange Cameroon client_credentials and a contracted sender ID. Placeholder credentials get you through migrations and a running stack, but every submit attempt fails at Orange’s own OAuth endpoint — the same boundary 36-handset-gate.adoc documents.

• openssl on your workstation, for generating a Postgres password and the SMS_HASH_PEPPER step 1 below needs, and for the Ed25519 keys (the envelope keys in step 1, the composer’s key in step 5): openssl genpkey -algorithm ed25519.

1. Clone and configure

git clone https://github.com/vaam-apps/vsms.git
cd vsms/deploy
cp .env.example .env
chmod 600 .env

# Production-readiness audit B2: deploy/docker-compose.yml's `backup`
# service mounts this as a `configs:` reference, which fails `docker
# compose up` outright ("bind source path does not exist") if the file
# is missing — required even when BACKUP_RCLONE_REMOTE ends up being a
# bare local path with no real credentials to fill in. See the "Secrets"
# section below for the full reasoning.
cp secrets/rclone.conf.example secrets/rclone.conf
chmod 600 secrets/rclone.conf

# VS-3: the signed transport's two server keys. `sms-gateway serve` refuses
# to start without both (and refuses the same key twice). Never reuse a dev
# key here; the next key is the one a later rotation promotes.
openssl genpkey -algorithm ed25519 -out secrets/envelope-current.pem
openssl genpkey -algorithm ed25519 -out secrets/envelope-next.pem
sudo chown 65532:65532 secrets/envelope-current.pem secrets/envelope-next.pem
sudo chmod 0400 secrets/envelope-current.pem secrets/envelope-next.pem

Fill in .env — every var traces to a real #[arg(long, env = ...)] in backends/apps/sms-gateway/src/main.rs / backends/apps/sms-worker/src/main.rs, to a ${NAME} placeholder in config/application*.yml (next section), or to a docker-compose.yml service block; see that file’s own comments for which. At minimum:

# a real, random Postgres password
openssl rand -base64 24
# a real, random SMS_HASH_PEPPER (minimum 32 bytes) — #134/#140
openssl rand -base64 48

secrets/envelope-current.pem signs every response and secrets/envelope-next.pem is the key the next rotation promotes. An integrator pins the public half of both; print them once the stack is up with docker compose run --rm sms-gateway server-keys (it needs no database) and hand the two lines to the integrator. They are paths, not secret values, in .env-land: the files are the secret.

The worker’s hooks role seals every sealed webhook endpoint’s delivery with secrets/envelope-current.pem too, so a receiver pins the same two public keys for the gateway’s answers and for the deliveries it receives. An endpoint’s delivery mode is its own (docs/integrating.md, "Choosing a delivery mode"): an hmac endpoint is signed with its own secret and needs no key, so a deployment that has not moved a receiver to the seal may leave the file out. A hooks worker without it still starts and still delivers every hmac endpoint, and claims no sealed endpoint’s attempt at all: those stay pending (or failed, resting out a backoff), untouched and without spending an attempt of maxAttempts, for a worker that holds the key, which delivers them as soon as it runs, so a late key loses nothing. Because the rows just wait, the keyless worker says so: every five minutes it logs a warn with the number of due sealed attempts it cannot serve, naming --envelope-signing-key-file, and it logs one at start. Treat either line as an unmounted key. The number counts the attempts of active endpoints only: a deactivated endpoint’s attempts are claimed by no worker, with the key or without it, so they say nothing about the key and a deployment whose only sealed endpoint is switched off does not warn for ever. An endpoint whose circuit is open is still counted: that is a pause which ends by itself (circuitOpenUntil, 15 minutes), after which its attempts are exactly what a keyless worker cannot take. Both containers run as uid 65532, which is why the one chown/chmod 0400 above serves both.

Rotating the envelope keys

Both processes read their key once, at start-up, so a rotation is a restart:

1. Check that every client and every webhook receiver already pins today’s next public key: it is the key being promoted, and an integrator pins both halves precisely so this step is a check and not a flag day (docker compose run --rm sms-gateway server-keys prints them). Anyone who does not yet has it must pin it before you go on, because the restart below makes it the key that signs.

2. Promote it: move secrets/envelope-next.pem over secrets/envelope-current.pem, mint a new envelope-next.pem, keep the chown/chmod above.

3. Restart the gateway and the worker together (docker compose up -d --force-recreate sms-gateway sms-worker). A worker left running keeps sealing deliveries with the old key; receivers that still pin it accept them, but the moment the old key is retired from their pins every delivery from that worker is refused. On Kubernetes the same step is kubectl rollout restart of both deployments: the chart mounts the worker’s key with subPath, and a subPath Secret mount never sees an edited Secret, so the worker will not pick the new key up on its own the way a directory mount eventually would.

4. Hand every client and every webhook receiver the newly minted next public key (server-keys after the restart prints it), so that they pin it before the next rotation promotes it.

5. Retire the old public key from the receivers' pins once no delivery sealed with it can still be retried (a retry is sealed afresh, so after the restart nothing is).

SMS_HASH_PEPPER backs Message.msisdnHash/bodyHash (HMAC-SHA256) — sms-gateway serve validates it (minimum 32 bytes) before doing anything else, so a missing or too-short value fails the container at boot, not at the first sendMessage call. Generate it once and keep it: rotating later does not retroactively rehash rows already written under the old value — see backends/crates/sms-api/src/pepper.rs's own module doc.

The YAML configuration (config/application-prod.yml)

Everything a deployment decides — the pepper’s source, the pool sizes, the idempotency and rate-limit policy, and every SMS provider with its host, settings and credentials — is one YAML document read by both sms-gateway and sms-worker (VSC-1; docs/architecture.md, section 9.4 has every key). The images bake the base file, config/application.yml, and this stack mounts config/application-prod.yml from your clone over the prod overlay the image bakes, so you edit it in the repository you cloned and a docker compose up -d restarts the services on it, with no rebuild. .env keeps only the secrets and per-host facts that file reads as ${NAME}.

The shipped overlay runs orange_cm with livemode: true, which turns on the rules a real deployment needs: every URL https, no stub host, and every secret a ${VAR} placeholder rather than a literal. To change the provider set:

• MTN or AWS instead of (or beside) Orange: uncomment that provider’s block in config/application-prod.yml, uncomment the matching pass-through lines under sms-gateway and sms-worker in docker-compose.yml, and set the variables in .env. For an MTN-only or AWS-only deployment, delete the Orange entry and leave its three variables unset: an entry whose variables are unset is a boot failure naming the variable, not a disabled provider.

• Anything else the file reads: every ${NAME} you add needs a line in both services' environment: (a bare NAME: forwards it when .env sets it). The process refuses to start on a ${NAME} it cannot resolve, naming it.

The process checks the document before it opens a key file or a connection, so a mistake in it stops sms-gateway and sms-worker at start, naming the key, before anything else is touched (step 4’s docker compose ps shows the exit; the reason is in docker compose logs sms-gateway).

SMS_PROFILE names a file. A profile with no application-<profile>.yml beside the base is refused, naming the file it expected (prodd is a typo, not a profile), and only the default, dev, may run without one. A release image run with no SMS_PROFILE therefore loads dev, whose overlay points at the local fakes, and the gateway stops on the first thing in it a release build refuses (fake_sns_cert_base); that message names the profile it loaded and says to set SMS_PROFILE. docker-compose.yml and the chart set it for you.

Important

Upgrading from a release that configured these as flags and variables. The flags are gone and are refused, not ignored. Move each value into config/application-prod.yml (the table in docs/architecture.md, section 9.4, maps every old name to its key), delete the old lines from .env, and expect the new .env to contain only the secrets and the ${NAME}`s the overlay reads. `docker-compose.yml no longer forwards ORANGE_CM_BASE_URL, SMS_IDEMPOTENCY_TTL_SECS, SMS_RATE_LIMIT_*, SMS_SOURCE_RATE_LIMIT_* or the AWS_SMS_* settings, so setting them in .env does nothing; a deployment that still passes one to a container by hand is told, at start, which key replaced it.

Before deploying: hmac endpoints with an empty or short key

An hmac delivery is byte-identical to what it was before the sealed mode existed (owner’s ruling R5: vsms serves apps beyond vaam), including for an endpoint whose secret is empty, which is signed with the empty key and an empty prevSecret a second v1= value. A signature under an empty key is one anyone can compute, so before the deploy count the endpoints that have one, against the database the migration is about to run on (it prints counts and ids, never a secret):

psql "$DATABASE_URL" -c "
  SELECT count(*) FILTER (WHERE secret = '')                       AS empty_secret,
         count(*) FILTER (WHERE length(secret) BETWEEN 1 AND 31)   AS short_secret,
         count(*) FILTER (WHERE prev_secret = '')                  AS empty_prev_secret
    FROM webhook_endpoints"

Add WHERE delivery_mode = 'hmac' once migration 0010_sealed_webhooks has run (before it, every row is an hmac endpoint and the column does not exist; after it, a sealed endpoint holds no secret and so counts for nothing). To see which, select id, app_id with the same predicates. After the deploy the worker names each such endpoint in a warn line (at most every ten minutes per endpoint) that reads "signs with an empty key". The fix is rotateWebhookSecret on each, twice when the secret itself is empty (the first rotation moves the empty secret into prevSecret), and handing the new secret to that endpoint’s receiver; a short, non-empty secret signs correctly and is only weak, so rotate those at leisure.

Narrowing the authentication methods (auth.methods)

vsms serves more than one application, so which ways of authenticating a deployment accepts is configuration (VSA-1, owner ruling R5; docs/architecture.md, section 4.9). The key is in the same document as the rest, in your overlay (config/application-prod.yml, or config.overlay in the chart):

auth:
  methods: [keypair]            # vaam's shape
  # methods: [keypair, hmac_webhooks]   # keep the shared-secret webhooks

Absent means all four, which is what every deployment had before the key existed, so an upgrade changes nothing until you write it. An empty list, a null (methods: with every item commented out), an unknown name, a duplicate, a list with neither keypair nor machine_tokens, machine_tokens without an op: block, human_login without a human_login: block, a human_login: block without op: (even under the absent default) or without the method, and an op: block when neither machine_tokens nor human_login is permitted are each refused at start, naming the rule. The change takes a restart of the gateway and the worker, like any other key in the file.

Before you turn hmac_webhooks off, count what it leaves behind:

psql "$DATABASE_URL" -c "
  SELECT e.id, e.app_id, e.active,
         count(a.*) FILTER (WHERE a.state IN ('pending', 'failed', 'delivering')) AS waiting
    FROM webhook_endpoints e LEFT JOIN webhook_attempts a ON a.endpoint_id = e.id
   WHERE e.delivery_mode = 'hmac'
   GROUP BY e.id ORDER BY waiting DESC"

Each of those endpoints stays, but is no longer sent to: no worker claims its attempts, so they stay pending, untouched, and are not counted against the endpoint’s max_attempts or its circuit breaker. They keep accumulating, one per event, until you act, and the waiting column above is how many there are. Each hooks worker with the method off warns every five minutes with held=<count> (the count stops at 10001 and reads more than 10000 beyond it: the waiting column above is the exact figure). replayWebhookAttempt of one of those endpoints' attempts is refused too (422, auth_method_disabled): a replayed attempt would be reset to pending and held like the rest, so the call would answer 200 and send nothing. A sealed endpoint’s attempts replay as ever. The way out for a receiver is switchWebhookDeliveryMode to sealed (an operator with webhook:manage; the receiver then needs an opener, see docs/integrating.md): the attempts that waited are then delivered sealed, since the mode is read for each attempt when it is delivered. An endpoint nobody will move can be deactivated (its attempts then wait for no worker and are no longer counted in the warning) or deleted as before. To resume the deliveries as they were, permit hmac_webhooks again and restart the workers: the next claim delivers them, with no replay.

Releasing a held backlog. When the method returns, or the endpoints are switched to sealed, the held attempts are delivered oldest first and ahead of the fresh ones, at about 20 a second for each hooks worker (a poll every second claims at most 20), so a deep backlog delays today’s events for as long as it takes to drain. Each is delivered with occurredAt set to the delivery time, because an attempt carries no creation time: a receiver sees an event from last month stamped now. If the oldest are no longer worth sending, delete them first, with the cutoff you choose (the count, then the delete):

psql "$DATABASE_URL" -c "
  SELECT count(*) FROM webhook_attempts a JOIN webhook_endpoints e ON e.id = a.endpoint_id
   WHERE e.delivery_mode = 'hmac' AND a.state IN ('pending', 'failed')
     AND a.next_attempt_at < now() - interval '7 days'"
psql "$DATABASE_URL" -c "
  DELETE FROM webhook_attempts a USING webhook_endpoints e
   WHERE e.id = a.endpoint_id AND e.delivery_mode = 'hmac'
     AND a.state IN ('pending', 'failed')
     AND a.next_attempt_at < now() - interval '7 days'"

Add AND e.id = '<endpoint id>' to release one endpoint’s backlog and keep the others. A deleted attempt is an event that endpoint never receives, and nothing re-creates it.

A hooks worker with the method off and no --envelope-signing-key-file serves nothing, hmac or sealed, and says so at start. It still starts, because the configuration is the whole deployment’s and another worker of it may hold the key; if none does, mount --envelope-signing-key-file on a hooks worker.

At start, sms-gateway serve logs the permitted and served methods and whether the key was written (authentication methods), and the gateway and every hooks worker log how many hmac endpoints exist: an info line normally, a warn when the method is off and endpoints remain. Check the gateway’s answer, which needs no credential:

curl -s https://$SMS_GATEWAY_DOMAIN/.well-known/vsms-auth     # {"methods":["keypair"]}

Machine tokens: the OpenID Provider, a token client, the signing key (VSA-3a)

A deployment that serves other applications than vaam can mint short-lived bearer tokens for a machine client (owner ruling R5; docs/architecture.md, section 4.10). It is off unless you write the op: block, and vaam’s methods: [keypair] keeps it off even then. The API accepts the tokens this provider mints (VSA-3b), as a bearer beside the signed envelope, exactly when machine_tokens is served. A token caller gets plain JSON (no response seal, no contract-lock 426); a deployment whose auth.methods is [machine_tokens] alone refuses a signed request (415) and cannot register a key or pair a console (422 auth_method_disabled) until VSA-5. register-client and pair-console-key apply the same rule when the configuration they are given (--config, which the images set through SMS_CONFIG) omits keypair.

op:
  issuer: https://sms.example.org     # no trailing slash; https under livemode
  token_rate_limit:
    burst: 10
    refill_per_second: 0.3333333333333333
  signing_key:
    refresh_interval_secs: 60
    rotation_overlap_secs: 1800       # must exceed refresh_interval_secs + the 900-second token lifetime

Every key is required and there is no secret in the block. With auth.methods absent the method is permitted; with a list, it must name machine_tokens. Since VSA-3b [machine_tokens] alone is a legal list: the API accepts a token, so the gateway has a way in. It has no signed way in, though: serve still needs the two envelope signing key files (they seal nothing for a token caller; the files are the deployment’s identity, the ones server-keys prints and the worker’s hooks role seals webhook deliveries with, and serve keeps requiring them whatever auth.methods says), a signed request is refused with a 415, and registering a key or pairing a console is a 422 until VSA-5. A list with neither keypair nor machine_tokens is still refused at boot (NoWayIn).

Order matters, once, on a fresh deployment. serve refuses to start while no signing key is active, so mint one first. Run it with run --rm, never exec: waiting for the gateway’s health check before rotating deadlocks.

docker compose --env-file .env run --rm sms-gateway rotate-signing-key

It creates a new RS256 key, makes it the active one, and keeps the previous key publishing in /jwks.json for rotation_overlap_secs, so no token already issued loses its key. A running gateway picks the new key up within refresh_interval_secs, without a restart. Rotate on a schedule you choose and immediately if the database is suspected leaked (the private half is a row).

Provisioning a token client (the private key is shown once, to a file):

umask 077
docker compose --env-file .env run --rm sms-gateway provision-client \
  --app-slug my-app --label "otp sender" --scope sms:send --key-out secrets/otp-sender.pem

A client that reads its own messages back (GET /messages) also needs --scope sms:read; --scope sms:send alone sends and reads nothing.

A client that only needs to know whether its own messages arrived, and must not read them back, takes --scope "sms:send delivery:read": lookupDeliveryStatus (POST /$procs/lookupDeliveryStatus) then answers, for its own app only, by message id, by its own clientRef or by a whole recipient number (60 lookups by number per hour, unpurged messages of the last 90 days only), with the masked recipient, the state, the timestamps and the newest receipt and no message body. Every lookup that reaches the database, a lookup refused for the budget included, is recorded in delivery_lookups; a malformed query (422) or a caller refused for lacking the scope (403) writes no row.

It writes an AppClient and an OauthClient that holds only the public JWK, and the private key to --key-out with mode 0600; it refuses to overwrite an existing file and never prints the key. Hand the file and the printed client_id to the application, which signs an RFC 7523 client assertion with it and posts it to /token (grant_type=client_credentials, client_assertion_type=…jwt-bearer). There is no client secret and no refresh token. console:pair is refused here, as it is everywhere except register-client.

Both commands read the same YAML as serve (--config, --profile), without the pepper, and refuse on a deployment that serves no tokens: auth_method_disabled: this deployment issues no tokens, naming register-client. Nothing is written. A client made with register-client signs its requests instead of asking for a token, and every deployment serves that.

The console sign-in: human_login:, the console client, a password (VSA-4a)

A person with a password can sign in to the console through the same provider (owner ruling R5; docs/architecture.md, section 4.10). It is off unless you write the human_login: block and op:, and vaam’s methods: [keypair] keeps it off even then. The API accepts the access token this sign-in issues (VSA-4b), as that person with the role their row says, with one cap: an owner or an admin is refused with a 403 and the text "this account’s role acts only from a paired console key", until a key is paired to that account (pair-console-key). Every other role acts. The console has no sign-in screen until VSA-5. This also holds on a deployment that serves [keypair, human_login], where before VSA-4b a token could not be used and the API answered only signed requests; the API now also accepts the person’s token there. A deployment on [machine_tokens, human_login] with keypair off cannot pair a console key (registerConsoleKey is refused), so its owner and admin have no way to act through the API at all: use the command line, or serve keypair.

op:                                   # as above: the sign-in rides the provider
  issuer: https://sms.example.org
  # …
human_login:
  console_client:
    client_id: sms-console            # 8 to 64 characters; the only client_id /login accepts
    redirect_uris:                    # exact match; https and no stub host under livemode
      - https://admin.sms.example.org/api/auth/callback
  login_rate_limit:                   # both before any password is hashed
    per_email:                        # one bucket per email address
      burst: 5
      refill_per_second: 0.1
    aggregate:                        # one bucket for every sign-in: the Argon2 CPU budget
      burst: 10
      refill_per_second: 1.0

Every key is required; there is no secret in the block. A human_login: block without op: is refused at start even when auth.methods is absent.

Order matters, once, on a fresh deployment. serve refuses to start while no signing key is active and while the console client row is missing or differs from the block, so, with run --rm and never exec:

docker compose --env-file .env run --rm sms-gateway rotate-signing-key
docker compose --env-file .env run --rm sms-gateway seed-console-client

seed-console-client registers the console as a public client: no key, the authorization_code grant alone, the openid profile scopes, PKCE required, and exactly the block’s redirect URIs. It reconciles: run it again after changing a redirect URI, and it says what was wrong. serve checks the row at start and stops with a message naming every difference and this command, and a row that carries the refresh_token grant a pre-#446 console was registered with is the difference it exists to catch (this deployment issues no refresh token).

Giving a person a password (generated, shown to nobody, written to a file):

umask 077
docker compose --env-file .env run --rm sms-gateway provision-user \
  --email you@example.com --display-name "Your Name" --role-key owner \
  --password --password-out secrets/you.password

The file is created with mode 0600 and never overwritten; the password is not printed. Read it once, hand it over by a channel you trust, and delete the file. An account that exists without a password gets one; an account that already has one is refused unless you pass --reset, which replaces it at once (the old password stops working). bootstrap takes --owner-password-out <file> with --owner-email: it also registers the console client in the same run. All three commands read the same YAML as serve (--config, --profile; provision-user and bootstrap only for the password flags) and refuse on a deployment that does not serve human_login: auth_method_disabled: this deployment signs no person in, before the database is touched, writing nothing and creating no file.

Passwords are Argon2id with no pepper, so a hash exported before 0009 (above) verifies once imported. The sign-in answers every refusal with one 401 invalid_credentials and throttles per email and per source address; the gateway logs human_login is served at start. Authorization codes live in the gateway’s memory for sixty seconds, so run one gateway replica (the chart does).

Single sign-on (VSA-4c) is the same sign-in answered by an identity provider (Keycloak) instead of a password: add a sso: block inside human_login: and the secret as SMS_SSO_CLIENT_SECRET (the chart’s existingSecrets.sso, the gateway only). It is off without the block, the callback to register at the provider is op.issuer plus /login/sso/callback (the gateway logs it at start), and a gateway whose environment still sets one of the old SMS_SSO_* variables without the YAML reading it is refused at boot. The whole of it, Keycloak’s side included, is docs/runbooks/sso.adoc.

Before deploying: the webhook_attempts index of migration 0011

Migration 0011_webhook_endpoint_due_idx builds an index on webhook_attempts with a plain CREATE INDEX: sms-migrate applies each migration in a transaction, where CONCURRENTLY is refused. A plain build holds a SHARE lock on the table, which lets reads through and blocks every insert, update and delete until the build ends. On an upgrade the writers are the previous release’s gateway and workers: the Helm migrate Job is a pre-install,pre-upgrade hook, so they are still running and still delivering when it starts, and each stalls on its write until the build finishes. Nothing purges webhook_attempts yet (the @@retain(days: 30) on WebhookAttempt is not enforced), so the build reads the table’s whole history. Size it first, on the database the migration is about to run on:

psql "$DATABASE_URL" -c "
  SELECT pg_size_pretty(pg_relation_size('webhook_attempts'))        AS heap,
         pg_size_pretty(pg_total_relation_size('webhook_attempts'))  AS with_indexes,
         (SELECT reltuples::bigint FROM pg_class
           WHERE oid = 'webhook_attempts'::regclass)                 AS approx_rows"

The build reads the heap once, so expect it to take about as long as a sequential scan of the heap figure, and writers wait that long. A table of a few tens of megabytes is a stall nobody notices; a table in the gigabytes is not. For one of the latter, build the index by hand, concurrently, before the deploy. It takes no lock that blocks writers, and the migration then finds the index there and runs no CREATE INDEX at all, so it takes no lock on the table either. (It looks the index up by name first; a bare CREATE INDEX IF NOT EXISTS would still queue for the SHARE lock before noticing the index exists, behind every open write, and block the next ones while it waits.)

psql "$DATABASE_URL" -c "CREATE INDEX CONCURRENTLY IF NOT EXISTS webhook_endpoint_due_idx ON webhook_attempts (endpoint_id, next_attempt_at) WHERE state IN ('pending','failed')"

Run it as that one statement (psql -c, or a session with no open transaction): CONCURRENTLY cannot run inside a transaction block. If it is interrupted or fails it leaves an invalid index of the same name behind. Do not deploy over it: the migration refuses (it raises an error naming this section, and sms-migrate rolls it back), because an invalid index serves no query. So check the result before deploying and rebuild if it is not valid:

psql "$DATABASE_URL" -At -c "
  SELECT indisvalid FROM pg_index
   WHERE indexrelid = 'webhook_endpoint_due_idx'::regclass"
# t  -> deploy.   f  -> DROP INDEX CONCURRENTLY webhook_endpoint_due_idx;  and run the build again.

The statement is the migration’s own, with CONCURRENTLY added (backends/crates/sms-api/tests/webhook_endpoint_due_idx_live_postgres.rs runs it and compares the index it builds with the migration’s). The migration also refuses a valid index of that name that has another definition. The alternative, for a large table, is to let the migration build it inside a maintenance window with the gateway and workers stopped. The index is read by a hooks claim whenever it asks for one endpoint’s attempts at a time, which is when some endpoint’s attempts must be left out (deactivated, circuit open, or sealed for a worker without the envelope signing key; backends/crates/sms-worker/src/claim.md). Without it every hooks worker still delivers, but each such claim reads the backlog once per endpoint it asks for, on every poll. Rolling back the migration (down.sql) drops this index and nothing else.

Keeping the OpenID Provider’s rows across 0009 and 0012

Migration 0009_drop_openid_provider (VS-4) dropped five tables, and 0012_restore_openid_provider (VSA-2, owner ruling R5: which ways in a deployment permits is configuration) creates them again, empty. 0009 is in no release tag: the latest, v0.9.0, predates it. So every deployment on v0.9.0 or earlier runs 0009 and 0012 together, in the one migrate Job of its next upgrade, and loses the rows before the new release starts:

Table	What is lost, and what to do
oauth_clients	Every OAuth client registration. Each machine client is provisioned again once the release serves the token endpoint (VSA-3a, Provisioning a token client below), with a new keypair.
oauth_signing_keys	The OP’s signing keys, private halves included. rotate-signing-key mints a new one (VSA-3a); a token the old key signed is not verifiable by it.
user_credentials	Every password hash. Each person is given a new password through provision-user --password --password-out once the release serves the login (VSA-4a, The console sign-in below).
client_assertions, sso_transactions	Spent client assertions and sign-ins in flight. Both are ephemeral: nothing outlasts the minutes it takes to need them, and nothing is worth exporting.

Only the first three are read, and only with op: written. This release has a token endpoint behind auth.methods and op: (VSA-3a, below), a /login behind human_login: (VSA-4a, below) and an SSO broker behind human_login.sso (VSA-4c, below), and no table is reachable by any route, so a deployment that never used them loses nothing it can see. A deployment that is already past 0009 has nothing to keep: 0009 deleted the rows when it ran, and 0012 only ever finds the tables missing.

To keep the first three tables, export them before the upgrade and import them after it. Do it only if you will turn the legacy ways back on; otherwise provisioning anew is the simpler path, and a file of signing keys and password hashes is one more thing to look after.

1. Check that 0009 has not run. If this returns a row, it has, and there is nothing to export:

   psql "$DATABASE_URL" -At -c "SELECT name FROM public.schema_migrations WHERE name = '0009_drop_openid_provider'"

2. Stop writes to the three tables: no client provisioned, no key rotated, no password set. Anything written after the export is not in it. A user or app client deleted after the export makes the import refuse (the foreign keys below), which is the right answer: export again.

3. Export, from a directory only you can read. The files hold the OP’s private keys and Argon2id hashes: they are secrets. \copy runs on the client, so the files land where psql runs. Explicit column lists, so a column added later fails the import instead of shifting the rest:

   umask 077 && mkdir op-export && cd op-export
   psql "$DATABASE_URL" -v ON_ERROR_STOP=1 <<'SQL'
   \copy oauth_signing_keys (created_at, updated_at, id, private_key_pem, active, expires_at) TO 'oauth_signing_keys.csv' WITH (FORMAT csv)
   \copy oauth_clients (created_at, updated_at, id, client_id, app_client_id, token_endpoint_auth_method, jwks, grant_types, scopes, redirect_uris, require_pkce, active) TO 'oauth_clients.csv' WITH (FORMAT csv)
   \copy user_credentials (created_at, updated_at, id, user_id, password_hash) TO 'user_credentials.csv' WITH (FORMAT csv)
   SQL

4. Upgrade. 0009 drops the tables and 0012 creates them again.

5. Import, from the same directory, in one transaction so a refusal leaves the tables empty rather than half full. The order matters only for the foreign keys (oauth_clients.app_client_id names an app_clients row and user_credentials.user_id a users row, both of which 0009 kept):

   psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -1 <<'SQL'
   \copy oauth_signing_keys (created_at, updated_at, id, private_key_pem, active, expires_at) FROM 'oauth_signing_keys.csv' WITH (FORMAT csv)
   \copy oauth_clients (created_at, updated_at, id, client_id, app_client_id, token_endpoint_auth_method, jwks, grant_types, scopes, redirect_uris, require_pkce, active) FROM 'oauth_clients.csv' WITH (FORMAT csv)
   \copy user_credentials (created_at, updated_at, id, user_id, password_hash) FROM 'user_credentials.csv' WITH (FORMAT csv)
   SQL

6. Destroy op-export/ and any copy of it. The files hold private signing keys and Argon2id hashes, so a plain rm is not enough; overwrite them first:

   cd .. && find op-export -type f -exec shred -u {} + && rmdir op-export

   shred is as good as the file system lets it be (a journaling, copy-on-write or SSD-backed volume may keep old blocks), so use the equivalent your platform offers, or keep the directory on an encrypted volume that is itself destroyed. Remember that backups may hold copies: a backup or snapshot of the machine, of the directory’s volume or of the database taken while the files or the imported rows existed carries the same keys and hashes, and is as secret as they are.

After an import, an oauth_clients row blocks a hard delete of its AppClient.oauth_clients.app_client_id is a plain NO ACTION foreign key (so is user_credentials.user_id, for a User), and the imported rows are read only by the token endpoint (VSA-3a, when op: is written) and the login (VSA-4a, when human_login: is). Until then, deleting an AppClient from the console (or a User) fails with a foreign-key error naming oauth_clients_app_client_id_fkey (user_credentials_user_id_fkey). Remove the imported rows that name it first, then delete it again:

DELETE FROM oauth_clients WHERE app_client_id = '<app client id>';
DELETE FROM user_credentials WHERE user_id = '<user id>';

The client or the password that row stood for is gone with it; provision-client and provision-user --password (below) provision a new one.

The statements above are the ones a_provider_export_taken_before_0009_is_restored_after_0012 (backends/crates/sms-api/tests/restore_openid_provider_live_postgres.rs) reads out of this page and runs, against a database that stopped at 0008 with a row in each table; the_runbooks_cleanup_unblocks_a_hard_delete runs the two DELETE`s above against rows that refuse the delete of their AppClient and User. Rolling back `0012 (down.sql, which sms-migrate never runs) drops the five tables and refuses while any holds a row, naming the counts.

2. Build and bring up Postgres + migrations first

docker compose --env-file .env up -d --build postgres migrate
docker compose logs migrate

Expect migrations up to date as the last line. migrate is a one-shot container (see backends/apps/sms-migrate/src/main.rs's own module doc) — it applies backends/migrations/postgres/{0001_init,0002_bootstrap} exactly once each, tracked in a schema_migrations table this deploy path owns, under a Postgres advisory lock. It never regenerates anything; the two up.sql files it runs are embedded verbatim into the binary at compile time (include_str!), from what’s committed. If it exits non-zero, nothing downstream is safe to start — fix this before continuing.

3. Bootstrap: orange_cm Provider/Route, first operator

sms-gateway serve calls resolve_provider_row_id (backends/apps/sms-gateway/src/main.rs) before binding its listener, so a fresh database makes the container exit immediately and crash-loop under restart: unless-stopped if an active Provider row is missing — you cannot docker compose exec into a container that never stays up. (It needs no signing key from the database: since VS-4 the gateway’s only keys are the two envelope keys of step 1.) admin has the equivalent gap one layer up: it starts fine, but nobody can use it without at least one operator account. sms-gateway bootstrap (production-readiness audit N2) closes both in one idempotent call, chaining the two steps this runbook used to walk through one at a time — see "What bootstrap does" below if you need to run, or understand, either of them individually.

R4 (CONTRIBUTING.md: "the admin console is optional, the backend must run without it") — the command below is for a deployment bringing up the console. For a genuinely backend-only deployment, omit --owner-email/--owner-display-name entirely — bootstrap still creates the orange_cm Provider/Route (everything sms-gateway/sms-worker themselves need to start), and explicitly skips the owner-account step rather than either attempting it with nothing to configure it from or silently doing nothing. Passing one of the two without the other is refused with a named error.

docker compose --env-file .env build sms-gateway
docker compose --env-file .env run --rm sms-gateway bootstrap \
  --owner-email you@example.com \
  --owner-display-name "Your Name"

(Not sms-gateway bootstrap as the second argument — the image’s ENTRYPOINT is already the binary; only the subcommand goes after the service name.) run --rm, not exec, for the same reason every one-off command in this runbook uses it: there is no running sms-gateway service container to exec into yet. Safe to re-run against an already-bootstrapped deployment: every step is a clean no-op the second time.

--owner-role-key (default owner) is also available if this deployment needs a different role — see Command::Bootstrap’s own `--help for the full flag list.

What bootstrap does, step by step

Read this if you need one of the two steps on its own, or are debugging a partially-bootstrapped database.

The orange_cm Provider row and a catch-all Route

resolve_provider_row_id in backends/apps/sms-gateway/src/main.rs runs before the listener binds too — the identical crash-loop trap, this time for a missing Provider row. Nothing seeded this row until #148: a fresh database made the container exit immediately and crash-loop under restart: unless-stopped, and this runbook used to work around it with a raw psql INSERT run after bringing the containers up — which meant watching sms-gateway crash-loop, then fixing it by hand. sms-gateway seed-dispatch closes that gap: a real CLI subcommand, going through the CrateStack delegate under a hand-built owner context (never raw SQL — R1), idempotent by construction. On its own:

docker compose --env-file .env run --rm sms-gateway seed-dispatch

Renamed from seed-provider (#62). The Provider row alone used to be enough to make this deployment functional; it no longer is. #62's routing rules engine made sms-worker's dispatch role refuse every message with no matching Route row — a deliberate cutover from routing’s old "any active provider" placeholder — so a deployment with an active Provider and no Route starts and reports healthy, but every message it accepts lands in rejected forever, silently. seed-dispatch now seeds both in one call: the Provider row exactly as seed-provider always did, plus a hardcoded catch-all Route (no flags to configure it — a real routing policy is the admin console’s job, not this bootstrap command’s) pointing at it. Both halves are idempotent — the Provider half by create
catching the 23505 on Provider.key's @unique index; the Route half, which has no unique column to catch a conflict on, by looking up an existing route for the provider first — so re-running the whole command against an already-seeded deployment is a clean no-op on both.

Its own defaults already match the row this runbook used to hand-write — --key orange_cm, --kind orange_cm_http, --credential-ref env:ORANGE_CM_CLIENT_SECRET, --max-tps 10, --max-daily-submissions 100000, --cost-per-segment-xaf 0 — so no flags are required here, and bootstrap itself exposes no flags for this step either (same defaults). config/credential-ref stay placeholders regardless of what’s passed: confirmed against send_test_message.rs's own doc comment, neither binary reads this row’s config/credentialRef to construct the real OrangeCmProvider — both build it from their own CLI flags/env instead (§2.4). This row’s job is only to exist, carry key = 'orange_cm', and end up state = 'active', which the subcommand also guarantees (a fresh row is created disabled — Provider.state's own @default — and activated in a second write).

Create the first operator

The console has no login (VS-5b): nobody types a password into it. A person uses it with a key their own browser made and something vouched for — a vaam admin phone, or, for the first operator and as the break-glass, you on the command line ([pairing-the-console] below). Either way the key is attached to an existing account, so the first operator’s account has to exist:

docker compose --env-file .env run --rm sms-gateway provision-user \
  --email you@example.com --display-name "Your Name" --role-key owner

provision-user prints no credential. A password is optional and explicit (--password --password-out <file>, on a deployment that serves human_login: see The console sign-in above); the console does not ask for one (it has no sign-in screen until VSA-5). What matters here is the account: pair-console-key --user you@example.com attaches the browser’s key to it, and the key speaks as that account’s role. A person paired by a vaam admin phone instead gets an admin account created for them on first pairing, never an owner.

--role-key takes any of §5.2’s built-in roles — owner, admin, operator, developer, auditor, support, all seeded by 0002_bootstrap, and pending, which holds no permission and is seeded by 0013_role_read_scope. system is not among them and cannot be created; roles_key_not_reserved_check rejects it, because a human account holding it would pass every hasRole('system') policy through generated CRUD.

The console authenticates upstream as the paired person’s own key, so a paired owner/admin/operator can perform the writes that need a human role (a Provider edit, for example). One exception remains by design: the composer’s previewMessage/sendMessage calls, which sendMessage structurally requires an app principal for (see backends/crates/sms-api/src/procedures.rs::caller_client_id), go through the composer’s own service key ([the-composers-key] below). Worth knowing: the messages list, its live updates and the dashboard read every app’s rows, not just one’s — any paired human whose role holds sms:read (owner, admin, operator and auditor; not developer, support or pending, since 0013_role_read_scope) can see across App boundaries, per Message's own row policy and the sms:read claim it asks for.

4. Bring up sms-gateway and sms-worker

docker compose --env-file .env up -d sms-gateway sms-worker
docker compose ps

Both wait on migrate finishing successfully (service_completed_successfully, not just "started"). sms-gateway's HEALTHCHECK hits its own GET /healthz (added by this same PR — there was no health endpoint for either binary before it); sms-worker has no HTTP surface, so its HEALTHCHECK instead checks the mtime of a heartbeat file its main.rs touches every 15s (see backends/apps/sms-worker/Dockerfile's own comment for why that’s a more meaningful signal than a bare "is the process running" check). Give both --start-period a few seconds before checking status. sms-gateway should now report (healthy) on the first attempt — both the envelope keys of step 1 and the Provider row from step 3’s bootstrap already exist, so neither startup dependency is missing this time.

5. Provision the admin console’s keys

The console holds two kinds of key, and neither is a login.

The composer’s key

previewMessage and sendMessage are procedures only an app may call, so a person’s browser key cannot, and the console’s composer sends through a service key of its own: an Ed25519 AppClient key, registered once, whose private half never reaches vsms. A real deployment’s first App is a business decision (quota, IP allowlist, a SenderId actually approved by a real provider account) this runbook cannot make for you; sms-gateway create-app --slug <slug> --name "<display name>" writes the row (idempotently, under a hand-built owner context):

docker compose --env-file .env run --rm sms-gateway create-app \
  --slug my-app --name "My App"

monthlyQuota/ipAllowlist/transliterateToGsm7 are fixed placeholders this command doesn’t expose flags for — adjust the row afterward from the console. Then generate the key, hand the console its private half, and register the public half:

umask 077
mkdir -p secrets && chmod 711 secrets
openssl genpkey -algorithm ed25519 -out secrets/console-composer.pem
sudo chown 10001:10001 secrets/console-composer.pem   # admin's runtime uid
sudo chmod 0400 secrets/console-composer.pem

# The public half as register-client takes it: the raw 32 bytes, base64url.
PUBLIC_KEY="$(openssl pkey -in secrets/console-composer.pem -pubout -outform DER \
  | tail -c 32 | base64 | tr '+/' '-_' | tr -d '=\n')"

docker compose --env-file .env run --rm sms-gateway register-client \
  --app my-app --client-name console-composer --label "console composer" \
  --scope sms:send --scope sms:read --public-key "$PUBLIC_KEY"

register-client is idempotent: the key is identified by its RFC 9679 thumbprint, so running it twice leaves one row. --client-name gives the composer its own principal (appc_<slug>_console-composer), so its rate-limit bucket, idempotency namespace and scopes are not shared with any other integration of the same App. Do not grant it console:pair: that scope belongs to vaam’s pairing client alone, and the gateway refuses it anywhere but this command line.

chmod 711 on secrets/ (not 700) matters: a directory with no execute ("search") bit for "other" blocks any other uid from even traversing into it, whatever the file inside allows. admin reads the key through a configs: reference (SMS_COMPOSER_KEY_FILE), which fails docker compose up outright if the file is missing, instead of a bind mount that would be created silently as an empty directory.

Also set SMS_ENVELOPE_SERVER_KEY_CURRENT and SMS_ENVELOPE_SERVER_KEY_NEXT in .env to the two public keys the gateway prints (they are what the browser pins, and what the composer’s client verifies answers with):

docker compose --env-file .env run --rm sms-gateway server-keys

A console deployed without the composer key serves every other screen; only the composer reports that it has no key.

The composer spends this key only for a person the gateway names. Before each send or preview the browser seals a call to authorizeComposition with the person’s own key, and the gateway answers it only for a signed-in owner, admin or operator holding sms:send; the console’s server opens that answer under the two public keys above, and goes on only if it names a person, that operation and the hash of the message about to be sent. An auditor, a support or developer user, a role that lost sms:send, and any app client’s key are refused (a 403 the composer shows), however they were paired.

Pairing the console

A person’s console key is made by their own browser the first time they open the console, and shown as a QR (docs/design/console-pairing.md). It becomes valid when the gateway has a ConsoleKey row for it, and there are two ways to make one:

• A vaam admin phone scans the QR and approves with biometrics, and vaam relays the pairing to registerConsoleKey with its pairing client (an AppClient holding console:pair, registered with sms-gateway register-client --scope console:pair, never through the console). The person gets an admin account on first pairing, never an owner. This path depends on vaam’s relay (VA-4); until it exists, use the next one.

• You, on the command line, which is also the break-glass and the only way to enrol an owner:

  docker compose --env-file .env run --rm sms-gateway pair-console-key \
    --user you@example.com --public-key "<the k field of the QR>"

  It attaches the key to an existing active account, with no proof of possession (you holding the database is the authority), creates no user and changes no role. The console notices on its own within a couple of seconds.

A key lasts 30 days and is never extended; the console says when it is close and asks for a new pairing when it ends. /console ("This console") revokes it.

6. Bring up admin and Caddy

docker compose --env-file .env up -d --build admin caddy
docker compose ps

admin's HEALTHCHECK hits its existing GET /api/health, which touches no external dependency and needs no credential: the console has no login, so there is nothing in front of it to exempt. caddy depends on both sms-gateway and admin being healthy before it starts routing — see deploy/Caddyfile's own comment for why the gateway and admin domains have to stay two separate origins rather than one path-routed domain (the console keeps its paired browser key in IndexedDB, which the browser scopes to the console’s own origin).

deploy/Caddyfile uses rate_limit, a third-party module the stock caddy:2-alpine image does not carry — deploy/docker-compose.yml's caddy service builds from deploy/caddy.Dockerfile (an xcaddy build adding it) for exactly that reason, not the stock image. An earlier revision of this paragraph said the compose edit to pick that Dockerfile up was still outstanding; that edit landed and this paragraph was simply never corrected to match — found while writing this file’s own "Backend-only deployment" section below, the same "documentation asserts something the code does not do" shape AGENTS.md keeps rediscovering.

7. Verify

curl -f https://$SMS_GATEWAY_DOMAIN/healthz
curl -f https://$SMS_ADMIN_DOMAIN/api/health

Both should return 200. Every other route of the gateway is sealed: an unsigned request, such as curl -i https://$SMS_GATEWAY_DOMAIN/jobs, is refused with a bare 401, and a request for a route that does not exist (/token, /login) answers 404. If you’re on the tls internal local/demo path, add -k or trust Caddy’s local CA first. This exact sequence — all five containers, real health checks, a real Caddy TLS hop in front of both origins — was run end to end against a throwaway config while building this PR, using tls internal in place of real DNS; see the PR description for what specifically was and wasn’t verified.

Enabling MTN delivery receipts (MADAPI)

Only if this deployment sends through MTN. Orange needs nothing here — its DR endpoint is whitelisted by a manual support ticket, covered in 36-handset-gate.adoc.

MTN delivery receipts do not arrive unless you register an endpoint first. MADAPI requires two independent things: requestDeliveryReceipt: true on every outbound message, which sms-provider-mtn always sends, and a deliveryReportUrl registered against your approved serviceCode. Skip the registration and messages submit successfully, reach submitted, and then sit there permanently with no receipt ever arriving — which looks exactly like a broken route and is why this section exists rather than a line in a table.

Unlike Orange’s whitelisting, this is a real API call, so run it once per approved short code:

docker compose run --rm sms-gateway mtn-subscribe-dlr \
  --delivery-report-url "https://$SMS_GATEWAY_DOMAIN/dlr/mtn_cm" \
  --target-system vsms

MTN’s credentials are the mtn_cm entry of the YAML configuration (see config/application-prod.yml and deploy/.env.example), so they are not repeated on the command line — the secret in particular should never land in shell history. The command loads every block of the document but security, so every other ${NAME} it writes must be set in the environment it runs in and SMS_HASH_PEPPER need not be (it hashes nothing, so it is neither required nor read): the compose service already has them, which is why the command above uses the sms-gateway service.

Note the path segment is mtn_cm, matching SmsProvider::key and what sms-gateway's /dlr/{providerKey} handler matches against. A subscription registered against any other path will be accepted by MTN and then silently 404 on every receipt.

run --rm, not exec: this command needs no database and no running gateway, and the gateway may legitimately not be up yet. run starts it from the sms-gateway service definition, which is what gives it the configuration mount and the environment.

Backend-only deployment (R4, CONTRIBUTING.md)

"Some deployments will ship the backends only, with no admin surface at all. That is a supported configuration, not a degraded one." Both admin and caddy carry profiles: [console] in deploy/docker-compose.yml — a bare docker compose --env-file .env up -d brings up postgres, migrate, sms-gateway, sms-worker, backup and prometheus (the last two unconditional: R4’s own fourth clause, "a backend-only deployment must still be observable and operable," and neither is console-specific) and nothing console-shaped at all — not admin, not the bundled Caddy edge. Add --profile console (or COMPOSE_PROFILES=console) to bring both back.

Why caddy is gated too, not just admin. deploy/Caddyfile is one file with two unconditional site blocks, {$SMS_GATEWAY_DOMAIN} and {$SMS_ADMIN_DOMAIN} — see that file’s own header for the reason two domains exist at all (the console keeps its paired browser key in IndexedDB, which the browser scopes to its own origin, so path-routing one domain to both services isn’t an option). There is no way to make only the second block optional without editing that file, which is a third-party-module build out of scope here. A backend-only compose deployment therefore has no bundled TLS edge for sms-gateway either — front it with your own reverse proxy/ingress, the same "operator’s own opinion" posture deploy/charts/vsms/values.yaml already takes for ingress.gateway (disabled by default), or enable --profile console anyway and simply never start admin itself.

A :?-required env var interpolates for every service in the file, independent of which profile is active or which services you actually ask to start — confirmed live, not assumed (docker compose config failed on SMS_API_URL with admin excluded by every measure available: no --profile console, not named on the command line). Every console-only var in admin's and caddy's own environment: blocks moved from :? to :- for exactly this reason; the fail-loud property survives at the point that actually matters instead — @vsms/env's zod validation refuses to boot admin without the gateway’s two public keys, and Caddy refuses to serve an empty site address once --profile console is actually used without SMS_GATEWAY_DOMAIN/SMS_ADMIN_DOMAIN set.

What was actually run, not just rendered

helm template --set admin.enabled=false (with every other value the chart requires unconditionally, and no console-only value) renders exactly Service vsms-gateway, Deployment vsms-gateway/vsms-worker, and the three bootstrap Jobs — no admin Deployment/Service anywhere. admin.enabled=true with the same input set fails immediately, naming admin.serverKeys.current is required. Both confirmed by rendering, not inferred from the chart’s own logic.

The compose half was run against a real stack — a fresh deploy/.env with every var this file marks required except every console-only one (SMS_ENVELOPE_SERVER_KEY_CURRENT, SMS_ENVELOPE_SERVER_KEY_NEXT, CADDY_ACME_EMAIL, SMS_GATEWAY_DOMAIN, SMS_ADMIN_DOMAIN all absent), the orange_cm entry’s host.url pointed at a manually-run sms-fake-orange container on the compose network (no fake-orange profile exists in this file yet — see "Known seams" above) rather than a real Orange account, since this section’s job is proving R4, not re-proving 36-handset-gate.adoc:

$ docker compose --env-file .env config --services
postgres
migrate
backup
sms-worker
sms-gateway
prometheus

No admin, no caddy — and this rendered with the console’s variables all unset, which is the property that matters (a :? anywhere in the file would have failed this same command). postgres, migrate, sms-gateway, sms-worker were then brought up for real (docker compose up -d, plus the one-shot seed-dispatch sub-step of step 3 above, see its own "What bootstrap does" subsection) — both reported healthy.

Provisioning and sending went entirely through sms-gateway's own CLI and the real HTTP API, never a browser. seed-demo-app below is what this proof actually ran at the time (#233) — it no longer exists as an sms-gateway subcommand (production-readiness audit S1: its logic moved to its own crate/image, vsms-demo-seed, ghcr.io/vaam-apps/vsms/demo, demo-only by its own --help text either way). A real backend-only production deployment reaching this same point today uses sms-gateway create-app --slug vsms-demo --name "vsms demo" instead (see step 5 above) — transcript kept verbatim as a historical record of what was actually run, not as a command to copy-paste today:

$ docker compose run --rm sms-gateway seed-demo-app
created App c0860e5f0ec7a547a3b7877 (slug="vsms-demo")
created SenderId "VSMS" (cd1271a3eada748335198b4)
registered SenderId "VSMS" as approved (provider=c7f86a470ccbb1767495d66)
activated SenderId "VSMS"
demo App/SenderId fixtures ready (slug="vsms-demo", sender="VSMS")

$ docker compose run --rm sms-gateway provision-client \
    --app-slug vsms-demo --label "backend-only R4 proof" \
    --scope sms:send --scope sms:read \
    --key-out secrets/backend-only-client-key.pem \
    --client-id-out secrets/backend-only-client-id
provisioned client: appc_b2604bf02dab4e729a7ee4e7a08e2571
private key written to: secrets/backend-only-client-key.pem
client id written to: secrets/backend-only-client-id

$ examples/rust/sms-send --issuer http://sms-gateway:8080 \
    --client-id appc_b2604bf02dab4e729a7ee4e7a08e2571 \
    --private-key-path secrets/backend-only-client-key.pem \
    --scope "sms:send sms:read" \
    --to +237677123456 --sender-id VSMS \
    --body "R4 backend-only proof: no console anywhere"

sent: messageId=c0d35cb260e5732220b99bc state=accepted encoding=gsm7 segments=1 operator=mtn estimatedCostXaf=0
read back: id=c0d35cb260e5732220b99bc state=accepted providerMessageRef=None

examples/rust/sms-send made the real sendMessage call over HTTP — the same code path a genuine third-party backend integration uses, not an in-process shortcut. The message reached delivered a few seconds later, confirmed three independent ways rather than trusted off one:

$ docker exec vsms-postgres-1 psql -U vsms -d vsms -tAc \
    "select id, state, operator, provider_message_ref from messages where id='c0d35cb260e5732220b99bc';"
c0d35cb260e5732220b99bc|delivered|mtn|res-c0d35cb260e5732220b99bc

$ docker logs sms-fake-orange | grep c0d35cb260e5732220b99bc
... fake orange: submit received reference="c0d35cb260e5732220b99bc" outcome=Accepted ...
... fake orange: DLR posted endpoint="http://sms-gateway:8080/dlr/orange_cm" reference="c0d35cb260e5732220b99bc" status=DeliveredToTerminal http_status=202

$ docker logs vsms-sms-gateway-1 | grep c0d35cb260e5732220b99bc
... message accepted message_id=c0d35cb260e5732220b99bc app_id=c0860e5f0ec7a547a3b7877 ... state=accepted
... DLR applied message_id=c0d35cb260e5732220b99bc from_state=submitted to_state=delivered provider_ref=c0d35cb260e5732220b99bc

docker ps at the end of this run showed exactly vsms-postgres-1, vsms-sms-gateway-1, vsms-sms-worker-1, and the manually-run sms-fake-orange stand-in — no admin, no caddy, anywhere on the host.

Every operator action this proof needed had a CLI subcommand — nothing required a browser. rotate-signing-key, seed-dispatch, seed-demo-app (demo-only, see above), provision-client: all four were plain sms-gateway subcommands at the time. (rotate-signing-key and provision-client were deleted with the OpenID Provider in VS-4; today the same proof registers its own Ed25519 key with register-client, see step 5.) The one real gap this proof found, not closed by #233 itself: a real production deployment’s first App had no CLI path. seed-demo-app was explicitly demo-only by its own --help text; the only other route to App.create was a human token from a console login (#194), which a genuinely backend-only deployment has no console to obtain from. Closed by sms-gateway create-app (production-readiness audit N1, step 5 above) — the same hand-built owner-role-context write every other bootstrap subcommand in this file already uses, not a raw psql/generated-CRUD write under a manually-bootstrapped context any more.

Rate limiting (#156) — what was actually proven, against a real stack

deploy/Caddyfile's rate_limit block was verified against a genuinely separate compose project (-p vsms156, non-default host port 18080, built from deploy/caddy.Dockerfile and the same sms-gateway/migrate images this tree produces elsewhere), not asserted from reading the config. It once also throttled the gateway’s /token endpoint; that endpoint, and with it the two token_ zones and SMS_TOKEN_RATE_LIMIT_*, were deleted with the OpenID Provider (VS-4).* What remains at the edge is the one unauthenticated route, /dlr/{providerKey}; every other route is sealed, and an unsigned request is refused at the envelope layer before any database work.

/dlr/{providerKey}, per-source-IP zone (tested at a scaled-down events=4, window=20s; shipped default 100/10s): the first 4 requests to /dlr/orange_cm got a real 400 from sms-gateway (malformed DLR payload, on purpose — proves the request reached the app), the 5th onward got 429 with Retry-After set, straight from Caddy (Server: Caddy, no application body). Waiting out the window recovered it.

Keying, verified both directions, not just claimed:

• A second, genuinely different source IP (a separate container on the compose network, confirmed via Caddy’s own log_key-enabled logs to be a distinct remote_ip) got its own, independent budget — unaffected by the first source’s bucket already reading 429 — and throttled identically once it sent enough requests.

• Sending X-Forwarded-For: 1.2.3.4 and, on a separate request, X-Forwarded-For: 9.9.9.9 from an already-throttled source did not create new buckets and did not lift the throttle — both still got 429. Confirms the key is genuinely {http.request.remote.host} (the real TCP peer Caddy accepted the connection from), not anything a client-supplied header can influence.

That burst was originally run against a composite (IP, path) key, and the run also showed a different providerKey from the same source getting its own bucket. That composite key has since been removed, so the per-path isolation it demonstrated no longer applies. The reason is in deploy/Caddyfile's own comment on the zone: a composite key silently disables ipv6_prefix, because the module only masks a key that parses as a bare IP. Keeping IPv6 masking was judged the more valuable of the two, since only one provider is wired up today.

Not verified by this burst test, and worth being explicit about: the aggregate zone (dlr_global) was confirmed to parse and load (caddy validate, and log_key showed it evaluating on every request) but was not driven past its own (200/10s) threshold by volume; it shares the exact same sliding-window mechanism already proven above, just with a static instead of a dynamic key.

Ordering and failure modes, summarized

Component	Waits on	What "not ready" looks like
migrate	postgres healthy	exits non-zero; nothing downstream should start (compose enforces this via service_completed_successfully)
sms-gateway	postgres healthy, migrate completed	will not even start without the two envelope keys of step 1 or step 3’s orange_cm Provider row — both are resolved before the listener ever binds, so either one missing crash-loops the container (#148)
sms-worker	postgres healthy, migrate completed	starts fine with no envelope keys (only sms-gateway loads them) and fine with no Provider/Route either — dispatch role specifically needs step 3’s Provider row and its catch-all Route (#62) to route anything, other roles don’t, and none of them crash-loop on the absence of either the way sms-gateway does on a missing Provider: a dispatch role with nothing to route to just rejects every message it claims, silently, forever
admin	sms-gateway healthy	starts; the composer reports it has no key until step 5’s is registered, and every other screen works
caddy	sms-gateway and admin healthy	won’t route until both are up

Two instances of migrate starting at once (a redeploy racing a still-up previous stack) serialise on a Postgres advisory lock (backends/apps/sms-migrate/src/main.rs) rather than double-applying — see that binary’s own module doc for why a schema_migrations tracking table exists only in this deploy path, not in the committed backends/migrations/ tree.

Secrets — the decision, and what it doesn’t protect against

No sops here, despite §9.2 mentioning it. This deployment’s secrets are: POSTGRES_PASSWORD, ORANGE_CM_CLIENT_SECRET (a ${...} placeholder in config/application-prod.yml, which holds no secret itself), SMS_HASH_PEPPER, the admin console’s console-composer.pem, and the backup service’s rclone.conf. The first three live in deploy/.env (chmod 600, never committed — .gitignore/.dockerignore both exclude it); the last two live in deploy/secrets/ (also never committed — .gitignore ignores everything in that directory except the two example/placeholder files this repo ships) mounted read-only into the one container that needs each.

deploy/secrets/rclone.conf must exist before the first up — even when BACKUP_RCLONE_REMOTE (.env) is a bare local path and rclone needs no real credentials at all. Production-readiness audit B2: a volumes: bind mount onto a not-yet-existing host path used to be created silently as an empty directory by the Docker daemon, and backup would then fail obscurely against a directory where it expected a config file. deploy/docker-compose.yml’s `backup/admin services now reference both files as configs: instead, which fails docker compose up outright ("bind source path does not exist") if either is missing — so this step is no longer optional busywork, it’s what keeps up from either silently misbehaving or refusing to start at all:

cp secrets/rclone.conf.example secrets/rclone.conf
chmod 600 secrets/rclone.conf
# then fill in a real remote's credentials — see that file's own header,
# or docs/runbooks/backup-restore.adoc — unless BACKUP_RCLONE_REMOTE is a
# bare local path, in which case the copied-verbatim empty file is
# already a valid, working config.

That is genuinely less than §9.2’s own aspiration, and the honest threat model is: this protects against the secrets ending up in git history, in a Docker image layer, or in docker inspect output visible to anyone without host access. It does not protect against anyone with read access to the VM’s filesystem, and it does not give you audited, rotatable secret access the way sops + age/KMS would. For a single VM with a small operator set, that trade is reasonable; it stops being reasonable the moment more than a couple of people need host access, or this deploys to more than one VM. Revisit with real sops (age-encrypted, keyed to each operator, decrypted only at container start) before either of those becomes true — not preemptively, since a half-wired sops setup that still leaves plaintext secrets in .env for the vars this file doesn’t cover would be worse than being honest about the current gap.

Releasing a new version

.github/workflows/release.yml builds and pushes four images (sms-gateway, sms-worker, admin, and migrate — #145 added the fourth, for the Helm chart’s migrate Job below) to GHCR, plus the deploy/charts/vsms Helm chart as an OCI artifact, on two triggers:

• Every push to main — :main (mutable, moves with each push) and :sha-<12 hex> (immutable, one commit). The chart publishes as 0.0.0-main.<sha>.

• Any v.. tag* — :$TAG and :latest, unchanged from what this workflow did before #145. The chart publishes as $TAG with its leading v stripped (Chart.yaml requires strict semver).

A third trigger, workflow_dispatch, exists only to smoke-test the pipeline itself from a branch that isn’t main and isn’t tagged — it tags images :branch-<slug> / :sha-<12 hex> and the chart 0.0.0-dev.<sha>, deliberately never touching :main/:latest. #145’s own PR used this to prove the pipeline works before this section existed to document it — see that PR for the exact GHCR references it produced.

On the VM (compose path), two ways to upgrade — production-readiness audit S6 closed the gap where only the first of these actually worked:

Build from source (unchanged, still the default — build: is still present on every service that has one):

cd deploy
git pull
docker compose --env-file .env up -d --build

Pull pre-built GHCR images — migrate/sms-gateway/sms-worker/admin now also carry a matching image: (docker-compose.yml’s own header comment has the full reasoning), so `pull has something real to fetch instead of silently no-op’ing against a locally-built, untagged image:

cd deploy
export VSMS_IMAGE_TAG=v0.3.1   # or the tag you're upgrading to
docker compose --env-file .env pull
docker compose --env-file .env up -d

VSMS_IMAGE_OWNER (default vaam-apps) is the second half of that reference — release.yml publishes to ghcr.io/${{ github.repository_owner }}/…​ unconditionally, and this repository’s own GitHub owner has moved twice: v0.2.1 and earlier published under ghcr.io/vymalo/…​, v0.3.0 alone under ghcr.io/vaam-store/…​, and v0.3.1 on lands under ghcr.io/vaam-apps/…​, this file’s own default now. Pulling an older release means setting BOTH knobs together, never one without the other:

export VSMS_IMAGE_OWNER=vymalo
export VSMS_IMAGE_TAG=v0.2.1

Either path is a real, working upgrade mechanism now; before this PR, docker compose pull was a documented no-op against a compose file with no image: anywhere, silently leaving up -d to build (or reuse) a local image regardless of what had actually been pulled.

Kubernetes (Helm)

deploy/charts/vsms — the same three long-running processes as the compose path (sms-gateway, sms-worker, admin), plus the one-shot steps compose’s own runbook above walks through by hand (apply migrations, seed the orange_cm Provider row), as Helm-hook Jobs that run automatically in the right order. Built directly on the bjw-s common library chart v4 (^4.6.2), per #145.

One correction to #145’s own text, found while building this, not assumed: the issue named oci://ghcr.io/bjw-s-labs/helm/common as the dependency reference. That artifact does not exist — confirmed against GHCR’s own API (gh api orgs/bjw-s-labs/packages?package_type=container lists 21 packages; helm/app-template and helm/multus are there, helm/common is not) and against bjw-s-labs/helm-charts' own .github/workflows/charts-release.yaml, whose release-library-charts job runs with publishToOciRegistry: false — library charts (common is one) publish to the classic index.yaml repo (https://bjw-s-labs.github.io/helm-charts) only, never to GHCR as OCI. deploy/charts/vsms/Chart.yaml depends on common through that classic repository URL instead — byte-identical content, same source repo, same release pipeline, just a different distribution channel — pinned to 4.6.2, the newest v4 release as of this PR. Only deploy/charts/vsms itself publishes to GHCR as OCI; that part of #145’s ask stands as written.

What the chart does not do

• Does not run Postgres. existingSecrets.database.name names a Secret containing a DATABASE_URL key the operator’s own database (or their own operator/subscription in front of one) supplies — the same "bring your own database" split the compose path already documents for why it doesn’t use sops.

• Does not create any Secret. Every credential (DATABASE_URL, SMS_HASH_PEPPER, ORANGE_CM_CLIENT_SECRET, MTN_CLIENT_SECRET and AWS_SMS_SECRET_ACCESS_KEY when those providers are configured, the composer’s service key PEM, the two envelope key PEMs) is referenced by name from an existingSecrets.* value; nothing under deploy/charts/vsms/templates ever writes one into a ConfigMap or a rendered manifest. The provider configuration itself, which holds ${NAME} placeholders and no secret, is the config.overlay ConfigMap (below). See values.yaml's own existingSecrets block for the exact key each Secret must contain.

Ordering — why a Helm hook, not an init container

sms-gateway serve calls resolve_provider_row_id(...)? before binding its listener (this file’s own steps 3 and 4, above) — a fresh database makes the container exit immediately on a missing Provider row, not fail its first DLR callback (#148: the orange_cmProvider row used to have no seeding path under Helm at all — no window exists between the pre-install hooks completing and the gateway Deployment being created for an operator to seed anything by hand, unlike compose, where a human can at least docker compose exec into Postgres while the container crash-loops). The chart’s seedDispatch controller is a pre-install Helm hook Job for exactly this reason: Helm does not create any other release resource — including the sms-gateway Deployment — until every pre-install hook Job has succeeded. A post-install step or a readiness check on the gateway would both deadlock permanently, since the gateway can never become ready without this step having already run.

migrate (weight -20) and seedDispatch (weight -15, renamed from seedProvider in #62 — see that renamed job’s own comment in values.yaml for why: it seeds a catch-all Route alongside the Provider row now, since a Provider with no Route routes nothing since #62’s engine landed) are both pre-install,pre-upgrade hooks — safe on every deploy because both are genuinely idempotent: migrate via backends/apps/sms-migrate's own advisory-lock-guarded schema_migrations tracking table, seedDispatch via sms-gateway seed-dispatch's own dedupe on both halves (Provider: create + catch-23505; Route: look up an existing one for the provider first — see step 3 above). The distinct weights keep the two hooks a single, strictly-ordered chain rather than two of them racing at a shared weight; see values.yaml's own comment on seedDispatch for the full reasoning. (A third hook, rotateSigningKey, minted the signing key of the embedded OpenID Provider and went with it in VS-4.)

The configuration: config.overlay

The chart has no provider values any more (VSC-1). The orange., mtn. and awsSms. blocks are gone, and the chart *refuses a release that still sets one (guard retired-values), naming where it went. What replaces them is config.overlay: the same YAML document the compose path edits, as text, which the chart renders into a ConfigMap and mounts — with subPath, over /config/application-<profile>.yml — into the gateway and the worker, so the two cannot disagree about a provider. subPath is what keeps the image’s base file: a ConfigMap mounted at /config would replace the directory. The mounted file does not update in place, so an edit has to roll the Pods, and the library chart does it: it puts checksum/configMaps, a hash of every ConfigMap the release renders, on each controller’s pod template (bjw-s common 4.6.2). The chart used to add a checksum/sms-config of its own to the gateway and the worker; it said the same thing and is gone. The library’s reach is wider than that one was: an overlay edit also rolls the admin console’s Pod, and changes the pod template of the migrate and seed Jobs, because every controller’s template carries the same hash. Expect the admin Pod in the rollout of a configuration-only change. A hook Job is created anew by each helm upgrade whatever its template says, so for the Jobs the changed hash is not a visible event.

A ConfigMap of your own, mounted beside the overlay, goes under configMaps and persistence as in any bjw-s release; the chart merges the overlay into that map, so yours is kept. The key overlay is the chart’s: a release that sets configMaps.overlay is refused (guard config-maps-overlay).

The overlay is required (guard config-overlay), and there is no switch to leave its ConfigMap out: the base file lists no provider, and a gateway with none refuses to start, so a release without one would be a crash-looping Pod found by kubectl get pods instead of by helm template. It may not contain {{ (guard config-overlay-template), because the library chart renders ConfigMap data through tpl, and config.profile must be a safe file name (guard config-profile). A release that still sets config.createOverlayConfigMap from an earlier revision of this change is refused (retired-values).

config:
  profile: prod
  overlay: |
    deployment:
      name: vsms
      livemode: true
      public_base_url: https://sms.example.com
    providers:
      - code: orange_cm
        credentials:
          client_id: <the Orange client id>
          client_secret: ${ORANGE_CM_CLIENT_SECRET}   # existingSecrets.orange
        settings:
          sender_number: "+237..."

The secrets the overlay reads as ${NAME} come from existingSecrets.hashPepper (always, and to the gateway alone: the worker does not load security.), and existingSecrets.orange, .mtn and .awsSms (each optional: an empty name leaves its variable out of both Deployments). Anything else the overlay reads is added to *both controllers' env. The two seed hooks follow the overlay by hand, because the chart cannot read an opaque overlay: controllers.seedDispatch.enabled: false when there is no orange_cm entry, and controllers.seedDispatchAws.enabled: true (with --max-tps and --cost-per-segment-xaf appended to its args) when there is an aws_sms one.

Installing

helm repo add bjw-s-labs https://bjw-s-labs.github.io/helm-charts   # or use --repository-config
helm dependency build deploy/charts/vsms

# Pre-create every Secret existingSecrets.* names — see values.yaml's own
# comments for the exact key each one must contain. Example for two of
# them:
kubectl create secret generic vsms-hash-pepper \
  --from-literal=SMS_HASH_PEPPER="$(openssl rand -base64 48)"
kubectl create secret generic vsms-console-composer-key \
  --from-file=console-composer.pem=./console-composer.pem

# overlay.yaml is the YAML overlay (config.overlay above), as a file.
helm install vsms deploy/charts/vsms \
  --set image.tag=<a real tag published by release.yml — see above> \
  --set-file config.overlay=overlay.yaml \
  --set admin.serverKeys.current=... \
  --set admin.serverKeys.next=... \
  --set existingSecrets.database.name=vsms-database \
  --set existingSecrets.hashPepper.name=vsms-hash-pepper \
  --set existingSecrets.orange.name=vsms-orange \
  --set existingSecrets.consoleComposerKey.name=vsms-console-composer-key

The composer’s key is optional (existingSecrets.consoleComposerKey.name may stay empty when the console is disabled or the composer is not used); the console’s own chart values are admin.serverKeys.current and admin.serverKeys.next, the two public keys sms-gateway server-keys prints. values.yaml’s own `existingSecrets block is the source of truth for the secrets: database, hashPepper, orange, mtn, awsSms and consoleComposerKey.

Every one of image.tag, the existingSecrets.*.name fields the gateway needs, the overlay, and (with the console on) admin.serverKeys.current/.next is enforced with Helm’s required function or a named chart guard — an install with any of them missing fails immediately with a named field, not a cryptic render error or a silently-broken Pod.

Verification actually performed for this PR

The VSC-1 change to this chart (the overlay, the guards and the removed values) was first written without helm, and read against bjw-s common 4.6.2’s own templates instead of rendered. Rendering it afterwards (helm v3.16.4, the tarball’s SHA-256 checked against get.helm.sh’s own .sha256sum) found a defect that reading had not: the overlay ConfigMap and the volume that mounts it got different names for any release whose name does not contain "vsms". templates/configmap-overlay.yaml was rendered before templates/common.yaml (Helm renders templates in reverse path order), so its name was computed before common.yaml set global.nameOverride: helm template foo produced ConfigMap foo-vsms-config and, in both Deployments, configMap: name: foo-config. The Pods would have hung in ContainerCreating, and helm lint --strict was green throughout.

The fix is that there is one helper for both names. The ConfigMap is now common’s own configMaps item (overlay, defined in templates/common.yaml after global.nameOverride is set) and the persistence item config-overlay mounts it by identifier, never by a name written beside it; templates/configmap-overlay.yaml is deleted. Rendered after the fix, under release names foo, vsms and my-vsms-prod, the ConfigMap is <release>-config and both Deployments mount that name. cargo xtask chart-render (the rules job in ci.yml, with azure/setup-helm pinned) now does this on every pull request: helm dependency build, helm lint --strict, helm template under a release name without "vsms" and one with, an assertion that every ConfigMap a workload mounts is one the chart renders and that the overlay is mounted by the gateway and the worker, and one render per guard, each of which must fail with its own name in the message. Its sabotage proof: with the old template restored, it fails with "a workload mounts ConfigMap config-render-check-config, which the chart does not render (it renders config-render-check-vsms-config)".

The same render run refused, each with its guard’s name: the retired orange, mtn and awsSms values (including orange: null and mtn: {}), the removed config.createOverlayConfigMap, an empty and a whitespace-only overlay, an overlay containing {{ (common renders ConfigMap data through tpl), and the profiles Prod, ../x and a/b. A flag that could only ever be true was dormant configuration and is gone: the overlay is always required and its ConfigMap always rendered, and a release that still sets the flag is refused (retired-values) rather than ignored.

The chart still has not met a cluster. Nothing here installed it, so the checksum annotation’s rolling restart, the subPath mount over the baked /config directory and the Pods' first start on the overlay remain unobserved.

The paragraph below describes the earlier chart and is kept as the record of what was verified then.

helm lint --strict and helm template (with a full set of test values — no chart default was left to guess at) both ran clean, and the rendered output was read, not assumed clean: hook annotations and weights land in the order described above, no existingSecrets value’s contents ever appear in a rendered manifest (only Secret names/keys do), and the probe paths match /healthz (gateway), /api/health (admin), and the worker’s heartbeat-file exec check exactly. See the PR description for whether this also reached a real kind/k3d cluster and, separately, whether the GHCR artifacts this section references were confirmed to exist (docker buildx imagetools inspect / helm pull --version), not just a green workflow run — this repo has been bitten by exactly that gap before (#87, AGENTS.md).