vsms

SMS Gateway — Architecture Design ​

Scope: a self-hosted A2P SMS gateway, Cameroon-first (MTN + Orange direct, with AWS End User Messaging SMS as an additional route) and extensible to any country, serving OTP and notification traffic for one organisation's own applications, with a Next.js admin console.

On "Cameroon-first". This document was written for Cameroon alone, and most of it still reads that way; treat a bare Cameroonian fact below as the default case rather than the only one. Cameroon stays the best-supported market and the default region, and a Cameroon deployment is not asked to configure anything it did not configure before. Every other country is reached through data and configuration rather than a code change. docs/design/multi-country.md records what that changed, what it deliberately did not, and which markets remain blocked on work nobody has done — sender-ID regimes, aggregator coverage and data-residency conflicts especially.

Stack (as chosen): Rust + TypeScript · CrateStack (.cstack schema-first) · the signed transport (§4.0: every request a COSE_Sign1 envelope, verified against a registered public key) · full RBAC through the verified signer's role and scopes · one Ed25519 key per calling app, one paired browser key per console user. There is no token, no password and no OpenID Provider (deleted in VS-4, 2026-10-27).

Decisions locked in this revision ​

QuestionAnswer
TenancySingle org, multiple apps. No tenantId in the core model.
Operator pathProvider abstraction over both HTTP and SMPP; ship HTTP first.
Admin UIHand-built Next.js + shadcn/ui over the generated TypeScript client.
Transporttransport rest (CrateStack default). gRPC is a later, optional addition.
Machine authA registered Ed25519 public key per app client; every request a COSE_Sign1 envelope (§4.0). No API keys, no tokens, no shared secrets.
IdentifiersCuid for every model id and FK.
EventingCrateStack @@emit + events::Subscriptions. No hand-rolled outbox table.
Background workOne sms-worker node, role-selectable, singletons elected by Postgres advisory lock.
State machinesEnforced in Postgres by transition tables + BEFORE UPDATE triggers, not only in Rust.

The schema in §2 was assembled into a single file and run through the real cratestack-parser 0.4.16 and the real Postgres migration emitter; the policy and @authorize rules — which live in cratestack-macros, not the parser — were proven by compiling a real include_server_schema! expansion. Rust APIs were checked against published crate sources. Every Mermaid diagram here was rendered with mmdc to confirm it parses. The validated schema ships alongside this document as schema.cstack.

0. Corrections to the brief ​

1. CrateStack does not consume .proto. The schema language is .cstack — Prisma-shaped. Protobuf runs the other direction: cratestack generate-proto emits a .proto plus a committed .pb.lock field-number lockfile, and transport grpc swaps the REST binding for a tonic service. That path is real but currently model CRUD only — procedures aren't wired into the generated gRPC service, and where/or predicates aren't wired into either generator. Since the send path is a procedure, REST is the only viable binding today.

2-4. Superseded (VS-4, 2026-10-27). The three corrections that stood here concerned authkestra-op, the OpenID Provider the gateway used to embed: that authkestra-op is the provider crate (not authkestra-oidc), that Authkestra has no RBAC, and that authkestra-op 0.2.3 could not serve client_credentials from a persisted client store. The provider, authkestra and the token plane are deleted (§4.0, §4.2, §4.3), so the corrections no longer correct anything in this tree; they remain in git history. RBAC is still ours to build (§5).

R. Engineering rules ​

Three rules that constrain every other decision in this document. They are stated here rather than scattered through it because each is a default with named exceptions, and the exceptions are the interesting part.

R1 — All data access goes through CrateStack delegates. Never raw sqlx ​

The generated Cratestack runtime is the only way application code touches the database. db.message().find_many()…, db.message().update(id).set(…).if_match(v), db.job().create(…), .run(&ctx) or .run_in_tx(&mut tx, &ctx). No sqlx::query!, no query_as, no query_scalar, no raw_sql.

This is not stylistic. Going through the delegates is what keeps four guarantees switched on, all of which raw SQL silently bypasses:

  • Row-level policy. @@allow compiles to SQL predicates appended to your WHERE. A raw query is unscoped — it sees every tenant's rows, every soft-deleted tombstone, every message belonging to another app.
  • Audit. @@audit rows are written inside the mutation's transaction. A raw UPDATE produces no audit entry, so the trail silently develops holes exactly where someone worked around the framework.
  • Events. @@emit writes the outbox row in the same transaction. A raw UPDATE that moves a message to delivered fires no message.delivered webhook. The customer is never told.
  • Optimistic locking and soft delete. @version bumping and deleted_at IS NULL scoping are applied by the delegate layer, not by the database.

A raw query that touches messages is therefore not "the same query, written by hand" — it's a different query with four safety properties removed.

The named exceptions. Raw SQL is permitted in exactly these places, and nowhere else:

ExceptionWhereWhy the delegates can't do it
DDL and migrationsbackends/migrations/**Triggers, partial indexes, foreign keys, column defaults, transition tables. Not data access at all — the emitter produces none of these (§2.10).
Advisory locksbackends/crates/sms-worker/src/lease.rspg_try_advisory_lock / pg_advisory_unlock. No delegate expression exists; it isn't a table.
LISTEN / NOTIFYbackends/crates/sms-worker/src/notify.rs, backends/crates/sms-api/src/cache.rsCache-invalidation fan-out (§8.3). No delegate expression exists.

That is the complete list. Two things people reach for that are not on it:

  • Transactions. Every delegate builder has .run_in_tx(&mut tx, &ctx) — 26 of them across cratestack-sqlx. You never need raw SQL for transactional work, and delegates inside a caller-managed transaction still write their audit rows and outbox rows into that same transaction. cratestack::run_in_isolated_tx(pool, isolation, closure) handles SERIALIZABLE with 40001 retries.
  • Row locking. .for_update() exists on find_many and find_unique, and appends a real FOR UPDATE.

The one genuine gap is SKIP LOCKED, which the framework cannot express — a workspace-wide grep finds zero occurrences, and skip_locked() / nowait() / lock_mode() are compile errors. §7.3 explains why the queue design doesn't need it, and what to do if profiling ever says otherwise.

Enforced in CI:

bash
# no raw sqlx outside the three named exceptions
! grep -rn --include='*.rs' -E 'sqlx::(query|query_as|query_scalar|raw_sql)\b' crates/ \
    | grep -vE 'sms-worker/src/(lease|notify)\.rs|sms-api/src/cache\.rs'

db.pool() is the escape hatch that makes raw SQL possible; the lint is what makes it deliberate. Adding a row to the exceptions table should feel like a design decision, because it is one.

R2 — State transitions are proposed by Rust and decided by Postgres ​

Legal edges live in message_state_transitions / job_state_transitions; BEFORE UPDATE triggers reject the rest with SQLSTATE SM001. Application code never assumes a transition is valid because it checked first. §2.10 and §7.4.

R3 — Nothing that must be written can be @server_only ​

@server_only excludes a field from both create and update inputs, so under R1 such a field can never be populated at all. It is for columns the database owns, not for secrets. Field secrecy comes from model-level @@allow and from keeping secrets out of the gateway database entirely. §2.0 has the full attribute matrix.

1. System topology ​

mermaid
flowchart TB
    APP["Your apps<br/>OTP · notifications"]
    HUMAN["Operators<br/>browser"]
    EDGE["Edge — Caddy + rustls<br/>TLS 1.3 · HSTS<br/>IP allowlist on /adm · rate limit on /dlr, /login, /token"]

    subgraph online["Online tier — stateless, scale to N"]
        direction LR
        API["sms-api<br/>Axum + CrateStack<br/>procedures<br/>+ event subscribers"]
        ADMIN["admin<br/>Next.js BFF<br/>shadcn/ui"]
    end

    WORKER["sms-worker<br/>all background processing<br/>roles in §7"]
    PG[("PostgreSQL<br/>system of record<br/>queues · state machines")]
    OPS["Orange CM · MTN<br/>aggregators"]

    APP -- "sealed request (COSE_Sign1)" --> EDGE
    HUMAN -- "sealed request, paired browser key" --> EDGE
    EDGE --> ADMIN
    EDGE --> API
    ADMIN -- "carries sealed bytes" --> API
    API -- "row + @@emit outbox row, one tx" --> PG
    PG <-- "SKIP LOCKED · advisory locks" --> WORKER
    WORKER -- "submit" --> OPS
    OPS -. "DLR callback" .-> EDGE
    WORKER -- "signed webhooks" --> APP

Three tiers and one database. The online tier is stateless and scales freely; everything with a schedule, a lease, or a long-lived connection lives in sms-worker; Postgres is the only coordination mechanism in the system — no broker, no Redis, no consensus library.

Why these process boundaries ​

NodeCrateWhy it's separate
sms-apicratestack-pg + AxumStateless, scale to N. Holds the CrateStack router, the procedures, and the event subscribers.
sms-workerplain tokioAll background processing, one binary. Roles are selected at startup; singleton roles self-elect. §7.
adminNext.js 15+ App Router—

One worker binary rather than three services is a deliberate simplification over the previous revision. The roles have genuinely different concurrency requirements — dispatch must be a singleton because of Orange's per-contract TPS cap, hooks wants to scale — but they share configuration, database pool, tracing setup, provider registry and deployment lifecycle. Splitting them into separate images buys isolation you don't need yet and costs you three deploy pipelines. Roles are already the unit of scaling, so when you do need hooks on its own node, you run the same image with --roles hooks and change nothing else.

2. Data model (schema.cstack) ​

2.0 Grammar and emitter constraints ​

Mechanical properties of the toolchain, each verified by running it — most against 0.4.16, two rows below marked 0.5.0 where the emitter's behaviour changed under us mid-project. None are obvious from the docs and each costs an afternoon if you find it the hard way.

ConstraintConsequence
Exactly one whitespace character between field name and type. parse_field splits on each whitespace char.No column alignment, ever. id Cuid fails with an invalid type reference error.
The parser is line-based.An @@allow cannot wrap. A wrapped continuation is parsed as a field and errors on the type.
enum and type bodies need one declaration per line, header ending in {.No single-line enum Foo { a b }.
@use(Mixin) is single-@. @@use(...) parses, is retained as an unknown model attribute, and silently does not expand the mixin.Never write @@use. There is no error.
Unknown attributes are silently ignored.A typo'd attribute is a no-op, not an error.
A misspelled @@allow action is silently dropped. With deny-by-default, the model becomes unreachable.Assert the generated policy set in a test.
@default contents are not validated; the value is spliced verbatim into DDL.Enum and string defaults must be single-quoted: @default('accepted'). Double quotes are a PG identifier. A typo'd variant reaches the DDL and fails at psql.
@default(cuid()) is not real. It parses, emits DEFAULT cuid() (no such PG function), and silently removes the field from the create input.Verified: no field 'id' on type CreateAppInput. Use @default(dbgenerated()) + a real SQL default.
@default(dbgenerated()) emits no DEFAULT clause — it only marks the field server-generated and drops it from create input.Hand-write ALTER COLUMN … SET DEFAULT …. §2.10.
Relation scalars must match the referenced field's type by name.Verified: Cuid ↔ Cuid passes; String FK → Cuid PK is a hard parse error.
Policy comparisons against auth().x also require type-name equality, including through relation traversal.Verified by compile: auth Principal { appId Cuid } vs WebhookEndpoint.appId Cuid passes; a String auth field is a compile error.
@authorize(M, action, args.x) requires name and arity match with M's PK.CancelInput.messageId must be Cuid.
A Cuid field cannot be compared to a literal in a policy — only to auth().x.We never do this.
Cuid is format-guarded only on REST query-string filters — [a-z0-9]{2,32}. Not on create, update, or DB decode.Ids must be lowercase alphanumeric with no prefix separator, or GET /messages?id=… returns 400.
There is no @@index. Only @unique emits an index.Every non-unique index is hand-written. §2.10.
Foreign keys are never emitted. No REFERENCES codepath exists in cratestack-migrate 0.4.16.Hand-written too.
No triggers, no CHECK without @db_enforce, no partial indexes.The state machines in §2.10 are entirely hand-written SQL. The schema declares the states; Postgres enforces the edges.
@db_enforce promotes @length/@range into a real CHECK — and is a silent no-op on @regex. Verified with an isolated two-field probe schema: identical @db_enforce on a @length field emits ADD CONSTRAINT … CHECK (length(...) BETWEEN …); on a @regex field it emits nothing, no error.Pattern constraints need a hand-written CHECK (col ~ '...') in §2.10. @regex alone is application-layer validation only.
0.5.0: enum-typed columns emit as TEXT NOT NULL + CHECK (col IN (...)), not a native CREATE TYPE ... AS ENUM. 0.4.16 emitted real enum types; nothing announced the change.Anything referencing an enum type name directly in hand-written SQL — our message_state/job_state transition-table columns — breaks silently at psql time (type "message_state" does not exist) on the next full regeneration. Now TEXT with a matching hand-written CHECK.
Scalar list fields (String[], Int[]) PANIC include_server_schema! — proc macro panicked: unsupported SQLx value type for this slice. The parser accepts them and the migration emitter happily writes TEXT[]; only the server macro fails.No model may declare a list field. Multi-values are delimited String columns (see below). Lists inside type blocks are fine — they never touch SQLx.
@version emits BIGINT NOT NULL with no default.Seeds and raw SQL fail without a hand-added default.
ANY @default(...) excludes the field from CreateXInput — literals included, not just dbgenerated(). is_generated_on_create is a bare starts_with("@default").Verified: CreateMessageInput has no field named 'priority' for priority Int @default(100). A @default on a caller-settable field is a bug. Keep it only where being unsettable is the point.
But defaulted fields ARE settable on UpdateXInput.So @default(auth().x) protects creation only; a PATCH can still overwrite it. That's the real footgun, not create.
@server_only excludes a field from create and update inputs.It is write-never through the framework. See R3 — secrets you must write cannot use it.
@server_only fields ARE readable server-side. The struct keeps them, FromRow reads them, SELECT includes them; only serde output and the fields= allowlist strip them (#[serde(skip_serializing, default)]).Worker and procedure code reads them through delegates normally.
@readonly and @server_only are identical for inputs and mutually exclusive (parser error if both). Neither may sit on @id.@readonly still serializes to responses; @server_only doesn't.
@pii / @sensitive redact audit snapshots ONLY. They do not strip from HTTP responses, traces or errors.A field marked @sensitive is still returned by GET /messages/{id}. Verified: hush still in json: true.
?sort=<server_only column> is not rejected — allowed_sorts has no @server_only filter, unlike allowed_fields.An ordering oracle over a column the caller can't read. Low impact here, but don't expose list routes on a model whose @server_only column is a real secret.
upsert does not exist when the @id has a @default.db.webhook_attempt().upsert(...) is a compile error. Dedupe is create + catching SQLSTATE 23505 (§8.3).
SKIP LOCKED / NOWAIT are not expressible. .for_update() is.§7.3 — the claim loops use optimistic CAS instead.
update_many / delete_many refuse to run with zero filters, and update_many has no if_match.Guard rail, not a limitation.
Nothing auto-touches updatedAt.A set_updated_at trigger in migration SQL (§2.10), not a field you remember to set.
No field-level read masking. Model-level access only.Field secrecy = model-level @@allow, or a separate model.
pluralize() is naive (ends_with('s') ? +"es" : +"s"), and there is no @@map.Every model name here pluralises cleanly. WebhookDelivery would have become webhook_deliverys — it's WebhookAttempt for that reason alone.

2.1 Header, enums, principal ​

cstack
datasource db {
  provider = "postgresql"
  url = env("DATABASE_URL")
}

enum MessageClass {
  otp
  transactional
  notification
  marketing
}

enum MessageState {
  accepted
  queued
  routed
  submitted
  delivered
  uncertain
  undelivered
  failed
  expired
  rejected
  cancelled
}

enum JobState {
  pending
  running
  succeeded
  failed
  dead
  cancelled
}

enum Encoding {
  gsm7
  ucs2
}

enum OperatorCode {
  mtn
  orange
  camtel
  nexttel
  unknown
}

enum ProviderKind {
  orange_cm_http
  mtn_http
  aggregator_http
  smpp
}

enum ProviderState {
  active
  degraded
  disabled
  draining
}

enum DeliveryOutcome {
  delivered
  uncertain
  failed
  expired
  rejected
  // A provider reported progress, not an outcome: the message is still in
  // flight and nothing about its fate has been decided. Orange sends two of
  // these (`DeliveredToNetwork`, `MessageWaiting`) on the happy path, so a
  // receipt row records them honestly while the message's own state is left
  // alone. Distinct from `unknown`, which means the adapter did not
  // recognise the status at all.
  in_flight
  unknown
}

enum AttemptState {
  pending
  delivering
  succeeded
  failed
  dead
}

enum OptOutSource {
  inbound_stop
  admin
  imported
  operator
}

// #72: how a ConsentRecord was captured.
enum ConsentChannel {
  web_form
  api
  ivr
  paper_form
  verbal
  sms_keyword
  imported
  admin
}

mixin Timestamps {
  createdAt DateTime @default(dbgenerated())
  updatedAt DateTime @default(dbgenerated())
}

auth Principal {
  sub String
  kind String
  role String
  appId Cuid
  messageRead String
  optOutRead String
}

The single auth block is the load-bearing design decision. CrateStack allows exactly one, and both humans and service accounts must fit it. kind discriminates: "user" principals carry a meaningful role and an empty appId; "app" principals carry role = "app" and a real appId. That one field gives per-app row scoping in SQL rather than in handler code:

cstack
@@allow("list", auth().kind == "user" || appId == auth().appId)

appId is Cuid, not String, because the type-name equality rule applies to policy comparisons as well as relations — and WebhookAttempt's policy traverses a relation to reach it (endpoint.appId == auth().appId), the form that enforces the check. A String auth field against a Cuid model field is a compile error, verified. sub stays String: it holds an OIDC subject or an OAuth client_id, neither of which is a model id.

hasRole('x') and inTenant('x') are the only two policy functions; anything else is a generation error. Both quote styles work.

role carries "system" for the internal principal. hasRole('system') reads the role claim, never kind — so the system context must set role = "system" or every @@allow("create", hasRole('system')) denies at runtime and nothing can write a message. Nothing in the toolchain checks this; it's a runtime failure that reads like a policy bug. Cover it with an integration test on the first send.

2.2 Applications and service accounts ​

There is no ApiKey model. Every machine caller is an AppClient holding at least one registered Ed25519 public key (ClientKey, §2.9); the gateway learns which app is calling by mapping the verified signer's AppClient.clientId back to an App. That mapping lives here rather than in the request because a signed request names a key, not an app (§4.0).

cstack
model App {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  name String @length(min: 2, max: 64)
  slug String @unique @regex("^[a-z0-9][a-z0-9-]{1,38}[a-z0-9]$")
  description String?
  defaultSenderIdId Cuid?
  defaultSender SenderId? @relation(fields: [defaultSenderIdId], references: [id])
  monthlyQuota Int
  ipAllowlist String
  transliterateToGsm7 Boolean
  active Boolean @default(true)
  deletedAt DateTime?

  clients AppClient[] @relation(fields: [id], references: [appId])
  messages Message[] @relation(fields: [id], references: [appId])

  @@soft_delete
  @@audit
  @@paged
  @@allow("read", hasRole('owner') || hasRole('admin') || hasRole('operator') || hasRole('auditor') || hasRole('developer'))
  @@allow("create", hasRole('owner') || hasRole('admin'))
  @@allow("update", hasRole('owner') || hasRole('admin'))
  @@allow("delete", hasRole('owner'))
  @@emit(created, updated, deleted)
}

model AppClient {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  appId Cuid
  app App @relation(fields: [appId], references: [id])
  clientId String @unique @length(min: 8, max: 64)
  label String @length(min: 1, max: 64)
  scopes String
  active Boolean @default(true)
  lastUsedAt DateTime?
  retiredAt DateTime?

  @@audit
  @@paged
  @@allow("read", hasRole('owner') || hasRole('admin') || hasRole('developer'))
  @@allow("update", hasRole('owner') || hasRole('admin'))
  @@allow("delete", hasRole('owner') || hasRole('admin'))
  @@emit(created, updated, deleted)
}

On delimited strings. No model here declares a list field, because String[] and Int[] panic the server macro outright — the parser accepts them and the migration emitter writes TEXT[], but include_server_schema! dies with unsupported SQLx value type for this slice. So every multi-value column is a space-delimited String, stored with leading and trailing separators: " sms:send sms:read ", not "sms:send sms:read".

The sentinel spaces are what make membership queries safe. .contains("sms:send") would also match a hypothetical sms:sendall; .contains(" sms:send ") cannot. sms-core owns pack(&[&str]) -> String and unpack(&str) -> Vec<&str> so no call site invents its own convention. For scopes this is barely a workaround — OAuth scope strings are space-delimited anyway, so the column and the convention agree.

AppClient exists so a caller's identity is separable from its credential. A client holds one or more ClientKey rows (§2.9), so key rotation needs no second identity: register the new key, cut the caller over, revoke the old one (both verify in between). Rotating by identity (provision a second client alongside the first, both mapped to the same App) stays available for the cases that want a clean audit boundary rather than a key swap.

Note what is deliberately absent: no secret and no hash — anywhere, for anyone. The only credential material in the database is the public half of a caller's keypair, in ClientKey.publicKey (and a person's in ConsoleKey.publicKey). Nothing here is a bearer credential that leaks by being read. (The OpenID Provider's tables — OauthClient, OauthSigningKey, ClientAssertion, UserCredential, SsoTransaction — were dropped by forward migration 0009_drop_openid_provider in VS-4, and are back since VSA-2 as 0012_restore_openid_provider: empty unless an operator imported rows by the runbook, and in any case read by nothing and reachable by nothing: on a deployment that did not import, "absent" above still describes every row a running system holds, and on one that did, the tables hold private signing keys and Argon2id hashes that nothing reads until VSA-3 and VSA-4. They are the one place a secret or a hash can live here again once VSA-3 and VSA-4 write to them; §2.8 has them.)

2.3 Sender IDs ​

Registration is per-channel, not national: Orange whitelists via a support form, MTN requires pre-registration through your aggregator, Africa's Talking wants a "Cameroon Local Fiche KYC". Every channel checked caps at 11 characters. So sender ID status is per-(sender, provider).

cstack
model SenderId {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  value String @unique @length(min: 3, max: 11) @db_enforce
  kind String
  notes String?
  active Boolean @default(false)

  registrations SenderIdRegistration[] @relation(fields: [id], references: [senderIdId])

  @@audit
  @@allow("read", auth().kind == "user" || auth().kind == "app")
  @@allow("create", hasRole('owner') || hasRole('admin') || hasRole('operator'))
  @@allow("update", hasRole('owner') || hasRole('admin') || hasRole('operator'))
  @@emit(created, updated)
}

model SenderIdRegistration {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  senderIdId Cuid
  senderId SenderId @relation(fields: [senderIdId], references: [id])
  providerId Cuid
  provider Provider @relation(fields: [providerId], references: [id])
  status String
  submittedAt DateTime?
  approvedAt DateTime?
  reference String?
  rejectionReason String?

  @@audit
  @@allow("read", auth().kind == "user")
  @@allow("create", hasRole('owner') || hasRole('admin') || hasRole('operator'))
  @@allow("update", hasRole('owner') || hasRole('admin') || hasRole('operator'))
}

@db_enforce promotes the @length into a real SQL CHECK. It's the one field where an out-of-band write putting a 40-character value in the table costs real money.

The router refuses to submit a message whose sender ID has no approved registration on the chosen provider — the check that stops you burning credit on messages the operator silently drops.

2.4 Providers and routing ​

cstack
model Provider {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  key String @unique @regex("^[a-z][a-z0-9_]{2,31}$")
  displayName String @length(min: 2, max: 64)
  kind ProviderKind
  state ProviderState @default('disabled')
  config String
  credentialRef String
  maxTps Float
  maxDailySubmissions Int
  supportsDlr Boolean
  supportsAlphaSender Boolean
  supportsUcs2 Boolean
  supportsConcat Boolean
  costPerSegmentXaf Decimal
  healthCheckedAt DateTime?
  healthy Boolean @default(false)
  // #63: the provider-side circuit breaker — same two-column shape as
  // WebhookEndpoint's own.
  consecutiveFailures Int @default(0)
  circuitOpenUntil DateTime?
  // #59.
  version Int @version

  routes Route[] @relation(fields: [id], references: [providerId])

  @@audit
  @@allow("read", hasRole('owner') || hasRole('admin') || hasRole('operator') || hasRole('auditor') || hasRole('system'))
  @@allow("create", hasRole('owner') || hasRole('admin'))
  @@allow("update", hasRole('owner') || hasRole('admin') || hasRole('operator') || hasRole('system'))
  @@allow("delete", hasRole('owner'))
  @@emit(created, updated, deleted)
}

model Route {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  name String @length(min: 2, max: 64)
  priority Int @range(min: 0, max: 1000)
  weight Int @range(min: 0, max: 1000)
  enabled Boolean
  matchOperator OperatorCode?
  matchClass MessageClass?
  matchAppId Cuid?
  matchPrefix String?
  providerId Cuid
  provider Provider @relation(fields: [providerId], references: [id])
  failoverRouteId Cuid?

  @@audit
  @@allow("read", hasRole('owner') || hasRole('admin') || hasRole('operator') || hasRole('auditor'))
  @@allow("create", hasRole('owner') || hasRole('admin'))
  @@allow("update", hasRole('owner') || hasRole('admin'))
  @@allow("delete", hasRole('owner') || hasRole('admin'))
  @@emit(created, updated, deleted)
}

config and credentialRef are plain columns, not @server_only, because an admin creating a provider has to write them and R3 rules that out. They're safe as plain fields for one reason: neither holds a secret. credentialRef is a pointer — vault://sms/providers/orange_cm, or an age-encrypted file key — which the worker resolves at startup and on SIGHUP. config holds non-secret settings. Provider credentials never enter this database at all, so no policy mistake can leak them.

That's the general shape of R3 in practice: when the framework won't let you hide a field, arrange for the field not to be worth hiding.

2.5 Message — the core ​

cstack
model Message {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  appId Cuid
  app App @relation(fields: [appId], references: [id])

  clientRef String?
  idempotencyKey String?

  msisdn String @pii @length(min: 8, max: 15)
  msisdnHash String
  operator OperatorCode

  senderIdValue String @length(min: 3, max: 11)
  class MessageClass
  priority Int @range(min: 0, max: 1000)

  body String? @sensitive
  bodyHash String
  bodyLength Int
  encoding Encoding
  segments Int @range(min: 1, max: 10)

  state MessageState @default('accepted')
  stateReason String?
  routeId Cuid?
  providerId Cuid?
  providerMessageRef String?
  providerMessageRefAlt String?

  attempts Int @default(0)
  maxAttempts Int
  leaseOwner String?
  leaseUntil DateTime?
  scheduledAt DateTime?
  expiresAt DateTime
  submittedAt DateTime?
  finalizedAt DateTime?

  // #67: stamped once, by purge_retention, past the 90-day retention
  // window below — see that job's own module doc for why an explicit
  // marker beats inferring "purged" from "body happens to be null".
  purgedAt DateTime?

  costXaf Decimal @default(0)
  version Int @version

  parts MessagePart[] @relation(fields: [id], references: [messageId])
  receipts DeliveryReceipt[] @relation(fields: [id], references: [messageId])

  @@audit
  @@paged
  @@retain(days: 90)
  @@allow("list", auth().kind == "user" || appId == auth().appId || hasRole('system'))
  @@allow("detail", auth().kind == "user" || appId == auth().appId || hasRole('system'))
  @@allow("create", hasRole('system'))
  @@allow("update", hasRole('system'))
  @@emit(created, updated)
}

Eight things worth defending:

body is @sensitive, and is stored in plaintext for every class, OTP included. @sensitive redacts it in audit snapshots only — verified: it adds no serde attribute, so GET /messages/{id} still returns the body to a principal that passes the detail policy. So @sensitive is not a confidentiality control, and there is no second control behind it: the body is retained for the model's full @@retain(days: 90) window.

An earlier revision of this paragraph claimed that for class = otp the send procedure set body = null, keeping only bodyHash/bodyLength/segments, justified as "an OTP gateway that stores OTP plaintext for 90 days is a credential database." That was never implemented, and — as written — is not implementable (#165). sms-api and sms-worker are separate OS processes coordinating through nothing but the Message row (no broker, no Redis; see §1). backends/crates/sms-worker/src/dispatch.rs reads message.body off the row claim.rs's candidates() fetched, and fails the message outright when it is NULL. Nulling at creation would therefore fail every OTP send, and break undelivered -> queued retries (§7.4, #122). Anyone reimplementing that paragraph as it stood reproduces that outage.

Redacting later — at a terminal state, once nothing can need the value again — was built and rejected on the merits (#183, closed). Two reasons. The credential argument does not survive contact with the validity window: OTP validity defaults to 15 minutes (§7.4's default_validity), so a code is dead long before retention is relevant; it is not a credential by the time it is retained. And the linkage argument is already lost elsewhere in the same row — senderIdValue and appId sit beside a plaintext msisdn, so "this number has an account with this brand" is present whether or not body is. What redaction removed was a spent code and some template wording, against a real cost in support visibility on exactly the messages subscribers complain about.

What genuinely remains is data minimisation under Law No. 2024/017 — holding content with no remaining purpose, independent of whether anyone could exploit it. That is a legal question, not an engineering one, and it is already before counsel in docs/legal/retention-briefing.md. If the answer is that OTP content retention changes their conclusions, the fix at that point is uniform, shorter body retention (§10, #67, #5) rather than an OTP-only special case — non-OTP bodies sit for the same 90 days regardless.

state MessageState @default('accepted') keeps its default on purpose. Everywhere else a @default on a caller-settable field is a bug (§2.0), but here being unsettable is the control: because any @default excludes the field from CreateMessageInput, no caller can create a message that is already submitted or delivered. attempts and costXaf keep their defaults for the same reason. operator, class, priority and maxAttempts lost theirs, because the send procedure computes all four per message and a default would have made them unwritable.

msisdn is @pii and msisdnHash is HMAC-SHA256 of the E.164 form under a server-held pepper (security.hash_pepper, delivered as ${SMS_HASH_PEPPER}, §9.4; sms_api::pepper, #134 — until then this was a plain unkeyed SHA-256, reversible in seconds over Cameroon's ~10^7-candidate mobile numbering space, which is exactly why msisdnHash needed to be keyed, not just hashed). Stored as hmac-sha256-v1:<hex> — the scheme tag distinguishes it from the pre-#134 unkeyed sha256: values and from any future pepper-rotation scheme (hmac-sha256-v2:). bodyHash is peppered under the same scheme and the same pepper, for the same reason: a templated OTP body is exactly as low-entropy and enumerable as an MSISDN, and nothing in this system or the admin console ever reads bodyHash back to cross-compare it against a value computed elsewhere, so there is no external contract keying it could break. Analytics, dedupe, rate limiting and opt-out matching all go through msisdnHash. Under Law No. 2024/017 that's the difference between a purge being one UPDATE and a forensic exercise — but only once the hash is genuinely unkeyed-reversal-proof, which unkeyed SHA-256 never was. Rotating the pepper invalidates every hash computed under the old one; see sms_api::pepper's module doc for the consequence (no rehash job exists yet, and a row whose msisdn has already been purged can never be rehashed at all).

version Int @version gives optimistic locking, and it isn't optional — the worker, the DLR receiver and an admin cancel all race the same row. You get ETag on GET, a required If-Match on PATCH/DELETE, 412 on stale or missing, and version = version + 1 folded into the same statement. It rejects a missing If-Match before SQL runs, so the admin must thread the ETag through every edit form.

leaseOwner + leaseUntil make the claim recoverable. A worker that dies mid-submit leaves a row in routed with an expired lease; the next claim cycle reclaims it. Without a lease, a crash strands the message until a human notices.

uncertain is a first-class state, not a synonym for failure. Orange returns DeliveryUncertain specifically on multi-network handoff — precisely the Orange→MTN case you care about. Treating it as failure makes OTP retry logic double-send. uncertain resolves on a later DLR, or ages out to expired.

providerMessageRef has an Alt twin for the SMPP hex/decimal trap in §6.2.

No @@allow for delete — messages are never deleted through the API. @@retain(days: 90) declares intent; the purge is a job.

@@allow("create", hasRole('system')) closes generic REST creation. Only sendMessage creates messages, through a delegate bound to a system context. That's what protects appId from the @default(auth().x) override footgun.

The state column is declared here but its transitions are enforced in Postgres — see §2.10 and §7.4.

cstack
model MessagePart {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  messageId Cuid
  message Message @relation(fields: [messageId], references: [id])
  partIndex Int @range(min: 0, max: 9)
  udhRef Int?
  providerPartRef String?
  state MessageState @default('queued')
  submittedAt DateTime?
  deliveredAt DateTime?

  @@paged
  @@allow("list", auth().kind == "user")
  @@allow("detail", auth().kind == "user")
  @@allow("create", hasRole('system'))
  @@allow("update", hasRole('system'))
}

model DeliveryReceipt {
  id Cuid @id @default(dbgenerated())
  messageId Cuid
  message Message @relation(fields: [messageId], references: [id])
  providerId Cuid
  providerMessageRef String
  outcome DeliveryOutcome
  rawStatus String
  errorCode String?
  networkCode OperatorCode
  receivedAt DateTime @default(dbgenerated())
  occurredAt DateTime?
  rawPayload String

  @@audit
  @@paged
  @@retain(days: 90)
  @@allow("list", auth().kind == "user" || hasRole('system'))
  @@allow("detail", auth().kind == "user" || hasRole('system'))
  @@allow("create", hasRole('system'))
  // #67: purge_retention deletes a receipt past its own 90-day @@retain
  // window — the first thing that ever removes one. Independent of
  // Message's own retention: eligibility is the receipt's own receivedAt.
  @@allow("delete", hasRole('system'))
}

Receipts were append-only until #67: rawPayload is a plain column rather than @server_only — the DLR handler has to write it (R3) — and read access is confined by the model policy, which admits only kind == "user" principals and, as of #67, a system context for the purge job. When a provider changes its DLR format without telling you, and they do, the raw payloads are the only way to reconstruct what happened — for 90 days; purge_retention is what ends that window rather than leaving the table to grow forever.

2.6 The job queue ​

The worker's generic background queue. Everything that isn't message dispatch or webhook delivery runs through here: retention purges, balance polls, health probes, audit anchoring, client reconciliation, secret cleanup.

cstack
model Job {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  kind String @length(min: 2, max: 64)
  dedupeKey String?
  payload String
  state JobState @default('pending')
  priority Int @range(min: 0, max: 1000)
  runAt DateTime
  leaseOwner String?
  leaseUntil DateTime?
  attempts Int @default(0)
  maxAttempts Int
  lastError String?
  startedAt DateTime?
  finishedAt DateTime?
  version Int @version

  @@paged
  @@retain(days: 14)
  @@allow("list", hasRole('owner') || hasRole('admin') || hasRole('operator'))
  @@allow("detail", hasRole('owner') || hasRole('admin') || hasRole('operator'))
  @@allow("create", hasRole('owner') || hasRole('admin') || hasRole('system'))
  @@allow("update", hasRole('system'))
}

dedupeKey with a partial unique index on non-terminal states is what makes "enqueue the nightly purge" idempotent — the scheduler can fire twice without producing two purges. Admins get read access so the queue is visible in the console, and create so an operator can hand-enqueue a re-run, but update is system-only: nobody drives a job's state from the API.

runAt, priority and maxAttempts carry no @default precisely because the scheduler sets them — a @default(dbgenerated()) on runAt would have made scheduling a job for later impossible, since the field would be absent from CreateJobInput. state keeps its default so nothing can be enqueued already running.

Like Message, the transitions are enforced by a Postgres trigger, not by the schema.

cstack
model OptOut {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  msisdnHash String @unique
  msisdn String @pii
  source OptOutSource
  scope String
  reason String?
  optedOutAt DateTime

  @@audit
  @@paged
  @@allow("read", auth().optOutRead == "granted" || hasRole('system'))
  @@allow("create", hasRole('owner') || hasRole('admin') || hasRole('operator') || hasRole('system'))
  @@allow("delete", hasRole('owner') || hasRole('admin'))
  @@emit(created, deleted)
}

// #72: a standing grant of consent — who, when, through what channel, to
// what scope. Not the enforcement primitive (OptOut is — its presence is
// what blocks a send); this model is evidence a caller can point to when
// the classification or the consent itself is challenged. Withdrawing
// consent is a new OptOut row, not a field here, so "does this recipient
// currently receive X" never has two disagreeing sources of truth.
model ConsentRecord {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  appId Cuid
  app App @relation(fields: [appId], references: [id])

  msisdnHash String
  msisdn String @pii

  scope MessageClass
  channel ConsentChannel
  consentedAt DateTime
  evidenceRef String?

  @@audit
  @@paged
  @@allow("read", (auth().kind == "user" && auth().optOutRead == "granted") || appId == auth().appId || hasRole('system'))
  @@allow("create", hasRole('owner') || hasRole('admin') || hasRole('operator') || hasRole('system'))
  @@allow("delete", hasRole('owner') || hasRole('admin'))
  @@emit(created, deleted)
}

enum WebhookDeliveryMode {
  sealed
  hmac
}

model WebhookEndpoint {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  appId Cuid
  app App @relation(fields: [appId], references: [id])
  url String @uri                          // any URL; a `sealed` endpoint's is also held to a route rule, below
  deliveryMode WebhookDeliveryMode         // sealed | hmac
  audience String? @unique @length(min: 1, max: 255) @db_enforce @regex("...")  // never `vsms`; required for sealed
  eventTypes String
  secret String? @sensitive @length(min: 32) // hmac only
  prevSecret String? @sensitive            // hmac only, while a rotation's overlap is open
  secretRotatedAt DateTime?                // hmac only
  maskRecipient Boolean
  active Boolean @default(true)
  maxAttempts Int
  circuitOpenUntil DateTime?
  consecutiveFailures Int @default(0)

  attempts WebhookAttempt[] @relation(fields: [id], references: [endpointId])

  @@audit
  @@allow("read", hasRole('owner') || hasRole('admin') || hasRole('developer') || hasRole('system'))
  @@allow("create", hasRole('owner') || hasRole('admin') || hasRole('developer'))
  @@allow("update", hasRole('owner') || hasRole('admin') || hasRole('developer') || hasRole('system'))
  @@allow("delete", hasRole('owner') || hasRole('admin'))
  @@emit(created, updated, deleted)
}

model WebhookAttempt {
  id Cuid @id @default(dbgenerated())
  endpointId Cuid
  endpoint WebhookEndpoint @relation(fields: [endpointId], references: [id])
  sourceEventId Uuid
  aggregateId Cuid
  eventType String
  payload String
  state AttemptState @default('pending')
  attempts Int @default(0)
  leaseOwner String?
  leaseUntil DateTime?
  nextAttemptAt DateTime?
  lastStatusCode Int?
  lastError String?
  lastAttemptAt DateTime?
  deliveredAt DateTime?
  version Int @version

  @@paged
  @@retain(days: 30)
  @@allow("list", auth().kind == "user" || endpoint.appId == auth().appId || hasRole('system'))
  @@allow("detail", auth().kind == "user" || endpoint.appId == auth().appId || hasRole('system'))
  @@allow("create", hasRole('system'))
  @@allow("update", hasRole('system'))
}

ConsentRecord is appId-scoped; OptOut is not. An opt-out is a recipient's own standing decision and has to bind everywhere in this deployment — an inbound STOP has to suppress every app, not just the one whose message triggered it. A consent grant is the opposite: it is a specific app's own customer relationship (its own web form, its own marketing program), and there is no reason a grant captured for one app's marketing should authorise another's. scope MessageClass reuses the existing enum rather than a second free-form String the way OptOut.scope is — sendMessage's consent check (backends/crates/sms-api/src/procedures.rs::ensure_consent_on_file) matches this column against Message.class directly, so a typo'd scope string can never silently stop matching the class it describes.

No @@retain on ConsentRecord, and this is a real, unresolved tension. Message purges msisdn at 90 days because minimisation applies to content with no remaining purpose — but proof of consent is not that; it is the thing you produce because you were challenged, which argues for keeping it at least as long as the relationship (and probably a limitation-period buffer) lasts. Keeping msisdn in plaintext indefinitely to serve that need reproduces the exact problem #67 closed for Message. Not resolved here — see AGENTS.md's open-questions entry for #72. msisdnHash — what ensure_consent_on_file actually matches on — inherits the same pepper-rotation caveat as every other msisdnHash column in this schema (§2.5, sms_api::pepper's module doc): rotating the pepper (security.hash_pepper, §9.4) orphans every row written under the old one.

Withdrawing consent is a new OptOut row, not a field on this model. There is deliberately no revokedAt — OptOut's presence is already the single enforcement primitive sendMessage checks, and giving ConsentRecord its own independent revocation flag would create two sources of truth for "does this recipient currently receive X" that could disagree. This model is append-only evidence of what was granted and when; whether it still holds is answered by OptOut's absence.

What sendMessage's classification exemption actually proves, and what it does not — read before treating "audit trail" as solved. §10's own text says "keep the audit trail proving the classification." Message.class already carries @@audit, so every send writes a framework audit row, in the same transaction as the Message insert, capturing the declared class and the authenticated caller's identity — a real, contemporaneous record of who declared what, when. It does not prove the declaration was true: nothing here independently classifies a message body as marketing versus transactional, so a caller willing to mislabel traffic bypasses every check backends/crates/sms-api/src/consent.rs implements, and the audit trail records that they did so accurately, not honestly. See that module's own doc for the full reasoning — this is the honest boundary of what "the audit trail proves the classification" means in this codebase.

There is no OutboxEvent model. @@emit makes it redundant: CrateStack writes to cratestack_event_outbox inside the mutation's transaction (§8). WebhookAttempt is what a subscriber creates from that event, and it's the durable unit the hooks role retries against.

Two fields exist purely for correctness under the framework's delivery semantics, and §8.3 explains why both are mandatory:

  • sourceEventId Uuid is CrateStack's event_id, carried for tracing and duplicate diagnosis.
  • aggregateId + eventType form the real dedupe key, with a unique index on (endpoint_id, aggregate_id, event_type). Not sourceEventId — a Message.updated event fires on every update, including ones that don't change state, each with a fresh event_id. Keying on aggregate + derived type makes message.delivered fire exactly once per message per endpoint, no matter how many times the row is touched or how many workers drain concurrently.

The model is WebhookAttempt, not WebhookDelivery, solely because pluralize() would have produced webhook_deliverys and there is no @@map.

WebhookEndpoint is readable by humans only — the appId == auth().appId clause is gone. That's a direct consequence of R3: an hmac endpoint's secret and prevSecret must be writable by rotateWebhookSecret, so they can't be @server_only, so they'd be returned to any principal that can read the model. Rather than hand an app an endpoint object with a live signing secret in it, endpoints are configured by your developers in the admin console and apps never read them over the API. Apps still see their own WebhookAttempt rows, because that policy reaches appId through a relation traversal that never materialises the endpoint row.

deliveryMode (VS-8R) says how the endpoint's deliveries are authenticated, sealed or hmac, and both are always available (§4.4). It is required and has no default: a default would drop it from the create input, and an endpoint is created knowing which receiver contract it serves. The migration set every existing endpoint to hmac, because their receivers verify an HMAC today. The database holds the modes apart with CHECKs (0010_sealed_webhooks): a sealed endpoint has an audience and no secret, prevSecret or secretRotatedAt; an hmac endpoint has a secret. A write that breaks one is refused with SQLSTATE 23514 naming the constraint. The generated routes do not pass their errors through map_database_error, so a raw PATCH that breaks one is answered as a database error (500), and the console checks the same rules before it asks. Switching a mode is the switchWebhookDeliveryMode procedure (docs/integrating.md, "Choosing a delivery mode"): it validates first and answers 422, mints the hmac secret server-side, clears the secret columns and requires an audience for sealed, and a 23514 on one of these constraints that reaches a procedure is mapped to a 422 that names the rule. The console offers no switch; it fixes the mode at creation.

audience (VS-8) is a sealed endpoint's identity: the name the receiver's envelope layer is built with and the audience every sealed delivery's seal binds (cratestack ADR 0006 §4). It is required for sealed, 1 to 255 characters, and has no default: a default would silently mean "any receiver", and a seal binds exactly one recipient. It is null for hmac, which has no seal to address: the CHECK webhook_endpoints_audience_only_for_sealed_check (delivery_mode <> 'hmac' OR audience IS NULL) refuses an hmac row that names one, so no hmac endpoint can occupy a name a real sealed endpoint may need (the unique index holds the audience across both modes). switchWebhookDeliveryMode clears it going to hmac, and answers a 422 for it like for every other CHECK; a raw generated PATCH that breaks it is answered 500, as for the others, because nothing maps a generated route's CHECK violation. It is set when the endpoint is created and editable like url.

The audience is unique, and never vsms. One gateway key signs every app's deliveries, so the audience is what tells one receiver's seals from another's; two endpoints sharing a name would let whoever holds webhook:manage create an endpoint that names a victim's audience at a URL of their own, collect seals the victim's receiver would open, and replay them. audience is @unique (a unique index in 0010_sealed_webhooks, so a second endpoint, of any app, that asks for a taken name is refused; Postgres allows any number of NULLs, so hmac endpoints do not collide), and vsms (the name the gateway's own routes answer to) is refused by a @regex at the API and a CHECK in the database. A unique audience is a registry of names, not proof of who should hold one, so receivers choose a name only they can have (their own origin) and bind on the delivery's endpointId and appId (§4.4): the audience is defence in depth, not the binding. The migration backfills nothing: an earlier draft derived an audience from each URL's origin for every existing row, and an hmac row needs none while a guessed one would occupy a name a real endpoint may need.

url must be one a delivery can be sealed for, if the endpoint is sealed. A sealed delivery binds the path of the URL as its route (§4.4), so a URL with credentials, a query, a fragment, a {, } or % in its path, or a scheme other than http(s) could only fail at delivery, burn the endpoint's attempts and trip its breaker. An hmac delivery binds no route, and the URLs HMAC receivers were registered with (a ?token= query, credentials, a % in the path) are legitimate and must keep working and keep being editable, so url stays a plain @uri and the route rule is a hand-written CHECK on sealed rows alone, webhook_endpoints_url_sealable_for_sealed_check (delivery_mode <> 'sealed' OR url ~ '<pattern>', in 0010_sealed_webhooks). The pattern is sms_webhook::SEALABLE_URL_PATTERN, written to mean the same to Rust's regex and to Postgres, and sealed_webhooks_migration_live_postgres.rs asserts the migration carries exactly that constant. It is narrower than sms_webhook::delivery_route in what it reads as text (a host is ASCII, so an internationalised host is written in punycode, and an underscore in a host name is refused; a port is 1 to 65535) and wider in what it cannot see: it is a pattern, not a parser, so it lets [:], 1.2.3.999 and x.0 through, which URL parsing refuses. What holds is that the pattern together with @uri (the Url::parse every write through the API passes) never accepts what delivery_route refuses, and sms-api's webhook_endpoint_validation.rs holds that over a generated corpus with delivery_route as the oracle. A raw SQL write that skips @uri and names one of those shapes is stored and then fails at delivery as an ordinary failure, which is why the console asks both (urlCanBeSealed in frontends/packages/gateway/src/webhook-rules.ts, the pattern ported to TypeScript and held to the Rust constant by reading it, and to sms-webhook's tests/fixtures/sealable_url_cases.json, which the Rust side reads too). Because the rule depends on the mode, it cannot be a field attribute: a raw PATCH or POST that breaks it is refused by the database (23514, 500 through the generated routes), switchWebhookDeliveryMode answers 422 before the write, and the console's router and forms apply the same rule to sealed endpoints only.

secret is at least 32 characters when written through the API (@length; an empty or short one would sign with a key anyone can guess). The format beyond that is the minter's: the console, rotateWebhookSecret and switchWebhookDeliveryMode generate whsec_ and 64 hex characters, and an API client that supplies its own on create is free to choose any value of that length. It is not a CHECK, so a secret an app set before the bound existed keeps working and keeps rotating; prevSecret, which only ever receives an earlier secret, has no bound for the same reason.

#187: read is scoped to owner/admin/developer (plus system), not every human role. The clause used to be auth().kind == "user" — any authenticated human, including auditor/operator/support, none of which hold any webhook permission (§5.2's own table). @sensitive never mitigated this: it redacts audit snapshots only and adds no serde attribute, so an hmac endpoint's secret/prevSecret are still returned by the API either way. Narrowed to match who can already write, on the reasoning that an auditor's oversight need is served by seeing that a secret exists, was rotated, and when — every other field on this model, still fully readable — not by seeing the secret's own value: a signing secret is a credential, not a record. VS-8R makes both columns @sensitive, so the @@audit snapshots of every create and update redact them ([redacted-sensitive]); before that they held every HMAC secret the table ever carried in plaintext, and the anchor chain anchor_audit writes over those snapshots exists so that history cannot be rewritten. Rotate each hmac endpoint's secret once after the migration so the secret in use is not one the history holds.

2.8 RBAC tables ​

cstack
model User {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  subject String @unique
  email String @unique @email
  displayName String @length(min: 1, max: 128)
  roleKey String
  role Role @relation(fields: [roleKey], references: [key])
  active Boolean @default(true)
  lastLoginAt DateTime?
  mfaEnrolled Boolean @default(false)
  deletedAt DateTime?

  @@soft_delete
  @@audit
  @@paged
  @@allow("read", hasRole('owner') || hasRole('admin') || hasRole('auditor'))
  @@allow("create", hasRole('owner') || (hasRole('admin') && roleKey != 'owner'))
  @@allow("update", hasRole('owner') || (hasRole('admin') && roleKey != 'owner'))
  @@allow("delete", hasRole('owner'))
  @@emit(created, updated, deleted)
}

model Role {
  @use(Timestamps)

  id Cuid @id @default(dbgenerated())
  key String @unique @regex("^[a-z][a-z0-9_]{2,31}$")
  label String @length(min: 2, max: 64)
  description String?
  builtin Boolean @default(false)
  permissions String

  @@audit
  @@allow("read", auth().kind == "user")
  @@allow("create", hasRole('owner'))
  @@allow("update", hasRole('owner'))
  @@allow("delete", hasRole('owner'))
  @@emit(created, updated, deleted)
}

User.role references Role.key, not Role.id — both String, so the type rule holds, and it keeps the principal-facing identifier readable in the database. Role.permissions is a delimited String rather than a join table: the set is small, it's read on every principal lookup (cached for 60 seconds), and unpack() produces exactly the array that goes into the principal's perms. A join table would cost a query on the request hot path and buy nothing.

FederatedLink (console pairing, §4.3) — hasRole('system') on every action it has, shipped as forward migration 0005_sso_federation (not a regenerated 0001_init: that file has been applied to deployed databases and sms-migrate records migrations by name; cargo xtask migrations-current knows the table is carried by 0005, and fails if 0005's copy of the generator's statements drifts from schemas/vsms.cstack). The Keycloak single sign-on callback wrote it until VS-4 deleted that, and writes it again since VSA-4c (issuer the identity provider's, verbatim); console pairing (registerConsoleKey) writes it too, under the issuer vaam, which is not a URL and cannot be an identity provider's. SsoTransaction, the in-flight sign-in table 0005 shipped beside it, was dropped by 0009_drop_openid_provider and is one of the five 0012 restores (below).

  • FederatedLink(issuer, subject, userId) with a hand-written UNIQUE (issuer, subject) — the identity key; console pairing finds or provisions a User through (issuer = "vaam", subject = <the person's id at vaam>). It is not User.subject: that column is vsms's own subject (the row's id, written by provision_user.rs and the JIT path), while an upstream sub is unique only within its issuer and chosen by someone else; putting it in User.subject would give one column two meanings and let an upstream-chosen string collide with a local id-shaped subject. Never email. @@audit; no update or delete action (unlinking is deliberately not an operation yet).

The OpenID Provider and human-login models, restored empty (VSA-2). Owner ruling R5 (2026-10-27): which ways in a deployment permits is configuration (auth.methods, §4.9), and the default keeps every legacy way, machine tokens and human login among them. VS-4 deleted the state those two keep in the database; 0012_restore_openid_provider creates it again, persistence only, until the code that uses them came back. VSA-3a (§4.10) reads and writes oauth_clients, oauth_signing_keys and client_assertions for the OpenID Provider; user_credentials is read and written since VSA-4a by the password login and provision-user --password (§4.10), and sso_transactions is read and written since VSA-4c by the single-sign-on broker (§4.10). No route serves any of the five.

Model (table)HoldsPolicyNotes
OauthClient (oauth_clients)the OP's client registry: private_key_jwt machine clients and the public (PKCE) console clienthasRole('system') on read, create, update; @@audittokenEndpointAuthMethod (ClientAuthMethod) has no @default, on purpose: a defaulted field leaves the create input, and a missing method is how the OP's library spells "accepts a secret", the state private_key_jwt must never reach (create_inputs.rs turns its disappearance into a compile error). Two hand-written CHECKs (§2.10) keep the method and the rest of the row in agreement. appClientId is a plain NO ACTION foreign key.
OauthSigningKey (oauth_signing_keys)the OP's RS256 signing keys, private halves (privateKeyPem, @sensitive)hasRole('system') on read, create, update; @@audit@sensitive redacts audit snapshots only: the policy and @@internal("all") are the whole control over the key.
ClientAssertion (client_assertions)spent private_key_jwt jtishasRole('system') on read, create, deleteinsert-only: create plus catching 23505 on the @unique jti is the atomic "record if unseen".
UserCredential (user_credentials)Argon2id password hashes, one per User (userId @unique, plain NO ACTION foreign key)hasRole('system') on every actionseparate from User because the framework has no field-level read masking.
SsoTransaction (sso_transactions)in-flight Keycloak sign-ins: the hashed state, the nonce and PKCE codeVerifier (@sensitive)hasRole('system') on read, create, deletesingle use, consumed by delete; no updatedAt. Written by GET /login/sso/start and swept by the gateway's reaper since VSA-4c (§4.10).

What is exposed, exactly. All five carry @@internal("all"), which removes their generated REST routes, RPC arms and client stubs: sms-gateway routes prints the same 139 routes as before, the generated TypeScript client has no method for them, and every op's contract digest (cratestack contract digest) is byte-identical to before, so no signed client is invalidated. What remains is the generated Rust delegate (db.oauth_signing_key() and the rest), which exists for in-process callers and is gated by the policies above: system is constructed only inside a process (sms_api::auth::system_context), and GatewayAuth never produces it for a request. tests/provider_models_are_unreachable.rs holds both locks without a database (no route in route_table(), @@internal("all") on each model, no @@allow but hasRole('system')); the OP reaches three of them through those delegates (VSA-3a), and VSA-3/4 take @@internal("all") off a model only if a route for it is ever wanted, and none was before VS-4. schemas/vsms.cstack and the Rust SDK's vendored copy stay identical (cargo xtask sdk-schema-check); vsms has no contracts lock file, so none moves.

The migration. 0009 stays as written (migrations are forward-only, and sms-migrate records them by name). 0012 runs after it and creates the five tables with the shape they last had: the generator's statements for them (0001_init's four tables and 0005's sso_transactions) plus what 0002_bootstrap and 0005 added by hand (the cs_cuid() and now() defaults, the touch_updated_at triggers, the list defaults, the signing-key and expiry indexes and the two registration CHECKs of §2.10). A fresh database therefore creates them in 0001/0002/0005, drops them in 0009 and creates them again in 0012; restore_openid_provider_live_postgres.rs compares the result, catalog against catalog, with a database that stopped at 0008. cargo xtask migrations-current knows the pair: the 0012 entry of ADDED_BY_LATER takes the five tables' statements out of the generator's output (and requires them verbatim in 0012), DROPPED_BY_LATER takes the same five out of the frozen 0001_init, both read one list (OPENID_PROVIDER_TABLES), and the check refuses a restoring migration that sorts before the dropping one or drops its own tables. The tables are empty unless an operator imported rows by the runbook (then they hold private keys and hashes that nothing reads until VSA-3 and VSA-4): 0009 is in no release tag (the latest, v0.9.0, predates it), so every deployment on v0.9.0 or earlier runs 0009 and 0012 together on its next upgrade and loses its OAuth clients, signing keys, password hashes and in-flight sign-ins, and docs/runbooks/deployment.adoc, "Keeping the OpenID Provider's rows across 0009 and 0012", is how to export the first three before the upgrade and import them after. 0012's down.sql refuses while any of the five holds a row.

2.9 Procedures ​

cstack
type SendMessageInput {
  to String
  body String
  senderId String?
  class MessageClass?
  clientRef String?
  scheduledAt DateTime?
  validityMinutes Int?
}

type SendMessageResult {
  messageId Cuid
  state MessageState
  encoding Encoding
  segments Int
  operator OperatorCode
  estimatedCostXaf Decimal
}

type PreviewInput {
  body String
  to String?
}

type PreviewResult {
  encoding Encoding
  segments Int
  length Int
  perSegment Int
  offending String[]
  suggestion String?
  operator OperatorCode
  normalizedTo String?
}

type ProvisionClientInput {
  appId Cuid
  label String
  scopes String[]
}

type ProvisionClientResult {
  clientId String
  clientSecret String
}

type CancelInput {
  messageId Cuid
  reason String?
}

type EndpointInput {
  endpointId Cuid
}

type ReplayWebhookAttemptInput {
  attemptId Cuid
}

type EnqueueJobInput {
  kind String
  payload String
  dedupeKey String?
  runAt DateTime?
}

mutation procedure sendMessage(args: SendMessageInput): SendMessageResult
  @allow(auth().kind == "app" || hasRole('owner') || hasRole('admin') || hasRole('operator'))
  @isolation("read_committed")

procedure previewMessage(args: PreviewInput): PreviewResult
  @allow(auth() != null)

procedure listMessagesPage(appId: Cuid?, state: MessageState?, limit: Int?, offset: Int?): Page<Message>
  @allow(auth() != null)

mutation procedure cancelMessage(args: CancelInput): Message
  @allow(hasRole('owner') || hasRole('admin') || hasRole('operator'))
  @authorize(Message, detail, args.messageId)
  @isolation("serializable")

mutation procedure enqueueJob(args: EnqueueJobInput): Job
  @allow(hasRole('owner') || hasRole('admin'))
  @isolation("read_committed")

mutation procedure provisionAppClient(args: ProvisionClientInput): ProvisionClientResult
  @allow(hasRole('owner') || hasRole('admin'))
  @isolation("serializable")

// VS-8R: rotate an `hmac` endpoint's signing secret with an overlap window.
// A `sealed` endpoint has no secret and is refused (422).
mutation procedure rotateWebhookSecret(args: EndpointInput): WebhookEndpoint
  @allow(hasRole('owner') || hasRole('admin') || hasRole('developer'))
  @isolation("serializable")

// VS-8R: change an endpoint's delivery mode, validated (422 where a raw PATCH
// would break a CHECK), minting the hmac secret or requiring the sealed audience.
mutation procedure switchWebhookDeliveryMode(args: SwitchWebhookDeliveryModeInput): WebhookEndpoint
  @allow(hasRole('owner') || hasRole('admin') || hasRole('developer'))
  @isolation("serializable")

// #43: re-fire a stuck WebhookAttempt from the admin surface. See §8.5's
// own "Implementation, #43" note for the full design — the state-machine
// edges it needs, why `succeeded` stays out of reach, and the circuit-
// breaker decision.
mutation procedure replayWebhookAttempt(args: ReplayWebhookAttemptInput): WebhookAttempt
  @allow(hasRole('owner') || hasRole('admin') || hasRole('developer'))
  @authorize(WebhookAttempt, detail, args.attemptId)
  @isolation("serializable")

@authorize(Message, detail, args.messageId) requires the arg-path type to match Message.id in name and arity — Cuid/required on both sides, verified by compile. @allow, @authorize and @isolation can all sit on one procedure; order is unconstrained.

@isolation("serializable") on cancelMessage matters: cancel races the worker's claim of the same row. serializable plus @version plus the transition trigger turns that race into a clean retry rather than a message that gets cancelled and submitted. cratestack-sqlx wraps it in run_in_isolated_tx_with_retries.

provisionAppClient(appId, label, scopes, publicKey) is the HTTP path that creates an AppClient: it makes the client (appc_<uuid>) and registers the caller's Ed25519 public key for it in one serializable transaction, returning {clientId, keyId, thumbprint}. Nothing secret exists to return: the caller generated the keypair and vsms never sees the private half (it used to generate an RSA keypair server-side and return the private key once, for the token server VS-4 deleted). A thumbprint already registered to any client is a 409, and console:pair is refused for every HTTP caller (§4.2).

ClientKey (signed transport, VS-2) — an app client's request-signing public key, and the credential of the move from OAuth client_credentials to COSE_Sign1-signed requests, which VS-4 completed. Since VS-3 the envelope layer verifies against a non-revoked ClientKey (§4.0, "The signed transport") and the row is the only credential the API accepts from an app.

  • ClientKey(appClientId, alg, publicKey, thumbprint, label, revokedAt), shipped as forward migration 0006_client_keys by the same mechanism as 0005 (cargo xtask migrations-current carries a (directory, tables) entry per forward migration). alg is ed25519 only — cratestack-cose signs with Ed25519 and ESP256, never RSA, so every caller registers an Ed25519 key. publicKey is the raw 32-byte point (CHECK in 0006); the private half is generated by the caller and never reaches vsms, so there is nothing to return once.
  • thumbprint is the RFC 9679 COSE key thumbprint, lowercase hex, computed by cratestack::cose and never taken from the caller. @unique: the key's identity, so one public key cannot be two rows or belong to two clients, and the request's 8-byte kid is its first 16 hex characters.
  • Revoke, never delete. revokedAt set means refused; the row stays so a revoked key is never re-registered as new. There is no delete action.
  • Policy. create and update are hasRole('system') only; read admits owner, admin and system. The generated POST /client_keys therefore cannot plant a key (it would skip the thumbprint and key validation). registerAppClientKey(appClientId, publicKey, label?) and revokeAppClientKey(keyId), each @allow(owner || admin), are the writers and run their writes under the internal system context, the split provisionAppClient makes for AppClient. Registration is idempotent on the thumbprint (the same key for the same client returns the same row), refuses a revoked key and a key of another client with a 409, and refuses a retired AppClient.
  • Operator path. sms-gateway register-client --app <slug> --public-key <base64url> [--label] [--scope …] finds or creates the app's signing client (appc_<slug>; --scope is needed only the first time) and registers the key through the same entry point as the HTTP route. Run twice, it leaves one key and exits 0. It needs the App to exist (create-app), like every operator path.

backends/crates/sms-api/src/client_keys.md has the sequence and state diagrams.

ConsoleKey (signed transport, VS-5a) — a human's request-signing public key, the keypair that replaces the console's access/ID/refresh-token session. Since VS-3 the envelope AuthProvider verifies against it as its second signer kind (ConsoleKey → user principal, §4.0). There is no other way in since VS-4: no password, no TOTP, no single sign-on.

  • ConsoleKey(userId, alg, publicKey, thumbprint, label, pairedAt, expiresAt, revokedAt), forward migration 0007_console_keys (ADDED_BY_LATER in cargo xtask migrations-current, and ConsoleKey in the system-context golden list). alg is esp256 only: a WebCrypto non-extractable P-256 key. publicKey is the SEC1 uncompressed point, 65 bytes (0x04 ‖ X ‖ Y, base64url on the wire) — what crypto.subtle.exportKey("raw", …) returns and CoseVerifyKey::p256_sec1 takes; a compressed point is refused, not normalised. thumbprint is the RFC 9679 EC2/P-256 thumbprint from cratestack::cose, lowercase hex, @unique.
  • A key lasts 30 days (owner ruling R3): expiresAt = pairedAt + 30 days, set once, never extended; an expired or revoked key is never registered again. Expiry is data, not a job — VS-3's resolver treats expiresAt <= now and revokedAt as no key.
  • Policy. create/update are hasRole('system'); read admits owner, admin, system and the key's own user (auth().kind == "user" && userId == auth().sub). No delete.
  • registerConsoleKey(vaamUserId, displayName?, publicKey, possessionProof, issuedAt) is @allow(auth().kind == "app") plus a Layer 2 check of the scope console:pair (sms_api::console_keys::require_pair_scope, which reads the principal's scope and never a human's perms). Its one caller is vaam's service client, after a vaam admin approved the pairing on a bound phone (owner ruling: consoles pair with a vaam admin phone only). possessionProof is the console key's raw ECDSA-P-256-SHA-256 signature (r‖s, base64url) over "vsms.console-pair.v1\n" + publicKey + "\n" + issuedAt (unix seconds); an issuedAt more than 300 s from the server's clock is refused before the signature is read. It then finds or JIT-provisions the User through FederatedLink(issuer = "vaam", subject = vaamUserId) with role admin, never owner (owner ruling R2, and the #434 guard: the User is written under an admin context so User.create's own roleKey != 'owner' clause is a second, independent refusal), email <vaamUserId>@vaam.invalid. A linked person's role is never touched; a deactivated account is refused. Idempotent on the thumbprint; a key that is another user's, revoked or expired is a 409.
  • revokeConsoleKey(keyId) is @allow(auth().kind == "user") + @authorize(ConsoleKey, detail, args.keyId): owner and admin revoke any key, anyone else their own.
  • Break-glass (owner ruling Q2). sms-gateway pair-console-key --user <email|id> --public-key <…> [--label], run inside the cluster, attaches a key to an existing active user with no proof of possession (the operator holding the database is the authority, as for provision-user). It creates no user and changes no role.
  • The composer keeps its own Ed25519 service key, registered as a ClientKey by register-client (owner ruling Q5). The key is spent for a person only after authorizeComposition(operation, inputSha256), a procedure with @allow(auth().kind == "user" && (hasRole('owner') || hasRole('admin') || hasRole('operator'))) and a Layer 2 check of sms:send (sms_api::composition), whose sealed answer names the person, the operation and the hash of the exact input the browser then posts: a 200 to a sealed read proves nothing about who signed, because the envelope answers every signer it accepts, an app client's key included. A refusal is a sealed 403. docs/design/console-pairing.md, "The composer".
  • The console (VS-5b) is the browser that holds the key: it makes a non-extractable P-256 key, shows it as a QR in vaam's admin-pairing format signed under vsms.console-pair.v1, waits for the gateway to accept it, and seals every call in the browser through sdks/rust/vsms-sdk-wasm built --target web. Its OIDC client, session cookie and single-sign-on plumbing are deleted. docs/design/console-pairing.md is the account.

backends/crates/sms-api/src/console_keys.md has the sequence and state diagrams.

Implement these on the generated ProcedureRegistry:

rust
fn send_message(
    &self,
    db: &super::Cratestack,
    ctx: &::cratestack::CratestackContext,
    args: send_message::Args,
) -> impl ::core::future::Future<Output = Result<send_message::Output, ::cratestack::CratestackError>> + Send;

2.10 Hand-written SQL: defaults, indexes, and the state machines ​

VS-4 (2026-10-27). 0002_bootstrap is generated from the SQL blocks below and has been applied to deployed databases, so the blocks stay as they were: the statements about oauth_clients, oauth_signing_keys, client_assertions and user_credentials (defaults, triggers, the signing-key and expiry indexes, the client-registration CHECKs) still run, on every fresh database, right before 0009_drop_openid_provider drops those tables with all of it. They are history that cannot be edited, not design. VSA-2: 0012_restore_openid_provider creates the five tables again, and repeats these statements for them (it cannot reuse these blocks: they ran, and were dropped, before it); the live test that compares the two keeps the copy honest. cargo xtask migrations-current removes the dropped tables' statements from 0001_init before comparing it with the generator, and fails if 0009 does not drop every one.

The emitter produces no non-unique indexes, no foreign keys, no triggers, and — because of dbgenerated() — no column defaults. It also, as of cratestack-migrate 0.5.0, produces no native Postgres ENUM types: every enum-typed field emits as TEXT NOT NULL plus a CHECK (col IN (...)) constraint instead of CREATE TYPE ... AS ENUM plus a typed column. Earlier versions did emit real enum types, which is why the transition tables below are TEXT, matched with their own CHECK constraints, rather than message_state/job_state columns — those types no longer exist to reference. Generate the base migration:

bash
cratestack migrate diff --schema schemas/vsms.cstack \
  --out-dir backends/migrations --backend postgres --name init

Copy the output over backends/migrations/postgres/0001_init/{up,up.pre,down}.sql — up.pre.sql only exists when the generator scaffolds one, which it does whenever it detects a blocking operation (a CHECK/NOT NULL addition it can't prove is safe against an existing table's rows); delete the committed copy if a regeneration stops emitting one.

then append the following. Without it, every insert fails and the state machines don't exist.

Defaults and identifiers.

sql
CREATE EXTENSION IF NOT EXISTS pgcrypto;

-- `Cuid` is format-guarded on REST query filters as [a-z0-9]{2,32}, so ids must
-- carry NO prefix separator or `GET /messages?id=...` returns 400.
CREATE OR REPLACE FUNCTION cs_cuid() RETURNS TEXT AS $$
  SELECT 'c' || encode(gen_random_bytes(11), 'hex');   -- 23 chars, [a-z0-9]
$$ LANGUAGE SQL VOLATILE;

ALTER TABLE apps                    ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE app_clients             ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE oauth_clients           ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE oauth_signing_keys      ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE client_assertions       ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE messages                ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE message_parts           ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE delivery_receipts       ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE jobs                    ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE providers               ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE routes                  ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE sender_ids              ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE sender_id_registrations ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE opt_outs                ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE consent_records         ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE operator_prefix_rules   ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE webhook_endpoints       ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE webhook_attempts        ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE users                   ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE roles                   ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE user_credentials        ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE audit_anchors           ALTER COLUMN id SET DEFAULT cs_cuid();
ALTER TABLE route_validations       ALTER COLUMN id SET DEFAULT cs_cuid();

-- Timestamps mixin, and other dbgenerated() columns.
ALTER TABLE apps ALTER COLUMN created_at SET DEFAULT now(),
                 ALTER COLUMN updated_at SET DEFAULT now();
-- ... repeat for every table using @use(Timestamps)
ALTER TABLE delivery_receipts ALTER COLUMN received_at SET DEFAULT now();
-- #68: AuditAnchor doesn't @use(Timestamps) — it has no updated_at (nothing
-- ever updates an anchor row; see its own schema.cstack doc) — so it needs
-- its own one-off default the same way delivery_receipts.received_at does.
ALTER TABLE audit_anchors ALTER COLUMN created_at SET DEFAULT now();
-- #64: RouteValidation is the identical shape — an append-only evidence
-- record with no @use(Timestamps), so performed_at needs its own default
-- the same way received_at/created_at above do.
ALTER TABLE route_validations ALTER COLUMN performed_at SET DEFAULT now();

-- Nothing in the framework touches updated_at on write, and remembering to set
-- it in every call site is the kind of thing that works until it doesn't.
-- clock_timestamp(), not now(): now() is the transaction timestamp, so two
-- updates to the same row inside one transaction would carry an identical
-- updated_at, and updated_at would equal created_at on a row created and
-- updated in the same transaction.
CREATE OR REPLACE FUNCTION touch_updated_at() RETURNS trigger
LANGUAGE plpgsql AS $$
BEGIN
    NEW.updated_at := clock_timestamp();
    RETURN NEW;
END $$;

CREATE TRIGGER apps_touch BEFORE UPDATE ON apps
    FOR EACH ROW EXECUTE FUNCTION touch_updated_at();
-- ... repeat for every table using @use(Timestamps)

-- Multi-value columns are space-delimited TEXT with sentinel separators (§2.2),
-- because scalar list fields panic the server macro. Empty is a single space.
ALTER TABLE app_clients       ALTER COLUMN scopes SET DEFAULT ' ';
ALTER TABLE oauth_clients     ALTER COLUMN scopes SET DEFAULT ' ',
                              ALTER COLUMN grant_types SET DEFAULT ' ',
                              ALTER COLUMN redirect_uris SET DEFAULT ' ';
ALTER TABLE apps              ALTER COLUMN ip_allowlist SET DEFAULT ' ';
ALTER TABLE roles             ALTER COLUMN permissions SET DEFAULT ' ';
ALTER TABLE webhook_endpoints ALTER COLUMN event_types SET DEFAULT ' ';

-- @version emits BIGINT NOT NULL with no default.
ALTER TABLE messages         ALTER COLUMN version SET DEFAULT 0;
ALTER TABLE webhook_attempts ALTER COLUMN version SET DEFAULT 0;
ALTER TABLE jobs             ALTER COLUMN version SET DEFAULT 0;

The message state machine, in Postgres.

sql
CREATE TABLE message_state_transitions (
    from_state TEXT NOT NULL,
    to_state   TEXT NOT NULL,
    PRIMARY KEY (from_state, to_state),
    -- The native `message_state` enum type is gone as of cratestack-migrate
    -- 0.5.0 (see above); these CHECKs are what used to be free with the type.
    CONSTRAINT message_state_transitions_from_check
        CHECK (from_state IN ('accepted', 'queued', 'routed', 'submitted', 'delivered', 'uncertain', 'undelivered', 'failed', 'expired', 'rejected', 'cancelled')),
    CONSTRAINT message_state_transitions_to_check
        CHECK (to_state IN ('accepted', 'queued', 'routed', 'submitted', 'delivered', 'uncertain', 'undelivered', 'failed', 'expired', 'rejected', 'cancelled'))
);

INSERT INTO message_state_transitions (from_state, to_state) VALUES
    ('accepted','queued'),      ('accepted','rejected'),    ('accepted','cancelled'),
    ('accepted','expired'),
    ('queued','routed'),        ('queued','cancelled'),     ('queued','expired'),
    ('queued','failed'),
    ('routed','submitted'),     ('routed','queued'),        ('routed','failed'),
    ('routed','expired'),       ('routed','cancelled'),     ('routed','uncertain'),
    ('submitted','delivered'),  ('submitted','uncertain'),  ('submitted','undelivered'),
    ('submitted','failed'),     ('submitted','expired'),
    ('uncertain','delivered'),  ('uncertain','failed'),     ('uncertain','expired'),
    ('undelivered','queued'),   ('undelivered','failed'),   ('undelivered','expired');
-- delivered, failed, expired, rejected, cancelled have NO outgoing rows.
-- Terminality is therefore data, not code: nothing leaves them.

CREATE OR REPLACE FUNCTION messages_guard_transition() RETURNS trigger
LANGUAGE plpgsql AS $$
BEGIN
    IF NEW.state IS NOT DISTINCT FROM OLD.state THEN
        RETURN NEW;
    END IF;

    IF NOT EXISTS (
        SELECT 1 FROM message_state_transitions
        WHERE from_state = OLD.state AND to_state = NEW.state
    ) THEN
        RAISE EXCEPTION
            'illegal message transition % -> % on %', OLD.state, NEW.state, OLD.id
            USING ERRCODE = 'SM001';
    END IF;

    IF NEW.state IN ('delivered','failed','expired','rejected','cancelled')
       AND NEW.finalized_at IS NULL THEN
        NEW.finalized_at := now();
    END IF;

    IF NEW.state = 'submitted' AND NEW.submitted_at IS NULL THEN
        NEW.submitted_at := now();
    END IF;

    RETURN NEW;
END $$;

CREATE TRIGGER messages_state_guard
    BEFORE UPDATE ON messages
    FOR EACH ROW EXECUTE FUNCTION messages_guard_transition();

The job state machine, same shape:

sql
CREATE TABLE job_state_transitions (
    from_state TEXT NOT NULL,
    to_state   TEXT NOT NULL,
    PRIMARY KEY (from_state, to_state),
    CONSTRAINT job_state_transitions_from_check
        CHECK (from_state IN ('pending', 'running', 'succeeded', 'failed', 'dead', 'cancelled')),
    CONSTRAINT job_state_transitions_to_check
        CHECK (to_state IN ('pending', 'running', 'succeeded', 'failed', 'dead', 'cancelled'))
);

INSERT INTO job_state_transitions (from_state, to_state) VALUES
    ('pending','running'),  ('pending','cancelled'),
    ('running','succeeded'),('running','failed'),   ('running','pending'),
    ('failed','pending'),   ('failed','dead'),      ('failed','cancelled'),
    ('dead','pending');
-- succeeded, cancelled are terminal. `dead -> pending` (#56): the one
-- caller is `requeueJob` (backends/crates/sms-api/src/procedures.rs) — an operator's
-- explicit "try this again" action from the admin Jobs screen, never
-- proposed by the automatic pipeline (`backends/crates/sms-worker/src/jobs.rs`'s own
-- `apply_failure` only ever writes `failed -> {pending, dead}`, never reads
-- a `dead` row again). Same shape as `attempt_state_transitions`'
-- `dead -> pending` (#43) two sections below, added for the identical
-- reason: a `dead` job is otherwise a true dead end, and this is the one
-- sanctioned way back from it.

CREATE OR REPLACE FUNCTION jobs_guard_transition() RETURNS trigger
LANGUAGE plpgsql AS $$
BEGIN
    IF NEW.state IS NOT DISTINCT FROM OLD.state THEN
        RETURN NEW;
    END IF;
    IF NOT EXISTS (
        SELECT 1 FROM job_state_transitions
        WHERE from_state = OLD.state AND to_state = NEW.state
    ) THEN
        RAISE EXCEPTION 'illegal job transition % -> % on %', OLD.state, NEW.state, OLD.id
            USING ERRCODE = 'SM001';
    END IF;
    IF NEW.state = 'running'   AND NEW.started_at  IS NULL THEN NEW.started_at  := now(); END IF;
    IF NEW.state IN ('succeeded','dead','cancelled') AND NEW.finished_at IS NULL THEN
        NEW.finished_at := now();
    END IF;
    RETURN NEW;
END $$;

CREATE TRIGGER jobs_state_guard
    BEFORE UPDATE ON jobs
    FOR EACH ROW EXECUTE FUNCTION jobs_guard_transition();

The webhook attempt state machine. AttemptState shipped with #38/#39 (subscribers, drain) but no transition table or trigger — nothing yet drove it, so R2's "proposed by Rust, decided by Postgres" had nothing to decide against. #40 (the hooks role, backends/crates/sms-worker/src/hooks.rs) is the first code to actually write WebhookAttempt.state, so it inherits the open question #38/#39's own PR left explicit. Resolved here in favour of the same discipline messages/jobs already get, not an exception: a table plus a BEFORE UPDATE trigger, same shape as the two above.

Two states are "waiting" (pending — never yet attempted; failed — attempted at least once, waiting out its backoff) and both are covered by webhook_due_idx below. The same two states are covered a second time, per endpoint, by webhook_endpoint_due_idx (endpoint_id, next_attempt_at), which a claim reads whenever it has to leave an endpoint out (deactivated, circuit open, or sealed for a worker without the envelope signing key): one endpoint_id = $1 query per remaining endpoint, §7.3. When every endpoint is claimable by the worker it claims by webhook_due_idx, one time-ordered read. It is migration 0011, and so not in the block below: 0002_bootstrap is generated from that block and a deployed database never runs it again. One is "in flight" (delivering). succeeded is genuinely terminal — no row below ever has it as a from_state. dead is not, as of #43: an operator's explicit replay (replayWebhookAttempt) is the one caller allowed to move a dead (or failed) row back to pending with a fresh counter; nothing in the automatic pipeline (hooks.rs) ever proposes either edge, only this procedure does. A crash-abandoned delivering lease is reclaimed the same way Message's own routed state is (§7.3's claim.rs): a same-state write that renews the lease without incrementing attempts — resuming the same in-flight attempt rather than asserting the customer endpoint never received it. Same-state writes bypass the transition-table check entirely (the trigger's own early return), so that reclaim needs no row in the table below, exactly like routed's reclaim needs none in message_state_transitions.

sql
CREATE TABLE attempt_state_transitions (
    from_state TEXT NOT NULL,
    to_state   TEXT NOT NULL,
    PRIMARY KEY (from_state, to_state),
    CONSTRAINT attempt_state_transitions_from_check
        CHECK (from_state IN ('pending', 'delivering', 'succeeded', 'failed', 'dead')),
    CONSTRAINT attempt_state_transitions_to_check
        CHECK (to_state IN ('pending', 'delivering', 'succeeded', 'failed', 'dead'))
);

INSERT INTO attempt_state_transitions (from_state, to_state) VALUES
    ('pending','delivering'),    ('failed','delivering'),
    ('delivering','succeeded'),  ('delivering','failed'),  ('delivering','dead'),
    ('failed','pending'),        ('dead','pending');
-- succeeded is the only true terminal state. `delivering -> dead` covers
-- both reasons §8.5 stops retrying outright: `maxAttempts` exhausted, and
-- an immediate 410 Gone (which also deactivates the endpoint — hooks.rs,
-- not this trigger). `failed -> dead` does not exist: the exhausted-
-- attempts check happens once, at the delivering -> {failed | dead}
-- decision the hooks role's own write makes, not as a second hop through
-- failed.
--
-- `failed -> pending` and `dead -> pending` (#43): the replay edges.
-- `replayWebhookAttempt` (backends/crates/sms-api/src/procedures.rs) is the only
-- caller of either — an operator's explicit "re-fire this after fixing the
-- receiving end" action, never proposed by the automatic pipeline. No
-- `succeeded -> pending` edge exists, on purpose: re-firing a webhook the
-- receiver already processed successfully is a materially more dangerous
-- operation than re-firing one that never got through, and this story
-- (#43) is about the latter. `delivering -> pending` also does not exist,
-- so a replay can never race a lease a worker currently holds — the
-- procedure's own read happens outside any lease the claim loop takes, and
-- `if_match(version)` on its write turns a race against a concurrent claim
-- into a `PreconditionFailed`, not a corrupted attempt.

CREATE OR REPLACE FUNCTION attempts_guard_transition() RETURNS trigger
LANGUAGE plpgsql AS $$
BEGIN
    IF NEW.state IS NOT DISTINCT FROM OLD.state THEN
        RETURN NEW;
    END IF;
    IF NOT EXISTS (
        SELECT 1 FROM attempt_state_transitions
        WHERE from_state = OLD.state AND to_state = NEW.state
    ) THEN
        RAISE EXCEPTION 'illegal webhook attempt transition % -> % on %', OLD.state, NEW.state, OLD.id
            USING ERRCODE = 'SM001';
    END IF;
    IF NEW.state = 'succeeded' AND NEW.delivered_at IS NULL THEN
        NEW.delivered_at := now();
    END IF;
    RETURN NEW;
END $$;

CREATE TRIGGER attempts_state_guard
    BEFORE UPDATE ON webhook_attempts
    FOR EACH ROW EXECUTE FUNCTION attempts_guard_transition();

Indexes.

sql
-- The dispatch claim path.
CREATE INDEX messages_dispatch_idx
    ON messages (priority DESC, created_at)
    WHERE state IN ('accepted','queued') AND lease_until IS NULL;

CREATE INDEX messages_lease_reclaim_idx
    ON messages (lease_until)
    WHERE lease_until IS NOT NULL AND state IN ('queued','routed','undelivered');

CREATE INDEX messages_app_created_idx   ON messages (app_id, created_at DESC);
CREATE INDEX messages_state_created_idx ON messages (state, created_at DESC);
CREATE INDEX messages_msisdn_hash_idx   ON messages (msisdn_hash, created_at DESC);

-- #67's purge_retention candidate query — a terminal message, not yet
-- purged, past its own createdAt cutoff. Partial and narrow, same style as
-- messages_dispatch_idx/messages_lease_reclaim_idx above, rather than
-- leaning on messages_state_created_idx alone: that index still has to
-- scan every non-purged row of five different states before this job's
-- own extra purged_at filter narrows it.
CREATE INDEX messages_purge_idx ON messages (created_at)
    WHERE purged_at IS NULL
      AND state IN ('delivered','failed','expired','rejected','cancelled');
CREATE INDEX messages_provider_ref_idx  ON messages (provider_id, provider_message_ref)
    WHERE provider_message_ref IS NOT NULL;
CREATE INDEX messages_provider_ref_alt_idx ON messages (provider_id, provider_message_ref_alt)
    WHERE provider_message_ref_alt IS NOT NULL;

CREATE UNIQUE INDEX messages_app_idem_key
    ON messages (app_id, idempotency_key)
    WHERE idempotency_key IS NOT NULL;

-- The job claim path, plus dedupe on non-terminal jobs only.
CREATE INDEX jobs_claim_idx
    ON jobs (priority DESC, run_at)
    WHERE state = 'pending';

CREATE INDEX jobs_lease_reclaim_idx
    ON jobs (lease_until)
    WHERE state = 'running';

CREATE UNIQUE INDEX jobs_dedupe_idx
    ON jobs (kind, dedupe_key)
    WHERE dedupe_key IS NOT NULL AND state IN ('pending','running','failed');

-- THE webhook dedupe key. Not optional: see §8.3.
CREATE UNIQUE INDEX webhook_attempts_dedupe
    ON webhook_attempts (endpoint_id, aggregate_id, event_type);

CREATE INDEX webhook_due_idx ON webhook_attempts (next_attempt_at)
    WHERE state IN ('pending','failed');

-- The hooks role's crash-reclaim query (a stale `delivering` lease) — same
-- role `messages_lease_reclaim_idx`/`jobs_lease_reclaim_idx` play for their
-- own claim loops.
CREATE INDEX webhook_attempts_lease_reclaim_idx ON webhook_attempts (lease_until)
    WHERE state = 'delivering';

CREATE INDEX receipts_lookup_idx  ON delivery_receipts (provider_id, provider_message_ref);
CREATE INDEX receipts_message_idx ON delivery_receipts (message_id);
-- #67's purge_retention delete query — receipts age off their own
-- received_at, independent of their parent message's age (see §2.5).
CREATE INDEX receipts_received_at_idx ON delivery_receipts (received_at);
CREATE INDEX app_clients_app_idx  ON app_clients (app_id);
CREATE INDEX routes_match_idx     ON routes (enabled, priority DESC);

-- #72: `Procedures::ensure_consent_on_file`'s own lookup — no `@unique` on
-- `msisdnHash` here the way `OptOut.msisdnHash` has one, because a single
-- recipient can legitimately hold more than one ConsentRecord (different
-- scopes, or a re-consent over time; this model is append-only evidence,
-- not a single mutable flag — see its own schema.cstack comment).
CREATE INDEX consent_records_lookup_idx
    ON consent_records (app_id, msisdn_hash, scope);

-- The OP reads exactly one row at startup: the newest active signing key.
CREATE INDEX oauth_signing_keys_active_idx ON oauth_signing_keys (created_at DESC)
    WHERE active;

-- Reaping spent client assertions. A `jti` need only be remembered until its
-- own `exp`; after that the assertion is refused on `exp` regardless, so
-- keeping the row would only grow the table.
CREATE INDEX client_assertions_expiry_idx ON client_assertions (expires_at);

-- #68's anchor_audit job reads exactly one row every run: the most recent
-- anchor, to chain the next one off its chainHash. Never a large table (at
-- most one row per scheduled run, per day per §7.5), but the same
-- "index the one lookup a singleton job always makes" reasoning
-- oauth_signing_keys_active_idx above already applies.
CREATE INDEX audit_anchors_period_end_idx ON audit_anchors (period_end DESC);

-- #64's grey_route_watch job reads exactly one thing per route: the most
-- recent validation, to compute staleness. Same "index the one lookup a
-- job's own query always makes" reasoning as oauth_signing_keys_active_idx/
-- audit_anchors_period_end_idx above.
CREATE INDEX route_validations_route_performed_idx
    ON route_validations (route_id, performed_at DESC);

-- The framework's own outbox. `ensure_event_outbox_table` creates this lazily
-- on the first emitting write, which is too late to index it here: applying
-- the migration to a fresh database fails with
--   ERROR: relation "cratestack_event_outbox" does not exist
-- Create it ourselves, with the framework's exact shape, so the index has
-- something to attach to. The framework's IF NOT EXISTS then no-ops.
CREATE TABLE IF NOT EXISTS cratestack_event_outbox (
    event_id UUID PRIMARY KEY,
    model TEXT NOT NULL,
    operation TEXT NOT NULL,
    occurred_at TIMESTAMPTZ NOT NULL,
    payload JSONB NOT NULL,
    delivered_at TIMESTAMPTZ,
    attempts BIGINT NOT NULL DEFAULT 0,
    last_error TEXT
);

CREATE INDEX cratestack_event_outbox_undelivered_idx
    ON cratestack_event_outbox (occurred_at, event_id)
    WHERE delivered_at IS NULL;

Referential integrity. At =0.6.7, cratestack migrate diff emitted none of this — every FK below was hand-written for that reason. As of the =0.7.8 bump, the emitter itself now writes a plain (ON DELETE NO ACTION) FOREIGN KEY into 0001_init for every @relation(fields:[...], references:[...]) in the schema — all twelve of them, covering every relation below. Most of those are exactly what we want and need no help from here any more.

Three are not: message_parts, delivery_receipts, and webhook_attempts need ON DELETE CASCADE so a purged parent takes its children with it, and the framework's emitted default doesn't provide that. Two FOREIGN KEY constraints on the same column enforce independently — leaving the emitted NO ACTION constraint in place alongside a hand-written CASCADE one does not "win" toward the more permissive behaviour, it silently reintroduces the block: Postgres checks both, and the NO ACTION one raises violates foreign key constraint before the CASCADE one ever runs. Confirmed live: deleting a messages row with a message_parts child against that combination fails instead of cascading. So for those three, the migration drops the emitted constraint by name and replaces it with the cascading one, rather than adding a second constraint beside it.

The other five relations (messages, app_clients, routes, users, and oauth_clients until 0009 dropped it) get exactly the emitted default already, so nothing further is written here for them — a hand-written duplicate would be redundant, not additive, and this section only earns its keep by doing what the framework can't.

sql
ALTER TABLE message_parts DROP CONSTRAINT message_parts_message_id_fkey;
ALTER TABLE message_parts ADD CONSTRAINT parts_message_fk
    FOREIGN KEY (message_id) REFERENCES messages(id) ON DELETE CASCADE;
ALTER TABLE delivery_receipts DROP CONSTRAINT delivery_receipts_message_id_fkey;
ALTER TABLE delivery_receipts ADD CONSTRAINT receipts_message_fk
    FOREIGN KEY (message_id) REFERENCES messages(id) ON DELETE CASCADE;
ALTER TABLE webhook_attempts DROP CONSTRAINT webhook_attempts_endpoint_id_fkey;
ALTER TABLE webhook_attempts ADD CONSTRAINT wha_endpoint_fk
    FOREIGN KEY (endpoint_id) REFERENCES webhook_endpoints(id) ON DELETE CASCADE;

Format constraints @db_enforce cannot express. It backs @length/@range with a real CHECK — confirmed against SenderId.value, the schema's one other @db_enforce field — but is a silent no-op on @regex: no CHECK, no error, no warning. Verified with an isolated two-field probe schema (§2.0). Anything pattern-shaped needs its CHECK written by hand:

sql
ALTER TABLE operator_prefix_rules ADD CONSTRAINT operator_prefix_rules_prefix_format_check
    CHECK (prefix ~ '^[0-9]{1,4}$');

Reserved Role.key values (#194). Role.key's own @regex (^[a-z][a-z0-9_]{2,31}$) has no way to exclude specific literals, and @db_enforce is a silent no-op on @regex fields regardless (the same gap the operator_prefix_rules format check above exists to close) — so nothing in the schema stops an owner from creating a Role keyed exactly "system" through ordinary generated CRUD (Role.create's own @@allow is hasRole('owner')) and assigning a human User to it. hasRole('system') matches on Role.key verbatim, with no other check in Layer 1 — system is supposed to be synthetic, constructible only inside a process (§5.2), never a real database row a human account can reach. A Role named "system" would let a human pass every hasRole('system') policy in the schema (the signer directory's key and nonce tables, among others) through generated CRUD. "app" is reserved alongside it, defensively — GatewayAuth's own machine-caller sentinel role: "app" matches no hasRole(...) clause today (see system_context's doc in sms-api's auth.rs), so a human Role keyed "app" is confusing rather than exploitable, but reserving it costs nothing and keeps that true rather than assumed. This is one of two independent guards — backends/crates/sms-api/src/auth.rs's load_human_principal refuses a "system"/"app" role_key at the point of use regardless of what the database allows; see that function's own doc for why both exist rather than just one:

sql
ALTER TABLE roles ADD CONSTRAINT roles_key_not_reserved_check
    CHECK (key NOT IN ('system', 'app'));

Who may hand out owner is the same kind of invariant, guarded the same two-independent-ways way, but in Rust and in the schema rather than in a CHECK — see §5.2 ("Only an owner may hand out owner"). It needs no DDL: the schema half is a policy-only change to User's @@allow, which the migration emitter never sees.

Client-registration invariants (the table they constrain, oauth_clients, was dropped by 0009 in VS-4 and is back, empty, with both CHECKs, from 0012 in VSA-2). The schema can say a client names one auth method; it cannot say that the method and the rest of the row agree. Both of these are the difference between a misconfigured client failing at INSERT and failing at /token, where the symptom is an authentication error nobody can explain:

sql
-- private_key_jwt without a key is a client that can never authenticate;
-- `none` *with* a key is a public client someone believed was confidential.
--
-- "Without a key" has to mean an empty key set too, not just a NULL column:
-- `{"keys":[]}` is not null and is still keyless. The predicate is written to
-- be *total* — it returns true or false for every JSON shape, never NULL —
-- because a NULL in a CHECK passes. `jsonb_typeof(...) = 'array'` alone is
-- NULL when `keys` is absent, and `jsonb_array_length` raises when `keys` is
-- an object, so neither is usable on its own. Verified against Postgres 16
-- for `{"keys":[{...}]}` (pass) and each of `{"keys":[]}`, `{}`,
-- `{"keys":null}`, `{"keys":{}}`, `{"keys":"abc"}`, `null`, `[]` (all fail).
--
-- What this does NOT do is validate the keys themselves — `{"keys":[{}]}`
-- passes. Structural JWK validation needs to parse each key and belongs in
-- `provisionAppClient`, which parses them anyway. This constraint exists to
-- catch the registration that is empty or malformed at the top level, which
-- is the mistake people actually make.
--
-- A `jwks` that is not JSON at all fails on the `::jsonb` cast with a json
-- syntax error rather than a constraint violation. Still rejected, just with
-- a less obvious message.
ALTER TABLE oauth_clients ADD CONSTRAINT oauth_clients_auth_method_jwks_check
    CHECK ((token_endpoint_auth_method = 'private_key_jwt'
              AND COALESCE(jsonb_typeof(jwks::jsonb -> 'keys'), '') = 'array'
              AND jsonb_path_exists(jwks::jsonb, '$.keys[0]'))
        OR (token_endpoint_auth_method = 'none' AND jwks IS NULL));

-- A client that presents no credential at all has only PKCE standing between
-- it and anyone who can reach /token. `none` without require_pkce is an open
-- token endpoint, so the database refuses to store one.
ALTER TABLE oauth_clients ADD CONSTRAINT oauth_clients_public_requires_pkce_check
    CHECK (token_endpoint_auth_method <> 'none' OR require_pkce);

Seed data: operator prefix rules, current best evidence per §3.4. 650–659 is fully partitioned between MTN and Orange, so no separate 65 row is needed; 66, 640–642 and everything else are deliberately left unseeded rather than guessed — a gap here resolves to OperatorCode::unknown at lookup, which is honest, where a fabricated row would not be. source defaults to 'seed'. version is 0 on every row (#59): this INSERT bypasses the CrateStack ORM entirely, so it has to supply the same initial value create_record_with_executor seeds server-side for every other row — the column is NOT NULL with no SQL DEFAULT (§2.0: @version is deliberately excluded from any caller-settable default), and omitting it here fails migration application with null value in column "version" ... violates not-null constraint, caught applying this migration for real against a scratch Postgres, not by cratestack check:

sql
INSERT INTO operator_prefix_rules (prefix, operator, confidence, notes, version) VALUES
    ('62',  'camtel', 'unverified', 'Camtel; unverified per architecture.md §3.4', 0),
    ('67',  'mtn',    'likely',     'MTN 67x per architecture.md §3.4', 0),
    ('68',  'unknown','contested',  'Contested between sources per architecture.md §3.4 — do not treat as reliable', 0),
    ('69',  'orange', 'likely',     'Orange 69x per architecture.md §3.4', 0),
    ('650', 'mtn',    'likely',     'MTN 650-654 per architecture.md §3.4', 0),
    ('651', 'mtn',    'likely',     'MTN 650-654 per architecture.md §3.4', 0),
    ('652', 'mtn',    'likely',     'MTN 650-654 per architecture.md §3.4', 0),
    ('653', 'mtn',    'likely',     'MTN 650-654 per architecture.md §3.4', 0),
    ('654', 'mtn',    'likely',     'MTN 650-654 per architecture.md §3.4', 0),
    ('655', 'orange', 'likely',     'Orange 655-659 per architecture.md §3.4', 0),
    ('656', 'orange', 'likely',     'Orange 655-659 per architecture.md §3.4', 0),
    ('657', 'orange', 'likely',     'Orange 655-659 per architecture.md §3.4', 0),
    ('658', 'orange', 'likely',     'Orange 655-659 per architecture.md §3.4', 0),
    ('659', 'orange', 'likely',     'Orange 655-659 per architecture.md §3.4', 0);

The built-in roles (#194). User.roleKey is a foreign key to Role.key, so no human account can exist until a Role row does — and nothing seeded one. sms-gateway provision-user, the command that bootstraps the very first operator, therefore failed on every fresh database with violates foreign key constraint "users_role_key_fkey", which made #194's login flow unbootstrappable in a real deployment as well as in just demo. Role.builtin existed from the start and implies exactly this seed; it was simply never written.

The six rows below are §5.2's table as 0002_bootstrap seeds it (the generated, frozen migration), verbatim; 0013_role_read_scope then takes sms:read from developer and support and adds pending, and 0014_optout_read_scope gives auditor optout:read, which is why §5.2's table now differs from these rows by exactly that. system is deliberately absent and cannot be added here: roles_key_not_reserved_check (above) rejects it, and §5.2 is explicit that system is synthetic — constructible only inside a process, never a database row a human account could be assigned to.

permissions is sentinel-packed in sms_core::pack's format — a leading space, then each value followed by a space — so a LIKE '% perm %' or .contains(" perm ") test cannot false-match a longer permission that merely starts with the same characters. version is 0 for the same reason every other hand-written seed sets it: these INSERTs bypass the framework's own server-side @version seeding, and the column is NOT NULL.

sql
INSERT INTO roles (key, label, description, builtin, permissions, version) VALUES
    ('owner',     'Owner',     'Break-glass. 1-2 humans.',            true,
     ' sms:read sms:send message:cancel message:create message:update app:read app:write client:provision provider:read provider:update provider:delete route:read route:write sender:manage optout:manage webhook:manage job:read job:enqueue worker:read dashboard:read audit:read user:manage user:delete role:manage ', 0),
    ('admin',     'Admin',     'Day-to-day administration.',          true,
     ' sms:read sms:send message:cancel app:read app:write client:provision provider:read provider:update route:read route:write sender:manage optout:manage webhook:manage job:read job:enqueue worker:read dashboard:read audit:read user:manage ', 0),
    ('operator',  'Operator',  'Runs traffic.',                       true,
     ' sms:read sms:send message:cancel provider:read provider:update route:read sender:manage optout:manage job:read job:enqueue worker:read dashboard:read ', 0),
    ('developer', 'Developer', 'Integrates apps.',                    true,
     ' app:read webhook:manage sms:read sms:send ', 0),
    ('auditor',   'Auditor',   'Read-only oversight. No mutations.',  true,
     ' sms:read app:read provider:read route:read job:read worker:read dashboard:read delivery:read audit:read ', 0),
    ('support',   'Support',   'First-line support.',                 true,
     ' sms:read optout:manage delivery:read ', 0);

Correction, #211, found while wiring the console to forward a human's own session token instead of its machine credential for the first time: the six rows above originally used message:read/message:send, not sms:read/sms:send, and no role carried dashboard:read at all — despite the very next section's own sentence claiming service-account scopes and human perms are "the same literals... reused verbatim." They were not. require_permission (backends/crates/sms-api/src/rbac.rs) checks the literal "sms:read"/"sms:send" for listMessageReceipts/sendMessage, and "dashboard:read" for dashboardSummary — a human role carrying message:read instead satisfied neither check. This was silent until now because no human token had ever reached sms-api before #211: the console authenticated upstream as its own machine credential (whose AppClient scope, SMS_CONSOLE_SCOPE, already correctly said sms:read/dashboard:read etc.), so the mismatch only lived in the human side of the vocabulary, which nothing had ever exercised. message:cancel/message:create/message:update are untouched — no procedure checks those literals today, so there was nothing to rename. Fixed here, data-only (no schema/DDL change — permissions is a String column; regenerated via ci/gen-bootstrap-sql.py, not hand-edited) rather than filed as a separate follow-up, because #211's own "prove the allow case genuinely works" requirement would otherwise fail the moment a human tried to open the message-detail or dashboard screens.

Three notes on the triggers.

SM001 is a user-defined SQLSTATE. CrateStack's CratestackError::DatabaseTyped carries DbErrorInfo { detail, sqlstate, constraint }, so you branch on it exactly rather than substring-matching an error message:

rust
match err {
    CratestackError::DatabaseTyped(info) if info.sqlstate.as_deref() == Some("SM001") =>
        CratestackError::Conflict("message is no longer in a state that allows this".into()),
    other => other,
}

Without that mapping an illegal transition surfaces as a 500 DATABASE_ERROR, which is both wrong and unhelpful — it's a 409.

The trigger is BEFORE UPDATE on the whole row, not BEFORE UPDATE OF state, with an early return when the state is unchanged. UPDATE OF fires on the column appearing in the SET list, which is almost the same thing but depends on how the ORM builds its statement; the unconditional form plus the early return is airtight and costs one comparison.

And this migration reports has_blocking = true. The single blocking operation is the @db_enforce CHECK on SenderId.value — harmless against an empty table, but it takes an ACCESS EXCLUSIVE lock and full-scans a populated one. If you ever add a @db_enforce constraint to a live table, split it by hand into ADD CONSTRAINT … NOT VALID then a separate VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE. The emitter won't do that for you — zero-downtime migration generation is an explicit non-goal.

cratestack migrate diff has no dry-run or --check mode — its flags are --schema, --out-dir, --backend, --name, --allow-destructive, and migrate has no other subcommand. CI drift gates are cratestack diff <old> <new> and generate-typescript --check.

3. The send path ​

3.1 API surface ​

text
POST /v1/$procs/sendMessage      # the send API
POST /v1/$procs/previewMessage   # encoding + cost dry-run, free
GET  /v1/messages/{id}           # generated CRUD, app-scoped by policy
GET  /v1/messages?limit&offset&sort=-createdAt&where=...
POST /v1/$procs/cancelMessage
GET  /v1/webhook-endpoints  ·  POST  ·  PATCH  ·  DELETE
POST /dlr/{providerKey}          # provider callbacks, NOT CrateStack-routed; answers 200, not 202
GET  /healthz  ·  /readyz  ·  /metrics

Everything under /adm/* is the same generated router with an IP allowlist at the edge and a kind == "user" gate.

List envelope:

json
{ "items": [], "totalCount": 1284,
  "pageInfo": { "limit": 20, "offset": 40, "hasNextPage": true, "hasPreviousPage": true } }

Errors are {"code": "VALIDATION_ERROR", "message": "...", "details": {...}} with SCREAMING_SNAKE codes on REST. (Under transport rpc or grpc they become lowercase not_found — error handling is binding-specific. One more reason to stay on REST.)

3.2 What sendMessage does ​

mermaid
sequenceDiagram
    autonumber
    participant C as Calling app
    participant API as sms-api
    participant PG as Postgres
    participant W as sms-worker<br/>dispatch role
    participant P as Provider

    C->>API: POST /v1/$procs/sendMessage<br/>sealed (COSE_Sign1) + Idempotency-Key
    API->>API: verify the envelope (ClientKey, nonce)
    API->>PG: signer → AppClient → App (cached 60s)
    API->>API: normalise MSISDN, classify encoding, segment
    API->>PG: opt-out check on msisdnHash
    API->>PG: quota + per-MSISDN OTP limit
    API->>PG: resolve sender ID, require approved registration

    rect rgb(238, 244, 252)
        note over API,PG: one transaction
        API->>PG: INSERT message (state = accepted)
        API->>PG: INSERT cratestack_event_outbox row
    end
    API-->>C: 202 Accepted { messageId, segments, estimatedCostXaf }

    W->>PG: claim: UPDATE ... FOR UPDATE SKIP LOCKED<br/>accepted → routed (trigger validates)
    W->>P: submit
    P-->>W: 201 + provider ref
    W->>PG: routed → submitted (trigger stamps submitted_at)

    P-->>API: POST /dlr/{provider}
    API->>PG: INSERT delivery_receipt<br/>submitted → delivered

Steps 1 to 9 are all pre-persistence. A message that reaches the database is one you have already decided you can send — the worker's job is delivery, not validation. This diagram predates #72 and is not redrawn for it (mermaid parsing is CI-gated per this file's own note on that job; a diagram edit means re-verifying it, and this one is otherwise still accurate) — read backends/crates/sms-api/src/consent.rs's own module doc for the real, current step order: two more accept-time checks now sit between "opt-out check" and "quota", both class-gated (marketing/notification only; otp/transactional are exempt) — self-imposed marketing quiet hours (08:00–20:00 WAT), then a ConsentRecord lookup.

The client_id → App lookup is where the absence of API keys shows up. A signed request names a key, not an app, so the gateway derives the App from the verified signer's AppClient (§4.0). That's on the hot path, so cache it — 60 seconds is short enough that retiring a client takes effect promptly and long enough that the lookup never matters.

Return 202 Accepted, never 200. You have not sent anything yet, and an API that implies otherwise produces callers that don't handle DLRs. This rule is about this endpoint — sendMessage — and nothing else. It was once mis-cited to justify answering 202 on the inbound POST /dlr/{providerKey} provider callback, where the logic inverts: Orange's published contract requires a 200 OK to acknowledge a receipt, and there is no "not sent yet" to communicate on a callback reporting something that already happened. See §4.1 and §6.2.

3.3 Encoding — write this crate first ​

The highest-return component in the project, and the conventional wisdom about it is wrong.

é, è, à, ù, ì, ò are all in the GSM 03.38 default alphabet. One character each. The widespread belief that French accents force UCS-2 is false.

ç is not. Lowercase c-cedilla is absent from GSM-7; only uppercase Ç (0x09) is encodable. So "Votre code a été reçu" silently drops from 160 characters to 70, doubling segments and cost. Same for ê â î ô û ë ï ÿ œ È À Ù Ê and — the sneaky one — the typographic apostrophe ’ (U+2019), which arrives constantly via copy-paste from Word and Google Docs.

€ [ ] { } \ | ^ ~ are in the extension table and cost two characters each.

rust
pub enum SmsEncoding { Gsm7, Ucs2 }

pub struct EncodingReport {
    pub encoding: SmsEncoding,
    pub length: usize,                 // encoding units, escapes counted as 2
    pub segments: u8,
    pub per_segment: usize,            // 160/153 gsm7, 70/67 ucs2
    pub offending: Vec<OffendingChar>, // char, byte offset, suggested replacement
    pub suggestion: Option<String>,    // transliterated body
}

pub fn analyse(body: &str) -> EncodingReport;
pub fn normalise(body: &str) -> String;  // ’→' “”→" –—→- NBSP→space, NFC
pub fn transliterate_to_gsm7(body: &str) -> (String, Vec<Replacement>);

normalise runs on every message unconditionally — replacing ’ with ' changes nothing a recipient can perceive and can halve your bill. transliterate_to_gsm7 (ç→c, ê→e) is opt-in per app via App.transliterateToGsm7, because for a notification it's fine and for a name field it's corruption. Surface both in previewMessage and in the admin composer, with offending characters highlighted.

Some SMSCs transliterate ç themselves, some don't, and the behaviour is route-dependent and undocumented. Don't rely on it. Test per route with a real handset.

Segment sizes: GSM-7 = 160 single / 153 concatenated; UCS-2 = 70 single / 67 concatenated. The concatenated numbers are lower because of the 6-byte UDH.

3.4 MSISDN normalisation ​

sms-msisdn validates against Google libphonenumber's own metadata, embedded at compile time by the phonenumber crate — every country, with Cameroon as the default region. A bare national number with no country code of its own is read as Cameroonian, so 677123456 still means +237677123456 with nothing to configure; Msisdn::parse_in/parse_mobile_in take an ISO 3166-1 alpha-2 region for everyone else. A leading + names the country itself and outranks the region either way. This is stage 1 of the multi-country program, implementing decisions 1 to 3 of that program's own design record.

One trap worth knowing, verified against the metadata rather than assumed: an international prefix that is not + is the dialling region's own, not a universal 00. Cameroon, most of Europe and much of Africa dial 00; North America dials 011; Nigeria dials 009. So 00336… is a French number when read against CM and is not one when read against US. Integrators should send +, which is unambiguous everywhere.

Cameroon is a closed 9-digit plan since November 2014. E.164 is +237 + 9 digits.

text
mobile     (?:24[23]|6(?:[25-9]\d|4[01]))\d{6}
fixedLine  2(?:22|33)\d{6}
general    [26]\d{8}|88\d{6,7}

Valid mobile leading pairs: 62, 65, 66, 67, 68, 69, plus 640–641 only. 63x and 642–649 are not valid. This document and sms-msisdn's own hand-written table both used to claim 642 was assigned; libphonenumber 9.0.33 assigns 640 and 641 only, and following the metadata rather than carrying a local override is the whole point of the swap. Reject fixed-line and toll-free at the API boundary — a Twilio-style 21614 failure three seconds later is a worse experience than a synchronous 422.

Addressability, not mobility, is the send-path test. parse_mobile accepts Mobile and FixedLineOrMobile, because the North American Numbering Plan does not encode the distinction at all and accepting only Mobile rejects every US and Canadian number. Cameroon is unaffected: its metadata returns Mobile for mobile ranges and FixedLine for 2xx, so a Cameroonian fixed line is refused exactly as before.

Operator inference is a hint in a database table, never a hardcoded match. Best current evidence: MTN = 67x, 650–654; Orange = 69x, 655–659. 68x is genuinely contested between sources. Camtel = 62x is unverified. Number portability is legally live (since Sept 2017) but commercially near-dead, and sources contradict each other on whether it works at all — so prefix routing is right almost always and must never be load-bearing. Record the delivering network from DLRs where reported, and let observed data correct the table.

4. Security ​

4.0 The signed transport (VS-3) ​

Every route the generated router serves is wrapped in cratestack's EnvelopeLayer, as the router's last layer (the coarse per-source rate limiter sits outside it). Its mode follows the ways in the deployment opened (auth.methods, §4.9): Required when the signed envelope is the only one (the default, and vaam), so a request is a COSE_Sign1 message and nothing unsigned reaches the router; Optional when a token is served too (machine_tokens, or since VSA-4b human_login: [keypair, human_login] is Optional, where it was Required while no person's token could be used), so a signed request is opened and sealed as ever and an unsigned one reaches the bearer door (§4.10); Off when keypair is off and tokens are the only way in, so a request with a COSE body is refused with a 415 and never forwarded. An Authorization header beside a good signature is refused with a 400 in every mode.

mermaid
sequenceDiagram
    participant C as Caller (key)
    participant L as EnvelopeLayer
    participant D as SignerDirectory
    participant N as EnvelopeNonce
    participant R as RBAC, policies, handler
    C->>L: COSE_Sign1 (audience vsms, route, iat, cti)
    L->>D: resolve kid (non-revoked ClientKey, or unexpired ConsoleKey)
    alt unknown, revoked, expired, bad signature
        L-->>C: 401, unsigned, coarse
    end
    L->>N: record cti once (unique index)
    alt cti seen inside its window
        L-->>C: 401, unsigned, coarse
    end
    L->>D: standing (active AppClient, or active User and Role)
    L->>R: principal (app: clientId + scopes, user: role + permissions)
    R-->>L: answer
    L-->>C: COSE_Sign1 signed by the gateway's current key
  • Two signer kinds. An Ed25519 ClientKey of an active, unretired AppClient is an app principal (sub = clientId, scopes from AppClient.scopes). An ESP256 ConsoleKey, not revoked and not expired, of an active User whose role is not reserved, is a user principal. Key rows are read on every request, so a revocation or expiry takes effect on the next call on every replica; only the owning client's or person's standing is cached, for 60 seconds.
  • Every refusal of a signature is one unsigned 401 (unknown, revoked, expired, forged, replayed, inactive owner). A holder of a dead key learns nothing from the answer. A payload that is not application/json is an unsigned 415 before any key is looked up.
  • Replay protection is a table, not a cache. EnvelopeNonce (forward migration 0008) holds one row per accepted cti, behind a unique index, so two replicas receiving one request record it once between them. An expired row that has not been swept yet is deleted and the insert retried once; the worker's sweep_envelope_nonces job deletes expired rows in bounded batches.
  • The server holds two Ed25519 keys, --envelope-signing-key-file (signs every answer) and --envelope-next-signing-key-file (promoted by the next rotation), both PKCS#8 PEM and both required; the same key twice is refused and serve does not boot without them. They are the only keys it needs: it takes no issuer and no signing key of an identity provider (VS-4). sms-gateway server-keys prints the two public keys a client pins, so a rotation is: check that clients already pin today's next (the key being promoted), promote it, hand out the newly minted next, retire the old key. Whatever the mode, the keys are required: serve does not boot without them.
  • Outside the layer: /dlr/* (provider callbacks, which authenticate themselves), /healthz and /readyz, GET /.well-known/vsms-auth (§4.9) and, only when machine_tokens or human_login is served (§4.10), the OpenID Provider's /token, /jwks.json and /.well-known/openid-configuration, plus POST /login when human_login is. With neither served those are the router's 404, like /login/sso/* (deleted in VS-4, back with VSA-4c, and served only when human_login.sso is written): the_deleted_openid_provider_routes_are_not_served and with_the_method_off_nothing_is_mounted_and_the_commands_refuse request each from a real sms-gateway serve. A machine token is a credential of the layer's routes, through the bearer door (§4.10), only when machine_tokens is served.
  • Layer order, outside in: per-source rate limit, envelope, the bearer door (§4.10), per-principal rate limit (charged to client:<id> or user:<id>), idempotency (fingerprinted on the verified principal), the owner-grant guard, the four RBAC layers, the keypair gate and the hmac_webhooks gate (§4.9), the generated router. Beside the router, outside the envelope: /dlr/*, /healthz, /readyz and GET /.well-known/vsms-auth (§4.9).

4.1 Transport ​

TLS 1.3 preferred, 1.2 floor, at a Caddy or nginx edge with automatic Let's Encrypt. Rust services listen on loopback or a private network only. HSTS with a one-year max-age once you're confident. crypto-aws-lc-rs is available as a CrateStack Cargo feature if you ever need FIPS-validated TLS.

One hard external constraint: Orange will only call a DLR webhook on HTTPS port 443 with a CA-signed certificate. Self-signed is rejected outright. Your /dlr/* endpoint must be publicly reachable on 443 with a real cert before Orange will even whitelist it — and whitelisting is a manual support ticket, not self-service. Budget a week. The endpoint must also answer 200 OK — Orange's docs are explicit that a 200 is what acknowledges the receipt, and §3.2's 202 Accepted rule does not apply here (it governs this API's own send endpoint, not an inbound provider callback).

Internal traffic (worker ↔ smpp role, if ever split) uses mTLS with a private CA, or a Unix socket if co-located.

4.2 Machine callers: a registered Ed25519 key ​

Every calling app holds an Ed25519 keypair it generated itself; vsms stores only the public half (ClientKey, §2.9). A request is a COSE_Sign1 envelope signed with that key (§4.0): no grant, no token endpoint, no client assertion, no shared secret, no API key and no HMAC over the body. The replay protection a long-lived API key would have needed is the envelope's own iat freshness window plus the EnvelopeNonce table, so a captured request is refused the second time it arrives.

Three ways a key gets registered, all under one set of rules (idempotent on the RFC 9679 thumbprint; a revoked key or another client's key is a 409; a retired or inactive client is refused):

  • sms-gateway register-client --app <slug> --public-key <base64url> [--client-name N] [--scope S …] — the operator path. It finds or creates the app's signing client (appc_<slug>, or appc_<slug>_<name> for a second integration of the same app) and attaches the key. It is also the only path that may give a client console:pair, the scope that lets a client attach a console key to a person (§4.3).
  • provisionAppClient(appId, label, scopes, publicKey) — HTTP, owner/admin: makes a fresh appc_<uuid> client and registers the key in one transaction; refuses console:pair for every caller. The admin console's "Provision a service-account client" panel calls it with a public key the operator pastes in.
  • registerAppClientKey adds a further key to an existing client, and revokeAppClientKey ends one. Rotation is therefore register-then-revoke with both keys valid in between; a revocation takes effect on the next request on every replica, because key rows are read on every request.

Scopes live on AppClient.scopes, are read through the signer directory's 60-second standing cache, and reach the context as scope. Layer 2 (require_permission, §5.1) checks them, and a missing scope is a denial — a client with no scope can sign but do nothing.

What a stolen private key allows is whatever its client's scopes allow, until someone revokes the key. That is the whole exposure and it is bounded three ways: revocation is immediate; App.ipAllowlist narrows where a client may call from at the edge; and the per-principal rate limit (§4.6) caps what a forged flood costs. There is no bearer token whose theft outlives its owner noticing, and nothing in the database worth stealing: the only credential material is public keys.

4.3 Human callers ​

A person is a paired ConsoleKey (§2.9): a non-extractable P-256 key their browser made, which a vaam admin phone (registerConsoleKey) or, as the break-glass, an operator on the command line (sms-gateway pair-console-key --user <email|id> --public-key <k>) vouched for. Every request the console makes is sealed in the browser (§4.0, docs/design/console-pairing.md). There is no TOTP, no cookie and no refresh token, and — unless the deployment serves human_login (VSA-4a, §4.10), and human_login.sso for single sign-on (VSA-4c, §4.10) — no password, no single sign-on and no OpenID session either: owner rulings Q1 and Q2 (2026-10-26) — phone pairing only, break-glass is command-line key enrolment — and the standing ban on refresh tokens. Owner ruling R5 (2026-10-27) put the password back behind configuration: a deployment that serves human_login lets a person with a password sign in to the console (and, with human_login.sso, through its identity provider, VSA-4c), and, since VSA-4b, the API accepts the access token that sign-in issues, as that person with the role their row says: except an owner or an admin, which act only from a paired console key (403 DEVICE_REQUIRED, §4.10). A paired console user is admin, never owner (R2), and a key lasts 30 days (R3).

The principal is built per request by the signer directory from the verified key: the person's User.id as sub, kind user, the role of their Role, and its permissions as perms (§5.3). A deactivated user or a Role keyed system/app is refused (§2.10). Standing is cached for 60 seconds, so deactivating a person takes effect within one TTL, and a revoked or expired key row on the next request.

How an account comes to exist, with nothing to sign in with:

  • provisionUser (console) and sms-gateway provision-user make a User row and print no credential. sms-gateway bootstrap --owner-email … --owner-display-name … does the same for the first owner. A key is then attached by pairing, or by pair-console-key. On a deployment that serves human_login, provision-user --password --password-out <file> and bootstrap --owner-password-out <file> also give the person a generated password, written to a 0600 file and never printed (§4.10).
  • registerConsoleKey finds or JIT-provisions the person through FederatedLink(issuer = "vaam", subject = <vaam user id>), as admin.

What VS-4 deleted (2026-10-27), and what came back ​

For the record, since earlier revisions of this section described all of it. VSA-3a (owner ruling R5) restored the machine half behind auth.methods and op: (§4.10): the OpenID Provider's /token, /jwks.json and discovery, the authkestra family, the sms-auth crate (stores, keys, client provisioning), token_rate_limit, rotate-signing-key and provision-client. VSA-4a (§4.10) restored the password half: /login, seed-console-client, the Argon2id password code and the password options of provision-user and bootstrap. VSA-4c (§4.10, "Single sign-on") restored the SSO half: /login/sso/* behind human_login.sso, the broker in sms-auth, sms-fake-oidc as its dev-only fake identity provider, and the eleven SMS_SSO_* flags as that block (the flags themselves stay gone: RETIRED).

  • The OpenID Provider: /token, /jwks.json, /.well-known/openid-configuration, the human /login and the Keycloak /login/sso/start and /callback, the authkestra dependency family, the sms-auth crate, sms-fake-oidc, token_rate_limit, and serve's --issuer flag and its start-up requirement for an active signing key.
  • The tables oauth_clients, oauth_signing_keys, client_assertions, user_credentials and sso_transactions (0009_drop_openid_provider, forward-only: the password hashes and private signing keys must not come back from a down). VSA-2 (owner ruling R5) created them again, empty, as 0012_restore_openid_provider; the rows did not come back (§2.8).
  • The commands rotate-signing-key, provision-client (which generated an RSA key server-side for private_key_jwt) and seed-console-client, and the password half of provision-user and bootstrap (all back: VSA-3a and VSA-4a).
  • The Argon2id password code (sms_core::password; back with VSA-4a).

4.4 Outbound webhooks: sealed or HMAC, per endpoint ​

An endpoint's deliveryMode (§2.7) says how its deliveries are authenticated. Both modes are always available; sealed is the recommended one, and hmac is the original scheme, kept because vsms serves other apps besides vaam and a receiver that verifies X-Sms-Signature today keeps working untouched. A deployment can switch hmac off with auth.methods (§4.9, VSA-1); vaam will, and then no endpoint can be created, switched or re-keyed in that mode and the worker sends none.

sealed (recommended)hmac (legacy)
What authenticates a deliveryA COSE_Sign1 request under the gateway's envelope signing key, bound to an audience, route and contractX-Sms-Signature, HMAC-SHA256 under the endpoint's own secret
Per-endpoint secretNonesecret, prevSecret during a rotation, secretRotatedAt
The receiver holdsThe gateway's two public keys, its audience, endpoint id and app id (nothing secret)The endpoint's secret (and the previous one during a rotation)
RotationThe gateway key's own rotation (§4.0), the same for every receiverrotateWebhookSecret: the old secret keeps verifying until the next rotation
Envelope{id, eventId, type, occurredAt, appId, endpointId, data}{id, type, occurredAt, data}, plus the four X-Sms-* headers
Choose it whenThe receiver can run a cratestack EnvelopeLayer, or the Rust/Node SDK's WebhookOpenerThe receiver already verifies X-Sms-Signature and has not moved yet

Choosing and switching are in docs/integrating.md ("Choosing a delivery mode"). The rest of this section is the sealed mode, then the HMAC mode.

Sealed mode ​

A sealed delivery is the same envelope as every other request in this system: a COSE_Sign1 message from a registered key, opened by a cratestack EnvelopeLayer. There is no shared secret, no X-Sms-Signature and no rotation overlap (owner's ruling for vaam: "keypairs everywhere"). Until VS-8 this was "the one place symmetric secrets remain, because here you're the sender", and it still is for an hmac endpoint, below.

mermaid
sequenceDiagram
    participant H as sms-worker hooks
    participant S as Sealer (sms-webhook)
    participant R as Receiver (EnvelopeLayer, Required)
    H->>H: claim attempt, fetch endpoint (audience, url), build the envelope JSON
    H->>S: seal(audience, url, envelope)
    S-->>H: COSE_Sign1 (fresh nonce, iat = now), signed with the gateway's current envelope key
    H->>R: POST url, application/cose (no redirect followed)
    R->>R: key, audience, POST, route, contract digest, payload type, iat, nonce
    R-->>H: 2xx (sealed under the receiver's key, never read)
    H->>H: attempt succeeded

The key. The gateway's own envelope signing key (--envelope-signing-key-file, §4.0), not a new one. A receiver therefore pins what it already pins to open vsms's answers: the {current, next} public keys sms-gateway server-keys prints, and a rotation of vsms's key is the same three steps for both. A separate webhook key would buy rotation independence nobody has asked for, and cost every receiver a second pin and every operator a second keypair. The trade-off, stated plainly: the key that signs every answer the gateway gives is now also readable by the worker, a second process on a second host, and whoever takes that process can sign anything the gateway can (an answer a client would believe, a delivery a receiver would open). A dedicated webhook key would confine a stolen worker to forging webhooks; the shared key does not. It was accepted (owner's ruling: sign with the gateway's envelope key) because the worker already reads and writes the same database the gateway's answers describe, and because a leak of either key means the same rotation; if a deployment ever needs the two to fail apart, a separate key is a second key file for the worker and a second pin per receiver, and the digest and the envelope do not change. The price of the shared key is that the worker now holds the private half: sms-worker --envelope-signing-key-file (or SMS_ENVELOPE_SIGNING_KEY_FILE) is what hooks seals with. It is needed only for sealed endpoints: a hooks worker without it starts (and says so in its log), delivers every hmac endpoint, and claims no sealed endpoint's attempt, so a deployment that has not moved a receiver to the seal is not made to mount a key it never uses. The mode is decided where the rows are chosen, not after: a claim reads the endpoints first (a table of a few rows per app) and, when it has to leave one out, asks for each remaining one's due attempts through webhook_endpoint_due_idx, so a keyless claim's cost follows the hmac work and not the sealed backlog, which it never reads (backends/crates/sms-worker/src/claim.md has the measurements, and why it is not a subquery on deliveryMode, nor one IN list, either of which Postgres answers by walking webhook_due_idx across every endpoint). That is why nothing about a sealed row is written: it stays pending, or failed resting out its backoff, without spending an attempt of maxAttempts, off the endpoint's breaker, and a worker that holds the key delivers it as soon as it runs, so mounting the key an hour late loses nothing. A keyless worker would otherwise leave them pending silently, so hooks logs a warn at most every five minutes with the count of due sealed attempts it cannot serve (those of an active endpoint: a deactivated endpoint's are claimed by no worker, key or not, and an open circuit is only a pause), naming --envelope-signing-key-file, and the binary logs one at start. A worker that has the key claims both modes. Only current is read; a worker without hooks reads no key and a dispatch-only node holds none. The file is loaded exactly as the gateway's are: a path, never in the database, mounted read-only and readable by uid 65532 and no one else, which is the gateway's and the worker's distroless nonroot user alike: mode 0400 and owned by 65532 in deploy/docker-compose.yml, mode 0440 with the pod's fsGroup: 65532 in the Helm chart. The chart mounts the worker's single file with subPath, and a Kubernetes Secret mounted that way never sees a later edit of the Secret: after a rotation, restart the worker together with the gateway (docs/runbooks/deployment.adoc, "Rotating the envelope keys"). The gateway's key never signs a request to the gateway (its ClientKey directory has no row for it), so sharing it across the two uses opens no cross-protocol replay: every seal also binds the audience, the route and a contract digest, and the two uses share none of the three.

The audience. WebhookEndpoint.audience (§2.7), required for a sealed endpoint and chosen when it is created. It is unique across endpoints (a second endpoint asking for a taken name is refused) and it is never vsms, the audience the receiver uses for its own requests to vsms (the API and a CHECK refuse it). Because it is a name anyone with webhook:manage may ask for first, and is free before the receiver's endpoint carries it, after that endpoint is renamed or deleted, and while someone squats it, it is defence in depth and not the receiver's proof of anything: the receiver compares endpointId and appId, below.

Which endpoint and app a delivery was made for. The signed envelope carries endpointId (WebhookEndpoint.id) and appId (WebhookEndpoint.appId). The seal binds one audience, not an endpoint or an app, and one key signs every app's deliveries. WebhookEndpoint create and update are allowed for the global roles (owner, admin, developer) with a free appId, so whoever holds webhook:manage can create an endpoint under any app's id, at a URL of their own: its deliveries carry the victim's appId, and while the audience name is free (after the victim's endpoint is deleted or renamed, before it exists, or while someone squats it) they pass the signature, the audience, the route, the digest, the clock and the app id at the victim's receiver, with a nonce it has never seen. endpointId is the binding: it is minted by the gateway, so it is the one signed value the attacker cannot choose, and a receiver serves exactly one endpoint, whose id it learns when the endpoint is created (the console panel shows it). appId additionally catches an endpoint registered under another app, and the unique audience is defence in depth. So a receiver compares endpointId and appId with its own and refuses any difference, before acting on the event. The Rust, wasm and Node WebhookOpeners take both ids as required arguments, have no constructor without them (an empty one is a configuration error) and answer a mismatch with the same coarse Unverified as a forged seal, and a verified body with no endpointId is Malformed; a receiver that opens deliveries with its own EnvelopeLayer makes both comparisons in its handler (sms-webhook's lib.md, "What a receiver configures", step 6).

The route. The whole path of WebhookEndpoint.url, bound as the route template (sms_webhook::delivery_route): a receiver mounts one literal POST route, and that route is exactly the URL's path, not a suffix of it. An ingress may rewrite the host or the scheme but never the path; one that strips a prefix makes the receiver match a route the seal was not made for, and every delivery is a 401. A URL with a query, credentials, a fragment, or a {, } or % in its path cannot be a sealed route. A sealed endpoint cannot be stored with one (§2.7), so the worker's "delivery not sealed" error, recorded as the attempt's error and never sent, is the second line of defence for a row it cannot trust; it is an ordinary failure like any other (counted toward maxAttempts, backed off, able to open the breaker and to go dead), which the URL CHECK makes practically unreachable. Redirects are not followed for a sealed delivery: a 307 would re-send sealed bytes to a host they were not made for, and is an ordinary failed attempt. (An hmac delivery does follow them, below.)

The contract. There is no schema op for a webhook, so the delivery declares its own: CONTRACT_DOCUMENT in sms-webhook names the method, the payload type and the envelope's fields, and the digest every delivery binds is SHA-256("vsms/webhook-delivery/v2\0" || document) = c61394a50e9f02ece411674fb3f398ae323e685b7a14f9c7b85538aa33a30423 (v1, which had no appId or endpointId, was never released). A receiver declares it for its route (("POST /webhooks/vsms", &[CONTRACT_SHA])). A change to the envelope moves the digest and the version in the domain tag. cargo xtask sdk-parity-check holds the document equal across sms-webhook, the Rust SDK and the wasm crate.

What a receiver configures (all public; none of it is a secret): the gateway's pinned keys, its audience (one only it can have: its own origin), its route, the digest above, EnvelopeMode::Required, application/json payloads, a nonce store its replicas share, and its endpoint id and app id, compared in the handler. sms-webhook's lib.md has the snippet; sms-webhook/tests/sealed_delivery.rs runs it against a real EnvelopeLayer. The Rust SDK's WebhookOpener and the Node SDK's WebhookOpener do the same without the layer.

Retries re-seal. A retry is a new sealed request: Sealer::seal runs again on every attempt, so the nonce is fresh and the timestamp current. A delivery whose backoff outlasts the receiver's 300 s skew (the schedule runs to 24 h) still opens; a re-sent body would have been refused as stale. The nonce stops a captured request being replayed; it is not the dedupe key. The envelope's id (WebhookAttempt.id) is, and it is the same across retries and across an operator's replay (§8.5).

The envelope. {id, eventId, type, occurredAt, appId, endpointId, data}. appId and endpointId are the endpoint's own (see above). eventId is WebhookAttempt.sourceEventId, which used to travel in the X-Sms-Event-Id header and was signed by the HMAC; a header is no part of a seal, so it is in the body now. The four X-Sms-* headers are no longer sent.

The answer. Delivered means 2xx, as before (410 deactivates, anything else retries). If the receiver is a cratestack layer its answer is a message sealed under the receiver's own key. vsms does not open it and treats it as proof of nothing: opening would need the receiver's public keys pinned per endpoint, which is another credential store, another console field and another rotation story, for a property nothing here rests on. A 2xx is the receiver's HTTP server accepting the request, authenticated by TLS to the endpoint's host, exactly the trust the HMAC scheme had (it never authenticated answers either). An unsigned 2xx is accepted on the same terms. Authenticating the receiver's answer is the natural follow-up if a deployment ever needs delivery receipts that survive a hostile network; it would add a pinned-keys column to the endpoint.

HMAC mode ​

The one place symmetric secrets remain, because here you're the sender and your customers verify. An hmac endpoint's delivery is what it was before sealed mode existed (#41): the same body, headers and signature, so nothing a receiver verifies today changed. Nothing differs from the pre-sealed worker that a receiver could tell (owner ruling R5: vsms serves apps beyond vaam, so an hmac delivery is byte-identical to what 29f93c4 sent): an hmac URL keeps whatever it was registered with (the route rule of §2.7 is a sealed endpoint's alone), and a secret that is the empty string is still signed with, as is an empty prevSecret (see below).

An hmac delivery follows redirects, as it always did. reqwest's default policy applies, up to ten hops, a 307/308 re-sending the signed body with its headers, because receivers sit behind redirects today (a bare host to www, http to https) and the HMAC scheme must survive unchanged for the apps that rely on it (owner ruling R5). The caveat is the old one: whoever controls an hmac receiver's redirects controls where its deliveries (the recipient and the text, signed) are sent, a trust the endpoint's registrant already extends by choosing the URL. A sealed delivery follows none (§4.4). Both share the 10 s timeout, which a redirect chain shares too.

text
POST /your/webhook
X-Sms-Event:     message.delivered
X-Sms-Event-Id:  c8f2a1...
X-Sms-Timestamp: 1753699200
X-Sms-Signature: v1=<hex>,v1=<hex during rotation>

signing string = v1 \n {timestamp} \n {eventId} \n {sha256(body)}
key = WebhookEndpoint.secret

The body is the four-field envelope {id, type, occurredAt, data} as JSON, the bytes that are both signed and sent. Two v1= values during rotation, oldest last; receivers accept if any verifies, which makes rotation a non-event for them. rotateWebhookSecret moves secret → prevSecret, generates a new one, stamps secretRotatedAt; the overlap window has no end of its own: the job that would clear prevSecret after 24 hours (cleanup_secrets's other half) is not built, so prevSecret keeps verifying until the next rotation. A sealed endpoint has no secret and rotateWebhookSecret refuses it with a 422. A secret the API takes is at least 32 characters (§2.7), and the ones this system mints are whsec_ and 64 hex characters, the format sms_webhook::generate_secret() produces. The generated create route takes the column as given, so the caller supplies the first one (the console generates it in the browser from the CSPRNG, and an API client does the same; a create with no secret is refused by the CHECK), while rotateWebhookSecret mints every later one server-side, and so does switchWebhookDeliveryMode when it switches a sealed endpoint to hmac. An hmac row whose secret is empty (a CHECK refuses NULL but not "", and a secret from before the length bound can be one) is delivered exactly as it was before: signed with the empty key, an empty prevSecret a second v1= value. Withholding it would change what a receiver built against that behaviour sees (owner ruling R5), so the worker logs a warn naming the endpoint (at most every ten minutes per endpoint, never a key's value) and tells the operator to rotate; docs/runbooks/deployment.adoc, "Before deploying: hmac endpoints with an empty or short key", has the query that counts them first. A signature under the empty key is one anyone can compute, and sms_webhook::verify skips an empty candidate, so a receiver using it rejects such a delivery; rotate twice when the secret itself is empty, since the first rotation moves the empty one into prevSecret. The API returns it to the three roles that can write the endpoint; the audit snapshots redact it (@sensitive). Retries and an operator's replay re-sign the same body, so the body (and X-Sms-Event-Id) is the same across them; X-Sms-Timestamp is the attempt's.

Send X-Sms-Event-Id and mean it — delivery is at-least-once and receivers need a dedupe key. sms-webhook's hmac_v1 module has the canonical string and the constant-time verify, and examples/node/webhook-receiver/src/signature.ts is an independent verifier that the shared cross-language vectors (backends/crates/sms-webhook/tests/fixtures/cross_language_vectors.json, computed with openssl) hold equal to the Rust one. Neither SDK ships an HMAC verifier; the Rust and Node WebhookOpeners open sealed deliveries only.

4.5 Idempotency ​

Superseded in part (VS-3, VS-4). The sealed surface authenticates with the envelope (§4.0), so the bearer-token fingerprinting this section's later paragraphs describe (verify_idempotency_principal, client_id_fingerprint reading a token's sub, JWKS verification) is the design the envelope replaced; the layer order that holds today is §4.0's. The /token flood protection they mention (#156, #168) is deleted with the route.

Mounted as of #153 — backends/crates/sms-api/src/router.rs's router() builds both layers below; nothing here is aspirational any more. SqlxIdempotencyStore is the only shipped implementation — no Redis or in-memory variant — which makes it correct for multi-replica by default. The real import paths, verified against the pinned cratestack =0.7.8, are one level shallower than an earlier revision of this section claimed: sms-api depends only on the cratestack (née cratestack-pg) facade crate, not on cratestack-axum directly, and the facade's own pub use cratestack_axum::*; re-exports idempotency/ratelimit as submodules of cratestack itself:

rust
use cratestack::idempotency::{IdempotencyLayer, IdempotencyStore};
use cratestack::ratelimit::{InMemoryRateLimitStore, RateLimitConfig, RateLimitLayer, RateLimitStore};
use cratestack::SqlxIdempotencyStore;
use cratestack_codec_json::JsonCodec;   // separate crate — add to Cargo.toml

let idempotency_store: Arc<dyn IdempotencyStore> =
    Arc::new(SqlxIdempotencyStore::new(db.pool().clone()));
let rate_limit_store: Arc<dyn RateLimitStore> = Arc::new(InMemoryRateLimitStore::new());
// #163: a second, independent store for the coarser, source-keyed layer
// below — never shared with the per-principal one, even though their key
// namespaces already can't collide.
let source_rate_limit_store: Arc<dyn RateLimitStore> = Arc::new(InMemoryRateLimitStore::new());

// `.layer()` composes outside-in: the *last* call is what a request meets
// *first*. The two RateLimitLayers sit outermost — source_fingerprint's
// coarser one first, then client_id_fingerprint's per-principal one — a
// cheap 429 before any body is buffered, any signature verified, or the
// idempotency store touched — then `verify_idempotency_principal` (only
// for Idempotency-Key-bearing mutations — see below), then
// IdempotencyLayer, then the RBAC layer and the generated router
// underneath both.
let router = cratestack_schema::axum::router(db, procedures, JsonCodec, auth)
    .layer(from_fn_with_state(rbac_state, enforce_route_permission))
    .layer(IdempotencyLayer::new(idempotency_store, idempotency_ttl)
        .with_principal_fingerprint(verified_idempotency_fingerprint))
    .layer(from_fn_with_state(idempotency_auth_state, verify_idempotency_principal))
    .layer(RateLimitLayer::new(rate_limit_store, rate_limit_config)
        .with_key_fn(client_id_fingerprint))
    .layer(RateLimitLayer::new(source_rate_limit_store, source_rate_limit_config)
        .with_key_fn(source_fingerprint));

Fresh key reserves and runs; repeat returns the cached response with Idempotency-Replayed: true; concurrent repeat gets 409 + Retry-After: 1; same key with a different body gets 422 idempotency_key_conflict. The request hash covers method, full path including query string, content-type and body.

Neither layer uses the upstream default fingerprint (a raw Authorization-header hash), and the two use different replacements — this split was itself a bug fix, found live in this PR's own review, not by inspection. Both layers sit outside the generated router, including outside its own GatewayAuth extractor, so by the time either layer's fingerprint function runs, no signature has been checked yet. cratestack-axum's buffer_and_persist_response (src/idempotency/complete.rs) caches a handler's response under any status code — no success check. Combine an unverified sub read with that fact and the attack is: send Authorization: Bearer <unsigned JWT, sub = victim_client_id> plus an Idempotency-Key the victim will predictably reuse (clientRef-shaped keys like rust-example-1 are exactly the guessable kind integrators use) — GatewayAuth rejects the forged token with 401 deep in the router, but the layer has already reserved, and then caches, that 401 under the victim's own identity and that key. The victim's later, correctly signed request with the same key replays the cached 401 instead of ever running sendMessage: a targeted denial of service requiring no valid credential at all, worse than the duplicate-send bug this section exists to prevent.

The fix, verified live (see below): IdempotencyLayer uses verified_idempotency_fingerprint, backed by a small middleware (verify_idempotency_principal) that re-runs GatewayAuth::authenticate — real, JWKS-backed signature verification, the same call rbac::enforce_route_permission already makes as its own established precedent for "a Tower layer re-authenticating ahead of the generated router" — and stamps the verified sub into a request extension only on success. A request that fails verification (or carries none) gets bucketed into a single shared "unverified" partition instead, never into a specific client_id's own bucket. RateLimitLayer keeps the cheaper client_id_fingerprint (an unverified sub read) — its own failure mode from a forged sub is bounded to self-throttling evasion, never poisoning another caller's cache, so the stronger (and costlier) guarantee isn't needed there. That asymmetry is deliberate, not an oversight: RateLimitLayer's per-principal budget is bypassable by anyone willing to forge a fresh sub per request — distinct from #156, the unauthenticated /token edge, and closed by #163, not merely tracked: see below.

#163: a second, coarser RateLimitLayer bounds the aggregate a flood of forged subs can reach — client_id_fingerprint itself is unchanged. source_fingerprint (backends/crates/sms-api/src/router.rs) keys on ConnectInfo<SocketAddr> — the real TCP peer, populated because backends/apps/sms-gateway/src/main.rs now serves this router through into_make_service_with_connect_info::<SocketAddr>() — rather than anything a forger controls. Deliberately not X-Forwarded-For: deploy/docker-compose.yml has two internal callers of this router, not one (Caddy, fronting every external caller, and admin, which talks to sms-gateway directly, bypassing Caddy, for its own server-side API calls), so a blanket "trust the header" config can't tell them apart, and Docker Compose assigns container IPs dynamically with no static-allowlist infrastructure to pin a single trusted hop against — trusting an unpinned header here would be worse than not using IP at all, the same failure mode deploy/Caddyfile's own zones deliberately avoid (see that file's own comment on why it keys token_per_ip on remote.host, never a client-supplied header). In this deployment's real topology that makes the coarse bucket collapse to one shared budget per internal peer — not per external client — which is coarse but bounded: however many forged subs an attacker mints, every one of those requests still arrives over the same accepted connection from the same peer, so they all draw from the same bucket, capping the aggregate at a fixed number instead of an unbounded supply of fresh ones. Sized wider than the per-principal budget for exactly that reason (sms_api::default_source_rate_limit_config, burst 1200 / 10 tok/s vs. 120 / 2 tok/s) — it has to cover every legitimate client sharing that peer at once, not one. The fix adds a layer rather than replacing the bypassable one, so honest per-principal throughput is untouched — proven live in backends/crates/sms-api/src/router.rs's own test suite: a flood of five requests, each with a different forged sub, against a 3-token source bucket gets 404, 404, 404, 429, 429 (the generated router's own 404 fallback for the first three — proof they cleared both layers, not proof of routing; the last two throttled despite carrying brand-new subs no per-sub bucket had ever seen), while ten requests from one honest client at default production budgets never see a 429.

Verification only runs for requests that both qualify by HTTP method (cratestack_axum::idempotency::is_idempotent_target_method — POST/PATCH/PUT/DELETE, reused directly so the gate can't drift) and actually carry an Idempotency-Key header — IdempotencyService::call never reads the fingerprint closure otherwise. Measured live, not assumed: 30 real POST /$procs/sendMessage calls with a valid token and no Idempotency-Key averaged 9.75ms; 30 more, identical except a fresh Idempotency-Key each time, averaged 13.54ms — a ~3.8ms delta against a warm JWKS/AppClient cache, consistent with one extra RS256 verify plus a cached lookup, not a network round trip. On the one PROVIDER_WRITE_ROUTES route that also carries Layer 2 (§5.1), an Idempotency-Key-bearing request now pays a third authenticate call — an already-accepted double-verification made triple, narrow and rare enough (one route, one header) to accept rather than optimise away.

Proven against a real gateway, not merely reasoned about: a forged token claiming sub = <victim's real client_id> plus a predictable Idempotency-Key got 401, cached under principal unverified; the victim's own later, correctly signed request with the identical key still ran sendMessage for real (a genuinely new Message row, not a replayed 401); a second replay of the victim's own request, from a separate process with a freshly minted token, correctly returned Idempotency-Replayed: true with no third row created.

A missing Idempotency-Key bypasses the layer entirely — document loudly that OTP callers must send one, because a retried OTP send without a key is a second SMS and a second charge. Cluster-wide idempotency replication is explicitly not implemented; the SQL store is shared state, which is what makes multiple replicas safe, but don't assume more.

The messages_app_idem_key unique index (driven by sendMessage's clientRef, §3's own procedure walkthrough) is a second, independent line of defence at the data layer — independent in the literal sense that it runs regardless of whether a caller ever sends Idempotency-Key at all, not a fallback for when the HTTP layer is absent.

SqlxIdempotencyStore::ensure_schema() is never called by sms-gateway itself. The cratestack_idempotency table it writes is library bookkeeping, not part of schemas/vsms.cstack, but the deploy path's own rule against runtime DDL applies to it anyway: backends/migrations/postgres/0003_idempotency_table/up.sql is the one committed, drift-tested copy of IDEMPOTENCY_TABLE_DDL's exact text — living in the same directory, same directory-per-migration shape as 0001_init/0002_bootstrap even though cratestack migrate diff never generated it — and every path that needs the table applies that same file rather than each keeping its own copy: backends/apps/sms-migrate's advisory-lock-guarded one-shot migrate job (discovers and embeds every backends/migrations/postgres/*/up.sql via a build script and applies each with cratestack::sqlx::raw_sql, tracked in that binary's own schema_migrations bookkeeping table under the migration's own directory name, 0003_idempotency_table, so the serving process needs no CREATE TABLE privilege and two replicas starting together never race a DDL statement), ci/apply-migrations.sh (the CI/scratch-database path), and backends/crates/sms-test-support (the live-suite harness's own per-binary template database) — all three now discover it by the same directory walk, with no special case for this file. backends/crates/sms-api/tests/idempotency_table_ddl_matches_cratestack.rs asserts the copy hasn't drifted from the pinned library's constant.

idempotency_ttl/rate_limit_config above are the YAML keys api.idempotency_ttl_secs and api.rate_limit.{burst, refill_per_second} (§9.4; they were sms-gateway serve flags until VSC-1), defaulting to this section's own figures (24h / burst 120 / 2 tok/s) — configuration, not a feature flag; every construction site (the real binary and the live HTTP test suites) shares sms_api::DEFAULT_IDEMPOTENCY_TTL/sms_api::default_rate_limit_config() as the one source of truth for those defaults. source_rate_limit_config (#163) is the same shape, one level up: api.source_rate_limit.{burst, refill_per_second}, defaulting to sms_api::default_source_rate_limit_config() (burst 1200 / 10 tok/s). A test in sms-gateway holds the YAML base file to those sms_api constants.

4.6 Rate limiting ​

Three limiters on the API, easy to conflate, and five more on the OpenID Provider's routes once op: is written (VSA-3a, VSA-4a, and VSA-4c for the two single-sign-on rows; VS-4 had deleted them with the routes). The second group is the last five rows:

LimiterWherePurpose
Per-principal ingressRateLimitLayer (×2) on sms-apiStop a buggy — or forging — caller flooding you. 429 + Retry-After. Mounted as of #153; the second, coarser layer as of #163 — see §4.5.
Per-MSISDN OTPin sendMessageStop SMS-pumping fraud. E.g. 3 OTP / 10 min / number, 10 / day.
Per-provider egress TPSin the worker's dispatch roleRespect Orange's hard 5 TPS cap.
/token per client / per codetoken_rate_limit in sms-gatewayA flood of token requests: per client_id for client_credentials, per code for authorization_code (the console's client_id is public, so a bucket on it would be every operator's to empty).
/login per email, aggregatelogin_rate_limit in sms-gatewayArgon2 CPU and password guesses: a bucket per email address, then one for every sign-in together. No bucket per source address: behind the proxy and the console's server the gateway sees one peer for everybody.
/login and /token per sourceCaddy auth_per_source zoneThe one source the application cannot tell apart: keyed on the real client address, an IPv6 /64, where that address is known.
SSO start and callbackhuman_login.sso.rate_limit in sms-gatewayOne global bucket for each of /login/sso/start and /login/sso/callback (a start writes a row, a callback reads one), fail closed. Global because the gateway sees one peer, so a single source can empty it.
SSO per sourceCaddy sso_per_source zoneThe bound on that one source: keyed on the real client address, an IPv6 /64, over /login/sso/*, 30 a minute by default.

RateLimitConfig::new(burst, refill_per_second). InMemoryRateLimitStore is the only shipped store, with an unbounded key map and no persistence — single-replica dev only. For production implement the RateLimitStore trait (which is #[async_trait]) against Redis or Postgres. The bucket is wall-clock-driven, so a process pause longer than one fill window grants a fresh burst on resume.

sms-gateway's current deployment (deploy/docker-compose.yml) runs exactly one gateway replica, so InMemoryRateLimitStore's per-process nature is correct today, not a gap papered over — a second replica would need the Redis/Postgres-backed store this paragraph already says doesn't exist yet, before horizontal scaling is safe.

Per-principal ingress was keyed on an unverified token claim alone, on purpose — see §4.5's own explanation of why that's a deliberately different (and weaker) tradeoff than IdempotencyLayer makes — and #163 closed the consequence, without changing that tradeoff. Concretely: a caller willing to forge a fresh sub claim per request gets a fresh, full bucket every time from client_id_fingerprint alone. That's still bounded to self-throttling evasion, not a path to harming another caller — the property that made paying real-signature-verification cost on every request unnecessary — but it left the aggregate unbounded, which is what #163's second, ConnectInfo-peer-keyed RateLimitLayer bounds. See §4.5's own fuller writeup for the mechanism and the topology reason it doesn't trust X-Forwarded-For.

The per-MSISDN OTP limiter is the one that protects your money. SMS pumping — cycling OTP requests against premium-rate ranges — is the most common way a new gateway loses its balance overnight.

4.7 Audit ​

@@audit writes to cratestack_audit in the same transaction as the mutation, capturing event_id, model, operation, primary key, actor, before/after snapshots, request_id and occurred_at. Redaction happens before the row is written and is irreversible: @pii → "<redacted: pii>", @sensitive → "<redacted: sensitive>", @server_only → omitted.

The only shipped sinks are NoopAuditSink and MulticastAuditSink; anything else you write against AuditSink. The database table is canonical; sinks are best-effort projections.

What CrateStack does not give you: signed audit chains, immutable storage, HSM key management. If you need tamper-evidence — and for an OTP gateway under a regulator you plausibly do — add a nightly job that hashes the day's audit rows into a Merkle root and publishes the root somewhere append-only. Roughly 150 lines, and it converts "trust my database" into "verify my chain".

4.8 Secrets ​

  • Provider credentials: age/sops-encrypted files, or Vault. Provider.credentialRef holds only the pointer. The YAML configuration reads each secret as a ${VAR} placeholder (§9.4), and under livemode a literal secret in the file is refused at boot and by cargo xtask config-literal-secrets.
  • App clients and console users hold no secret at the gateway: only their public keys are stored (ClientKey, ConsoleKey). The private halves stay with the caller or in the browser's non-extractable key store. A token client (§4.10) is the same: OauthClient stores only the public JWK, and provision-client writes the private key once, to a 0600 file created with O_EXCL.
  • The OpenID Provider's own signing keys (OauthSigningKey.privateKeyPem, @sensitive) are the one private key vsms stores. They are database rows, never a file and never in the YAML (op: holds no secret), readable by the internal system context alone and by no route (@@internal("all")).
  • MSISDN hash pepper: encrypted store, loaded at boot, never plaintext in the database. Written in the YAML as security.hash_pepper: ${SMS_HASH_PEPPER} (§9.4): the file holds the placeholder, never the value. An hmac webhook endpoint's secret is a column (WebhookEndpoint.secret, @sensitive so audit snapshots redact it); a sealed endpoint has no secret (§4.4).
  • Envelope signing keys: two Ed25519 PKCS#8 PEM files (--envelope-signing-key-file, --envelope-next-signing-key-file), mounted read-only for uid 65532 (0400 and owned by it in deploy/docker-compose.yml, 0440 with fsGroup: 65532 in the Helm chart), never in the database. Rotate by promoting next (§4.0). The worker's hooks role loads the current one too, to seal webhook deliveries (§4.4).
  • No secret ever appears in an API response. @server_only enforces that mechanically, with one deliberate exception: an hmac webhook endpoint's secret/prevSecret are returned to the three roles that can write the endpoint (§2.7).

4.9 Authentication methods are deployment configuration (VSA-1) ​

vsms serves more than one application, and which ways of authenticating a deployment accepts is a property of the deployment, not of the binary (owner ruling R5, 2026-10-27). It is one YAML key, the same in vpay:

yaml
auth:
  methods: [keypair, hmac_webhooks]   # absent means all four
MethodWhat it permitsState in this release
keypairrequests signed by a registered key and the sealed answers (§4.0, §4.2, §4.3)served; the envelope is Required alone, Optional beside tokens, and when off a signed body is a 415
machine_tokensAuthorization: Bearer from the deployment's own OpenID Providerserved when op: is written (VSA-3a, §4.10), and the API accepts its tokens beside the envelope (VSA-3b)
hmac_webhookswebhook endpoints delivered with X-Sms-Signature (deliveryMode: hmac, §4.4) and their secretsserved, and gated by this switch
human_logina person's password sign-in to the console, and single sign-on under human_login.sso (VSA-4c)served when human_login: and op: are written (VSA-4a, §4.10); the API accepts the access token, never as owner or admin (VSA-4b)

Absent means all four, so a configuration written before the key existed gets what it always got, and every deployment of an app other than vaam keeps working untouched. vaam writes methods: [keypair] (its overlay is deploy/values/vsms.<mode>.yaml in vaam-apps, not here). The rules, one sms_config::ConfigError variant each, are backends/crates/sms-config/src/auth.md's:

  1. an empty list or a null (methods: with every item commented out reads as a null, not as absent: the operator was narrowing the list, and all four is the opposite of what they meant) is refused (NoAuthMethods);
  2. an unknown name is refused (UnknownAuthMethod);
  3. a duplicate is refused (DuplicateAuthMethod);
  4. a list that permits nothing able to authenticate a caller is refused (NoWayIn): vpay's rule is neither keypair nor machine_tokens (human_login signs a person in to the console and does not count). [hmac_webhooks] alone, and [hmac_webhooks, human_login], would be a gateway that answers every API call with the unsigned 401, and are refused. Since VSA-3b the API accepts a token, so machine_tokens is a way in and [machine_tokens] alone, with its op: block, boots; a deployment on it has no signed way in, so a signed request is a 415 and registering a key or pairing a console is a 422 auth_method_disabled (§4.10, keypair_gate.md), pairing a console until VSA-5. [keypair, machine_tokens] boots and serves /token;
  5. machine_tokens named explicitly with no op: block is refused (AuthMethodNotConfigured). The absent default never raises it: it permits the method and serves it only where its block is;
  6. human_login named explicitly with no human_login: block is refused (AuthMethodNotConfigured). The absent default never raises it either;
  7. a human_login: block with no op: block is refused (AuthMethodNotConfigured, naming op) — under the absent default too: the sign-in rides the provider, so the block would be accepted and serve nothing;
  8. a human_login: block on a list that does not permit human_login is refused (AuthMethodDisabled): dormant configuration is how an operator comes to believe a door is shut that is open;
  9. an op: block on a list that permits neither machine_tokens nor human_login is refused (AuthMethodDisabled), naming both ways out. op: serves either method, so it is refused only when neither is permitted.

(AuthMethodUnavailable, VSA-1's rule for a method with no code, and AuthMethod::is_restored are gone: with human_login back every method has code.)

vpay has the same two for human_login (staff_auth, dashboard_client); vsms's block is human_login: (VSA-4a; backends/crates/sms-config/src/human_login.md). keypair has no YAML (its keys are database rows) and hmac_webhooks has none either (endpoints are rows, so the gate is at run time, not at boot).

Permitted is not served. AuthMethods::served is what a client may be told: the permitted methods whose block is present — machine_tokens with op: (AuthMethods::machine_tokens_served), human_login with human_login: and op: (AuthMethods::human_login_served), the questions the router, the grant gate, the discovery document and the commands ask. The absent default permits four and serves two on a deployment that wrote neither block, three with op:, and all four with both. The provider is mounted exactly when one of the two is served (AuthMethods::op_served).

mermaid
sequenceDiagram
    participant C as Console or SDK
    participant G as sms-gateway (outside the envelope)
    C->>G: GET /.well-known/vsms-auth (unsigned)
    G-->>C: {"methods": ["keypair", "hmac_webhooks"]}
    Note over C: lists what the deployment offers, not what a caller may use

GET /.well-known/vsms-auth is merged beside /healthz, outside the envelope: a client has to be able to ask before it knows how to sign, and the answer is a list of public names. vpay's /.well-known/vpay-auth has the same shape. serve also logs, at boot, the permitted and served methods and whether the key was written, so an operator who believes they narrowed the list sees that they did not.

With machine_tokens off ​

/token, /jwks.json and /.well-known/openid-configuration are not mounted: each is the router's 404, indistinguishable from any path that never existed, and the discovery document does not list the method. sms-gateway provision-client and sms-gateway rotate-signing-key read the configuration before the database and refuse with auth_method_disabled: this deployment issues no tokens (…); use register-client to register a signing key instead, writing nothing. register-client is what every deployment serves. The same switch, AuthMethods::machine_tokens_served, decides all of it, so the commands and the server cannot disagree. (with_the_method_off_nothing_is_mounted_and_the_commands_refuse runs both halves for the absent default and for [keypair]; the_token_commands_refuse_before_the_database_when_the_method_is_off runs them with no database at all.)

With human_login off ​

POST /login is not mounted, /token refuses authorization_code with unsupported_grant_type (the grant gate, §4.10), and the discovery document does not list the method. seed-console-client, provision-user --password and bootstrap --owner-password-out read the configuration before the database and refuse with auth_method_disabled: this deployment signs no person in (…); a person gets in by pairing a browser key (pair-console-key), writing nothing and creating no file. The same switch (AuthMethods::human_login_served) decides all of it. (the_password_commands_refuse_before_the_database_when_human_login_is_not_served runs the three with no database at all.)

With hmac_webhooks off ​

mermaid
sequenceDiagram
    participant O as Operator (webhook:manage)
    participant G as Gateway (RBAC, then the gate)
    participant P as Procedures
    participant H as hooks worker
    participant E as hmac endpoint's receiver
    O->>G: POST /webhook_endpoints {deliveryMode: hmac}
    G-->>O: 422 auth_method_disabled, nothing written
    O->>P: switchWebhookDeliveryMode to hmac / rotateWebhookSecret / replayWebhookAttempt of an hmac endpoint's attempt
    P-->>O: 422 auth_method_disabled, nothing read
    O->>P: switchWebhookDeliveryMode to sealed
    P-->>O: 200, the way out
    H->>H: claim: hmac endpoints are not claimable while the method is off
    Note over H,E: the attempt stays pending, nothing is written, nothing is sent, the breaker is not touched
    O->>P: switchWebhookDeliveryMode to sealed (or permit hmac_webhooks and restart)
    H->>E: the next claim takes the attempt: sealed, or signed again
  • The API. Creating an endpoint with deliveryMode: hmac, updating one to hmac or giving it a secret or prevSecret, switchWebhookDeliveryMode to hmac, rotateWebhookSecret and replayWebhookAttempt of an hmac endpoint's attempt all answer 422, the envelope's code VALIDATION_ERROR (cratestack gives every 422 that one code) and a message that starts with the stable word auth_method_disabled:, which a client matches the way it matches idempotency_key_conflict. The procedures check a flag (the replay after it has found the attempt and read its endpoint's mode, so a sealed endpoint's attempt replays as ever; a replayed hmac attempt would be reset to pending and held back, answering 200 and sending nothing). The generated POST and PATCH are not ours (and do not pass their errors through map_database_error), so a layer, sms_api::hmac_gate::enforce_hmac_gate, decodes the body with the router's own codec into the handler's own input type, as the owner-grant guard does for User, and fails closed on a body it cannot read. It sits under the RBAC layers, so a caller without webhook:manage hears that first. A sealed create, an update that touches neither the mode nor a secret, a switch to sealed and a DELETE still work, so an operator can clean up what the method left behind. A test fails when the schema grows a write route under /webhook_endpoints that is neither guarded nor classified. The CHECKs of 0010_sealed_webhooks remain the backstop for every other writer, psql included.
  • The worker. No hooks worker claims an hmac endpoint's attempt while the method is off: the endpoint is excluded from the claim exactly as a sealed endpoint is for a worker without the signing key (VS-8), in both claim paths (the single time-ordered read when every endpoint is claimable, the per-endpoint fan-out otherwise), so a deep hmac backlog cannot fill a window and starve the sealed endpoints behind it. Nothing is written for such an attempt: it stays pending (or failed and resting, or delivering with an expired lease, whatever it was), is not counted toward the dead-letter limit (counting happens at a claim and there is none) and is off the endpoint's breaker (no request was made, so none failed). When the method is permitted again (changing auth.methods is a restart) the next claim delivers it, with every try it had and no replay. After switchWebhookDeliveryMode to sealed the pending attempts are delivered sealed: the switch rewrites the endpoint only, and the claim, the delivery and the envelope all read the endpoint's mode afresh for each attempt. sealed attempts beside them are delivered as before. A worker with the method off and no signing key serves nothing and warns at start (it does not refuse to start: the configuration is the deployment's, and another worker of it may hold the key). Every five minutes a worker with the method off warns with held, the number of due attempts it is holding back: counted per hmac endpoint, from the index alone, over at most 10 001 rows in all, so the cost does not grow with the backlog, and more than 10000 beyond that. When the backlog is released, the held attempts are delivered oldest first, ahead of the fresh ones, with occurredAt the delivery time; docs/runbooks/deployment.adoc has the SQL to delete the stale ones first. backends/crates/sms-worker/src/hooks.md, "When hmac_webhooks is off", has each choice and its reason.
  • Boot. serve and every hooks worker log how many hmac endpoints exist: an info line when the method is permitted or nothing is left behind, a warn when endpoints remain and the method is off, naming what to do (switch them to sealed, or delete them).

4.10 The OpenID Provider, machine tokens and the console sign-in (VSA-3a, VSA-4a) ​

Owner ruling R5 (2026-10-27) restored the provider that #446 deleted, behind auth.methods and a YAML block per method. VSA-3a restored the machine half and its two commands; VSA-3b makes the API accept what it mints, as a second way in beside the signed envelope; VSA-4a restores the console's password sign-in on the same provider (last subsection). The provider is mounted when machine_tokens or human_login is served; each grant is served only by its own method.

yaml
op:
  issuer: https://vsms.example.org     # iss of every token; no trailing slash; https under livemode
  token_rate_limit:                    # per client_id (per code for authorization_code), in front of /token (#168)
    burst: 10
    refill_per_second: 0.3333333333333333
  signing_key:
    refresh_interval_secs: 60          # how soon a running serve sees a rotation
    rotation_overlap_secs: 1800        # a rotated-out key keeps publishing; must exceed refresh_interval_secs + 900

These are 29f93c4^'s flags moved into the YAML the way VSC-1 moved the others: issuer was --issuer/SMS_OIDC_ISSUER, the bucket was --token-rate-limit-burst/--token-rate-limit-refill-per-second, and the two timings were constants. There is no secret in the block. It is served when auth.methods permits machine_tokens and op: is present (§4.9); backends/crates/sms-config/src/op.md has the rules.

mermaid
sequenceDiagram
    participant Op as Operator
    participant CLI as sms-gateway operator commands
    participant DB as Postgres (system context, delegates)
    participant G as sms-gateway serve
    participant C as Machine client
    Op->>CLI: rotate-signing-key
    CLI->>DB: create OauthSigningKey (RS256), deactivate the old one with an overlap
    Op->>CLI: provision-client --app-id --scope --key-out
    CLI->>DB: AppClient + OauthClient (public JWK only), one transaction
    CLI-->>Op: private key, once, 0600, to --key-out
    G->>DB: load_signing_keys (refuses to start with none), then every refresh_interval_secs
    C->>G: POST /token (client_assertion signed with its key)
    G->>DB: find OauthClient, record the assertion's jti (unique)
    G-->>C: access_token (RS256, 15 minutes), no refresh_token
    C->>G: GET /jwks.json
    G-->>C: every still-valid signing key
mermaid
stateDiagram-v2
    [*] --> Off: no op: block, or machine_tokens not permitted
    Off --> Refusing: provision-client, rotate-signing-key
    Refusing --> Off: auth_method_disabled, nothing written
    [*] --> NoKey: op: written and permitted
    NoKey --> NoKey: serve refuses to start (run rotate-signing-key first)
    NoKey --> Serving: rotate-signing-key
    Serving --> Serving: rotate-signing-key (the previous key overlaps in the JWKS)
  • A machine's grant is client_credentials with private_key_jwt, and nothing else. 29f93c4^ also advertised authorization_code and refresh_token; those were the console's password login (#194), which is the last subsection's. authkestra's client_credentials handler returns refresh_token: None unconditionally, so a machine client never held a refresh token, and provision-client registers client_credentials alone, so the handler refuses any other grant for it. The discovery document lists exactly the grants served (["client_credentials"] for a deployment that serves only machines) and a_served_provider_mints_a_verifiable_token_with_no_refresh_and_follows_a_rotation asserts it, the missing refresh_token and the refusal of the refresh grant, which since VSA-4a is the grant gate's unsupported_grant_type (owner ruling R5, question 8: there is no refresh grant).
  • No shared secret. OauthClient has no secret column: it holds the client's public JWK. provision-client generates the RSA keypair, stores the public half and writes the private PEM once to --key-out (0600, O_EXCL, refusing an existing file), never to stdout. A spent assertion's jti is a ClientAssertion row (create plus catching 23505), so a captured assertion is single-use.
  • The tables are reached through delegates under the system context. OauthClient, OauthSigningKey and ClientAssertion stay @@internal("all"), which removes routes and not the in-process delegates, so db.oauth_signing_key() and the rest exist and every call is sms_api::system_context(..), the only context their policies admit. No raw SQL, no new route, no schema change; provider_models_are_unreachable.rs stays green unchanged and the_restored_tables_answer_the_system_context_and_refuse_everyone_else runs the policy against the database.
  • serve fails at start with no active signing key, not on the first /token. A fresh deployment runs rotate-signing-key once first (run --rm, never exec: waiting for health before rotating deadlocks).
  • Rotation reaches a live process. A running serve reloads the keys every refresh_interval_secs; the previous key keeps publishing in /jwks.json for rotation_overlap_secs, which must exceed refresh_interval_secs plus the 15-minute access-token lifetime (sms_config refuses a smaller value): a running serve keeps signing with the old key until its next refresh, so the last old-key token is issued as late as T + refresh_interval_secs and lives until T + refresh_interval_secs + 900 s, and the old key stops being published at T + rotation_overlap_secs.
  • aud is client_id unless the client asks for another audience (allowed_audiences), which is why the bearer door requires it to contain sub rather than a fixed name (below).
  • The /token bucket is keyed on client_id for client_credentials, the dimension the edge cannot read from a form body, and on the SHA-256 of the code for authorization_code; it sits in front of /token only; a body it cannot read is bucketed, never skipped. The console's client_id is public, so a bucket on it would let anyone spamming code=x expire every operator's real code behind a 429; a bad code now spends only its own bucket.
  • Tokens carry no contract-lock protection and no response sealing: the 426 for an op whose shape is gone applies to sealed callers only, and a token caller's answers are plain JSON.

The bearer door (VSA-3b, a person's token VSA-4b) ​

The API accepts Authorization: Bearer <access_token> when machine_tokens or human_login is served, beside the signed envelope (sms_api::bearer, sms-gateway's op_bearer.rs). A machine's token (client_credentials) is the VSA-3b realm; a person's (authorization_code after POST /login) is VSA-4b's, and each realm is served by its own method with no key of its own. The envelope's mode follows the doors (§4.0): any token served makes it Optional beside the keypair. Between it and the per-principal limiter sits enforce_bearer_door.

mermaid
sequenceDiagram
    participant C as Caller
    participant E as EnvelopeLayer (Optional)
    participant B as Bearer door
    participant V as OpBearerVerifier
    participant K as OP keyring (same one /jwks.json serves)
    participant D as SignerDirectory
    participant R as Limits, idempotency, RBAC, policies, handler
    C->>E: POST /$procs/sendMessage, Authorization: Bearer
    E->>B: unsigned request (never sealed, no contract lock)
    B->>V: verify(token)
    V->>K: key for the header's kid
    alt unknown kid
        V->>V: reload the key set once (single flight, 2 s deadline, at most every 5 s)
    end
    V-->>B: Machine(client id, scope) or Person(User.id, console id), or a refusal
    B->>D: bearer_standing: machine, OauthClient active, AppClient live and the same client, and for a person the console client live and public, User and Role from the shared cache
    D-->>B: App standing, scopes are the AppClient's that the OauthClient lists, narrowed by the claim, never console:pair, or User standing, never owner or admin (403 DEVICE_REQUIRED)
    B->>R: VerifiedPrincipal client:id or user:id, BearerCaller
    R-->>C: plain JSON
mermaid
stateDiagram-v2
    [*] --> Signed: COSE body
    [*] --> Unsigned: no COSE body
    Signed --> [*]: Authorization present, 400
    Signed --> Admitted: otherwise
    Unsigned --> Refused: no bearer, wrong scheme, two headers
    Unsigned --> Verified: signature, iss, exp, nbf, kid, no cnf, and per realm: a machine's aud contains sub; a person's aud is the console alone, sub is the identity's external_id, a scope and no nonce
    Verified --> Refused: machine, client inactive, no live AppClient, another client's
    Verified --> Refused: person, console not a live public unlinked client, user unknown or inactive, reserved role
    Verified --> Forbidden: person is an owner or an admin
    Verified --> Admitted: standing read from rows
    Refused --> [*]: one 401, the layer's own body
    Forbidden --> [*]: one 403, DEVICE_REQUIRED
    Admitted --> [*]: client:id or user:id, one bucket, one idempotency namespace
  • A token names a client and narrows; it never grants. iss must be op.issuer, exp and sub are required, nbf is honoured, with 60 seconds of leeway; the header must carry a kid (never "the first key"); an identity claim (a person's token) is a HumanToken refusal where no person realm is served and the person realm's token otherwise, and no identity is a MachineToken refusal where only people are served; a cnf claim (a token bound to a key, whose proof this door never checks) is refused; and a machine's aud must contain sub (owner decision: no fixed vsms audience, so SDKs that already talk to the provider keep working). The role, app and scopes come from OauthClient and AppClient: a token acts with the AppClient's scopes that the OauthClient row also lists (/token checks only a requested scope against that row, so a token that asks for none would otherwise inherit a scope added to the AppClient alone), the token's scope claim is intersected with that when present, and console:pair is removed whatever any side says. perms are never read from a token.
  • Standing is read from rows on every request (OauthClient.active, uncached: a switched-off client stops on the next call; the AppClient for 60 seconds, as the signed door's). A retired or inactive AppClient, a client with none and an OauthClient linked to another client's AppClient all have no standing.
  • The keys are the provider's own, in process. The verifier reads the keyring /jwks.json publishes from, under the same lock, so the two cannot disagree; there is no HTTP fetch of our own JWKS and no kid-less fallback. An unknown kid forces one reload of the key set, single flight, with a two-second deadline and at most once every five seconds, so a rotation another process ran is seen without waiting for the periodic refresh, a flood of invented kids costs one read per interval and a database that stalls cannot park the requests behind it. The price is that a flood-triggered reload can leave a key another replica just rotated in refused for up to five seconds; it heals itself on the next reload.
  • One principal. Both doors record client:<clientId> for a machine and user:<User.id> for a person (their console key's name too), so a caller that uses both shares one rate-limit bucket and one Idempotency-Key namespace. A refusal is one 401 with the layer's body, byte for byte, whatever failed (the reason is logged at debug).
  • A token caller gets plain JSON. Unresolved routes stay Required (a route the layer cannot bind fails closed with a 500 in every mode), and an unsigned request's answer is never sealed, whatever its Accept says: no response seal and no contract-lock 426.
  • A person's token (VSA-4b). human_login served adds the person realm to the one verifier (sms-gateway's Realms, built from what is served and from nothing else). A person's access token is the one POST /token mints after POST /login: sub is the User.id, identity.external_id is the same, aud is the console's client_id and nothing else (a list of one is the same; any other list is refused), scope is the console's openid profile. The ID token beside it is refused (IdToken): it has the same issuer, subject, audience, identity, key and lifetime and differs only by carrying no scope (and a nonce when the sign-in sent one), and it is the token the console hands to a browser and may log. authkestra stamps no typ or token_use, so the test is a heuristic (a non-blank scope and no nonce); the cost is that a sign-in that asks for no scope gets an access token the API refuses, and the console asks for openid profile. The standing is read from rows: the console OauthClient on every request (active, tokenEndpointAuthMethod = none, linked to no AppClient, so a machine client's id in the audience is refused), then the User and Role through the same 60-second cache a paired console key's owner is read through (a deactivation or a promotion takes effect within one window, the window a key has), with the same reserved-role refusal. The person acts with the role's perms and no scope, exactly as with their console key; the OIDC scope is never read as a permission. It is VerifiedPrincipal("user:<User.id>"), the name their console key gets, so token and key spend one bucket and one Idempotency-Key namespace.
  • Owner and admin never act from a token (VSA-4b). A User has one roleKey and hasRole matches one string, so a token cannot carry "the role minus the powerful part": a person whose role is in DEVICE_ONLY_ROLES (owner, admin) behind a person's token is 403 {"code":"FORBIDDEN","message":"this account's role acts only from a paired console key; a signed-in session cannot carry it"} (DEVICE_REQUIRED, which names no role) and acts only from a paired console key, a device the server has seen. It is a 403 and not the coarse 401: the token and the account are real, and the answer is about what the token may do. refuse_device_only_role is one function called from two places, SignerDirectory::bearer_standing where the standing is built and GatewayAuth::authenticate on the standing the door recorded. The first runs after the standing-cache read, on the value just read, which is what covers the cache shared with the signed door (where an admin is legitimate); the second is a safety net that catches the first being moved into the cache loader, and any standing that reaches the provider another way. Keep both. The cap is keyed on the credential (a person's token), not on provider_id, so a single sign-on token (VSA-4c) is capped the same: it is issued with the same provider_id and User.id, and the_sso_access_token_is_the_person_and_a_promoted_admin_gets_403_device_required signs in through a fake identity provider and promotes the person.
  • keypair off. [machine_tokens] alone sets the layer Off: a signed body is a 415, registerAppClientKey and registerConsoleKey are a 422 auth_method_disabled (sms_api::keypair_gate), and register-client and pair-console-key refuse the same way when given a configuration that omits keypair. Pairing a console is unavailable on such a deployment until VSA-5, and so is [machine_tokens, human_login]'s owner and admin: they cannot pair a key, and a token cannot carry their role.

The console sign-in (VSA-4a) ​

human_login is served when auth.methods permits it and both human_login: and op: are written (§4.9): a person signs in through the provider's authorization_code grant with PKCE. The block names the one public client the console is and the limit in front of the password check:

yaml
human_login:
  console_client:
    client_id: sms-console                 # 8 to 64 characters; the only client_id /login accepts
    redirect_uris:                         # exact match; https and no stub host under livemode
      - https://admin.vsms.example.org/api/auth/callback
  login_rate_limit:                        # both before any password is hashed
    per_email:                             # one bucket per email address
      burst: 5
      refill_per_second: 0.1
    aggregate:                             # one bucket for every sign-in: the Argon2 CPU budget
      burst: 10
      refill_per_second: 1.0
mermaid
sequenceDiagram
    participant U as Console (its server, never a browser)
    participant L as POST /login
    participant R as Limiter (email, source)
    participant A as authenticate_user
    participant H as handle_authorize (authkestra)
    participant T as POST /token
    U->>L: email, password, client_id, redirect_uri, PKCE S256 challenge, state, nonce
    alt client_id is not the console's
        L-->>U: 401 invalid_credentials
    end
    L->>R: one token from the source bucket, one from the email bucket
    alt throttled
        R-->>U: 429 rate_limited, Retry-After (no password was hashed)
    end
    L->>A: system context sms-gateway:login
    A->>A: Argon2id verify (against a dummy hash when no usable account)
    alt no account, no password, inactive, deleted, wrong password, broken role
        A-->>U: 401 invalid_credentials (one answer)
    end
    A->>A: stamp lastLoginAt
    L->>H: Identity(User.id), the authorization request
    H-->>L: redirect carrying code=... or error=...
    L-->>U: 200 {redirect}  (an error redirect is a 400 invalid_request)
    U->>T: grant_type=authorization_code, code, code_verifier
    T-->>U: access_token (RS256, 15 min) + id_token, never a refresh_token
mermaid
stateDiagram-v2
    [*] --> Off: no human_login: block, or op:, or the method not permitted
    Off --> Refusing: seed-console-client, provision-user --password, bootstrap --owner-password-out
    Refusing --> Off: auth_method_disabled, before the database
    [*] --> Unchecked: served
    Unchecked --> Refused: serve boot: console client row missing, wrong redirect URIs, or refresh_token
    Refused --> Unchecked: seed-console-client reconciles it
    Unchecked --> Serving: the row matches human_login.console_client
  • Passwords are Argon2id and nothing else. sms_core::password: the argon2 crate's own defaults (19 MiB, 2 passes, 1 lane), a random salt, a PHC string in user_credentials.password_hash (hasRole('system') on every action; the hash is a separate model so no route can return it). No pepper: a hash exported before #446 (the runbook's "Keeping the OpenID Provider's rows across 0009 and 0012") must verify once imported. A new password is generated (24 alphanumeric characters), hashed, and written to a 0600 file created with O_EXCL by provision-user --password --password-out <file> or bootstrap --owner-password-out <file>; it is never printed. An existing password is refused unless --reset.
  • The check costs the same whoever you are. authenticate_user runs a real Argon2 verification on every call, against the stored hash or, when no usable account matches (unknown email, no password set, inactive, soft-deleted), against a dummy hash made by the process's own hasher. A hand-typed dummy that failed to parse would make the missing-account path faster than a real check; the dummy is made at first use from a random throwaway password nobody keeps, a unit test holds it to a real hash's variant and cost, and live tests hold the two paths' clocks and their statement counts together (an existing account used to cost the database one statement more). The hashing runs on the blocking pool.
  • One generic answer. Every refusal of the person or the attempt is 401 {"error": "invalid_credentials"}, including a client_id that is not the console's; a database outage is a 503, not a 401. lastLoginAt is stamped once a code has been issued (not when the password was merely right), which is why User.update admits hasRole('system') again (a policy, not a wire shape: the contract digests of cratestack contract digest are unchanged, vsms keeps no contracts lock, and migrations-current needs no migration).
  • The limiter is in front of the password, and has no source bucket. login_rate_limit is two buckets taken before any hashing: one per email (trimmed and lowercased, capped), then one aggregate for every sign-in, sized as the Argon2 CPU budget (a tenth of the API's per-peer refill: a verification is tens of milliseconds of a core and 19 MiB, a sealed call about a millisecond). The email bucket is first, so a refused address never spends the aggregate. A bucket per source address was removed in review: the gateway sees one peer for everybody (the proxy, or the console's server calling it directly), so it was a global bucket that one bad request every ten seconds kept empty and every operator out. A source is bounded at the edge, by deploy/Caddyfile's auth_per_source zone, where its address is known. Left standing: anyone who knows an operator's email can keep that one operator throttled (the price of a per-account bucket; the paired-key door is unaffected), and a flood from many addresses can empty the aggregate and answer the sign-in with 429. The buckets are in memory, which is the single-replica assumption the authorization codes share (below); a second replica would double the burst.
  • What the request alone proves wrong is refused first, as a 400. A challenge method other than S256, a code_challenge that is not 43 to 128 base64url characters and a redirect_uri the console is not registered with are 400 invalid_request before the limiter and before any hashing, so they cost no token and are never the person's 401.
  • handle_authorize is the library's, unmodified. PKCE with S256 only, exact redirect_uri match, response_type=code, the client's own scope and grant admission. Its answer for a request it refused is a redirect carrying error=; POST /login maps that to a 400 invalid_request so the console is never handed an error as though it were a sign-in. GET /authorize is not mounted and the discovery document has no authorization_endpoint.
  • The grant gate (sms-gateway's grant_gate.rs) sits in front of /token, outside its rate limiter: client_credentials only when machine_tokens is served, authorization_code only when human_login is, and anything else, refresh_token above all, is 400 unsupported_grant_type. Discovery lists exactly the grants served plus code_challenge_methods_supported: ["S256"].
  • No refresh token, three ways. The console client is registered for authorization_code alone and serve refuses to start on a row that carries more; the gate answers a refresh request unsupported_grant_type; and the store's refresh slot (RefusingRefreshStore) cannot hold one. offline_access is not among the console's scopes, so handle_authorize refuses a request for it.
  • serve checks the console client row at boot (sms_auth::console_client): missing, a redirect URI list that differs from the configuration, a refresh grant, a key, an AppClient link or a missing PKCE requirement each stop the process with a message that names every difference and the command, sms-gateway seed-console-client, which reconciles the row (and says what was wrong). A fresh deployment runs rotate-signing-key and seed-console-client before serve (run --rm, never exec).
  • Codes are in memory, so one replica. An authorization code lives 60 seconds in the process that issued it (the chart pins replicas: 1); a code lost to a restart costs a person one more sign-in. Moving it to Postgres is a follow-up, not a precondition.
  • What a password opens, since VSA-4b: everything the role admits, except owner and admin. The access token carries the person's User.id as sub and an identity claim, which the bearer door accepts as that person with the role their row says (above). An owner or an admin is refused with 403 DEVICE_REQUIRED and acts only from a paired console key. bootstrap --owner-password-out therefore yields a password that opens nothing at the API for the owner it creates; it signs the owner in to the provider, and the owner pairs a key by the command line (pair-console-key) to act. The console has no sign-in screen until VSA-5.

Single sign-on (VSA-4c) ​

human_login.sso adds a second way to answer the one question the console sign-in asks, who is this person? A person signs in at the staff identity provider (Keycloak) and the gateway, its relying party, turns that into the same authorization code a password would have earned. It is optional, nested in human_login: (so everything §4.9 says of human_login holds for it), and absent means the password is the only way in:

yaml
human_login:
  # console_client and login_rate_limit as above
  sso:
    issuer: https://auth.example.org/realms/staff   # byte for byte what the provider's discovery document says
    client_id: vsms-gateway                         # this gateway's confidential client at the provider
    client_secret: ${SMS_SSO_CLIENT_SECRET}         # a placeholder under livemode, and the gateway's alone
    scopes: [openid, email, profile]                # must include openid
    required_group: vsms-staff                      # an exact match in the groups claim: a gate, never a role
    groups_claim: groups
    accepted_amr: [otp]                             # the multi-factor rule: any one amr or acr value suffices
    accepted_acr: []                                # and the two lists are never both empty
    jit_role: pending                               # the role a first sign-in creates (holds no permission): never owner, admin, system or app
    jwks_min_refresh_secs: 30                       # floor between two fetches of the provider's signing keys
    rate_limit: { burst: 30, refill_per_second: 1.0 }
mermaid
sequenceDiagram
    participant B as Browser
    participant S as GET /login/sso/start
    participant K as Identity provider
    participant C as GET /login/sso/callback
    participant H as handle_authorize (authkestra)
    participant T as POST /token
    B->>S: the console's authorize request (eight OAuth parameters)
    alt bucket, parameters, console client_id, cheap checks, or a dry-run handle_authorize refuses
        S-->>B: 400, nothing stored, never an error redirect
    end
    S->>S: SsoTransaction (hashed state, nonce, PKCE verifier, the console request)
    S-->>B: 302 to the provider (code, S256, state, nonce)
    B->>K: sign in, with a second factor
    K-->>B: 302 to the callback with code and state
    B->>C: GET /login/sso/callback
    alt state unknown, replayed or expired
        C-->>B: 400
    end
    C->>K: code exchange (client_secret_basic, PKCE verifier)
    C->>C: id_token (alg, kid, signature, iss, aud, azp, exp, iat, nonce), group, MFA
    alt MFA claim missing on the first try
        C-->>B: 302 to the provider once more with prompt=login
    end
    alt refused
        C-->>B: 302 to the console redirect_uri with error=access_denied and the console's state
    end
    C->>C: find the FederatedLink by issuer and sub, or create the User and the link in one transaction
    C->>H: Identity(provider_id sms-console, external_id User.id)
    H-->>C: redirect carrying code=...
    C->>C: stamp lastLoginAt
    C-->>B: 302 to the console redirect_uri with the code
    B->>T: the console's server redeems the code with its PKCE verifier
mermaid
stateDiagram-v2
    [*] --> Off: no sso block, or human_login not served
    Off --> NotFound: GET /login/sso/start and /callback
    [*] --> Booting: human_login.sso served
    Booting --> Refused: the jit_role has no Role row, or the console client row is wrong
    Booting --> Serving: the provider is contacted lazily, a dead provider does not stop the gateway
    Serving --> Retrying: the id_token lacks the MFA claim, first try only
    Retrying --> Serving: one prompt=login redirect, never a second
    Serving --> SignedIn: verified, admitted, a code issued
  • The gateway stays the console's OP. The console and the API never see an upstream token. Only the step the OP cannot do for itself is federated, and the same unmodified handle_authorize issues the same code for the same Identity (provider_id sms-console, external_id the User.id) as POST /login does, so the token it earns is the person's at the bearer door with the role their row says and never as an owner or an admin (403 DEVICE_REQUIRED, above). Nothing in the cap is SSO-specific, which is why it is not.
  • The redirect URI is derived, not written. The callback registered at the identity provider is op.issuer followed by /login/sso/callback; serve logs it at boot. A redirect_uri key is an unknown field. The eleven SMS_SSO_* variables of the flags 29f93c4^ had are RETIRED (§9.4): a gateway whose environment still sets one that the overlay does not read as ${NAME} is refused at boot naming the key that replaced it, and only SMS_SSO_CLIENT_SECRET, which the overlay does read, is still a secret's source. That is the breaking change of VSA-4c, on purpose.
  • Only the gateway is handed the secret. The loader drops human_login.sso for the worker and the one-shot commands after its retired-variable and literal-secret checks and before placeholders resolve, so a worker starts without SMS_SSO_CLIENT_SECRET in its environment. Under livemode a literal secret is refused for every binary, a YAML number or boolean in any mode, and Debug prints <redacted>.
  • start is a chain of cheap refusals before a row exists (sms-gateway's sso.md): the bucket, every parameter present and at most 1024 bytes, the console's client_id (the OP knows other clients, and the library alone would let one through), the cheap checks POST /login makes (S256, a 43 to 128 character challenge, a registered redirect_uri), and a dry run of the real handle_authorize against a throwaway code store. Every refusal is a plain 400. The first broker forwarded the library's error redirect to the browser; VSA-4a's rule for the password path, that the console is never handed an error as though it were a sign-in, holds for both now.
  • What an id_token must be (sms_auth::sso::id_token): signed with an asymmetric algorithm (RS, PS or ES; none, HS* and EdDSA are refused), by a key in the issuer's JWKS found by kid (an unknown kid earns one refetch, never more often than jwks_min_refresh_secs, so spraying kids cannot make the gateway an amplifier), with iss equal to the configured issuer byte for byte (never trimmed or case-folded), aud containing this client, azp equal when present and required when aud has several values, exp, nbf and iat in range, and the nonce this transaction sent. Discovery is fetched lazily and its issuer must match the configured one (OIDC Discovery 4.3). Each failure is a fixed label in the log and one generic access_denied for the browser.
  • Review round 1. The issuer is at most 512 characters (FederatedLink.issuer's column; a longer one passed every other rule and failed the first account as a validation error). The e-mail clash check ignores case: users_email_key compares the exact text and the query builder has no case-insensitive comparison, so a first sign-in scans the users a page at a time and compares in Rust (a returning person never reaches it, and a table over 50,000 users fails closed as the operator's problem). Under livemode every endpoint the discovery document names is https, not only the issuer. A key older than its lifetime is still used when the refetch fails, or while another task is already fetching, and no lock is held across the network call, so a dead identity provider costs one sign-in its timeout and not every sign-in. The upstream code may be 4096 bytes (every other parameter 1024). The callback redirects only to a redirect_uri the console client is registered with now. Database errors are logged as SQLSTATE and constraint, never their text. The console must bind the upstream state to its own cookie and refuse a mismatch (sms-gateway's sso.md).
  • The group and the second factor. required_group is an exact match against groups_claim, a gate and never a role: nothing in a token sets or changes a role. The multi-factor rule is any one of accepted_amr or accepted_acr; both lists empty is a configuration error, never "no rule". A token that fails only the MFA rule on the first try sends the browser back to the provider once with prompt=login; any other failure, or a second token that still lacks the claim, is refused at once with no loop. Keycloak's acr of 1 or 0 proves nothing about a second factor and amr is empty until its flow's executions carry an authenticator config (docs/runbooks/sso.adoc).
  • Linking is by (iss, sub), never by email. A returning person is found through FederatedLink and their email is never read. A first sign-in creates the User (the one jit_role) and its link in one serializable transaction, under a synthetic admin principal (User.create admits owner, or admin creating a role that is not owner) and the link under system. Email and name are display data written once. A local account holding the email is email_in_use and never merged. Creating an account additionally needs email_verified: true (email_unverified otherwise) and an address that is not at .invalid (email_reserved: console pairing writes <id>@vaam.invalid, and a federated account must not collide with or impersonate one). A linked person who is deactivated or deleted is refused.
  • The JIT role is checked three ways, because the synthetic admin creator could create an admin. The loader refuses owner, admin, system and app (sms_config::REFUSED_JIT_ROLES, a copy of sms_api's RESERVED_ROLE_KEYS and DEVICE_ONLY_ROLES, which sms-config cannot import and a test in sms-auth pins); sms_auth::sso::store refuses to create under one at run time, as the operator's error and not the person's; and the database refuses an owner from an admin creator. No system clause was added to User.create: widening a policy to save a synthetic principal is the wrong trade. serve also checks at boot that the Role row exists.
  • lastLoginAt is stamped only once a code is issued, by the caller, and a first sign-in creates the account with none; a verified token whose request the library then refuses signed nobody in.
  • The JIT role decides what a first sign-in can do, and the recommended one holds nothing. jit_role: pending creates a person with no permission: since #469 a message is read only with sms:read, and pending holds none, so membership of required_group is no longer an access decision to messages (§5.2). The opt-out and consent reads and the role table are still open to any person (a follow-up). A deployment that names support gets a person who reads no message either, since 0013_role_read_scope; one that names a custom role gets whatever that role holds. A returning person whose role someone later raised reads what the new role reads.
  • The password stays the break-glass way in. A person created by SSO has no user_credentials row, so POST /login is a 401 for them. A dead identity provider fails a sign-in as temporarily_unavailable and nothing else: discovery is not demanded at boot, and an_unreachable_idp_does_not_stop_the_gateway_or_password_sign_in proves it.
  • Rows and limits. SsoTransaction lives five minutes and is consumed by a delete (of two concurrent callbacks for one state, exactly one wins); the gateway's reaper, spawned only when SSO is served, sweeps the rest. human_login.sso.rate_limit is one global in-process bucket for each route, failing closed. A single source can empty it for everyone, so the edge bounds a source first: Caddy's sso_per_source zone over /login/sso/* (RATE_LIMIT_SSO_PER_SOURCE_EVENTS and _WINDOW, 30 a minute; the compose Caddy service forwards only three variables, so retuning it there means adding these two).
  • What no CI run proves. Every test signs in against sms-fake-oidc, an in-process stand-in that can script a malformed token on purpose. Nothing runs against a real Keycloak, and the console has no sign-in button until VSA-5; both are by-hand checks in docs/runbooks/sso.adoc.

5. RBAC and identity ​

5.1 The two-layer rule ​

Your instinct — one app, so permissions map cleanly onto the verified principal — is right, with one adjustment for the policy engine.

hasRole('literal') and inTenant('literal') are the only two policy functions, each taking a single string literal. No array-contains, no arbitrary functions. So:

Layer 1 — coarse, in the schema, role-based. @@allow("update", hasRole('owner') || hasRole('admin')). Enforced as SQL predicates on find_many/find_unique/update/delete, and unsupported policy shapes fail generation rather than degrading silently. This is your real perimeter.

Layer 2 — fine, in procedures and a Tower layer, permission-based. perms is a string array on a user principal and scope a space-delimited string on an app principal, both set on the context by the signer directory (§5.3), never read from a request. Check both in Rust before the procedure body runs.

Layer 3 — cosmetic, in the admin UI. perms drives which buttons render. Never a security boundary.

The invariant: every permission checked in layer 2 sits behind a role gate in layer 1 that is at least as restrictive. Layer 2 narrows; it never widens.

A trap worth a test: a misspelled action in @@allow is silently dropped rather than rejected, and deny-by-default then makes that operation unreachable. Assert the full generated policy set, not just that the schema compiles.

5.2 Roles and scopes ​

Humans get roles. Service accounts get scopes. Different vocabularies on purpose — an app has no business holding user:manage.

RolePurposePermissions
ownerBreak-glass. 1–2 humans.everything, incl. role:manage, user:delete, provider:delete, client:provision
adminDay-to-day administrationall except role editing and owner-level deletes
operatorRuns trafficsms:read/send, message:cancel, provider:read/update, route:read, sender:manage, optout:manage, job:read/enqueue, worker:read, dashboard:read
developerIntegrates appsapp:read, webhook:manage, sms:send
auditorRead-only oversight*:read, audit:read, optout:read. No mutations anywhere.
supportFirst-lineoptout:manage, delivery:read
pendingWaits to be promotednothing. A first single-sign-on sign-in lands here until an owner or admin moves the person to a real role.
systemInternal onlymessage:create/update, receipt:create, job:update. Never issued to a human, never reachable from any HTTP route.

The table after 0013_role_read_scope (owner's decision, after #469). The built-in roles that keep sms:read are owner, admin, operator and auditor. developer and support lost it, nothing else of theirs changed, and pending was added with no permission at all. 0002_bootstrap is frozen (§2.10 generates it, and a shipped migration is never edited), so 0013_role_read_scope is the forward migration that takes a database from that seed to this table: it removes the one token from the two built-in rows by key (a custom role is never touched), bumps their version, and inserts pending; it refuses, and rolls back whole, if a role keyed pending already exists and holds permissions (an owner made it, and jit_role: pending would hand them to every first sign-in), and keeps an empty one. It is idempotent, data only, writes no @@audit row (raw SQL), and the contract digest does not move. Who loses what, and the shape of the pending role, are under the message-read paragraphs below.

The table after 0014_optout_read_scope (owner's decision, 2026-11-01). A person reads OptOut and ConsentRecord only with optout:read or optout:manage; the built-in auditor holds neither, so the migration gives it optout:read and touches nothing else. owner, admin, operator and support already hold optout:manage (they record and search opt-outs) and lose nothing; they are deliberately not given optout:read as well. developer and pending hold neither and no longer read them. The migration is data only (the policy change is in schemas/vsms.cstack, so there is no DDL and the contract digest does not move), appends the token to the one built-in row keyed auditor (a custom role is never touched, nor is an auditor that already holds it: the token test makes a second run match nothing), bumps its version, and writes the set sentinel-packed whatever shape it had, in the same token-array style as 0013. It is idempotent and writes no @@audit row (raw SQL). 0002_bootstrap stays frozen: the SQL block below is the seed as shipped.

Only an owner may hand out owner (2026-10-20). owner is the top of the human ladder — only an owner may delete a User or edit a Role — but User's create/update policy used to admit admin with no condition on the value written, and in production an admin updated its own record to roleKey = owner (cratestack_audit: actor role admin, before admin, after owner). The policy engine can decide only half of this, so the fix is two guards that each cover what the other cannot see:

mermaid
sequenceDiagram
    participant A as admin (non-owner)
    participant M as enforce_owner_grant_guard (owner_grant.rs)
    participant R as generated PATCH /users/{id}
    participant P as User @@allow update (SQL)
    A->>M: PATCH /users/{id} {"roleKey":"owner"}
    M->>M: authenticate, role is not owner, decode body
    M-->>A: 403 Forbidden
    A->>M: PATCH /users/{id} {"roleKey":"operator"}
    M->>R: original bytes
    R->>P: UPDATE users ... AND policy(stored row)
    P-->>R: stored roleKey = owner gives 0 rows, Forbidden
    P-->>A: otherwise 200
mermaid
stateDiagram-v2
    [*] --> NonOwner: create by owner or admin
    [*] --> Owner: create by owner only
    NonOwner --> NonOwner: owner or admin changes role
    NonOwner --> Owner: owner only
    Owner --> NonOwner: owner only
    Owner --> Owner: owner only
    NonOwner --> NonOwner: admin tries owner, 403 from the middleware
    Owner --> Owner: admin edit or deactivate, 0 rows from the policy
  • User @@allow("create") / @@allow("update") are hasRole('owner') || (hasRole('admin') && roleKey != 'owner'). A create policy is evaluated against the input, so an admin cannot create an owner (also through provisionUser, whose User create runs under the caller's own context). An update policy compiles to UPDATE … WHERE <policy> against the stored row, so an admin cannot demote, deactivate or otherwise edit an existing owner. It cannot see the new value (cratestack-sqlx update_exec.rs appends the policy to the WHERE and checks nothing after the write), so on its own it lets admin → owner through.
  • sms_api::owner_grant::enforce_owner_grant_guard (backends/crates/sms-api/src/owner_grant.rs, mounted in router::router) is the other half: for POST /users and PATCH /users/{id}, when the authenticated role is not owner, it buffers the body, decodes it with the router's own codec into the same generated input type the handler decodes, and answers 403 if roleKey is owner. An undecodable body from a non-owner is refused (400, or 415 for a content type the router cannot decode), never forwarded. An owner's request is not inspected. Adding a user-write route fails every_generated_user_write_route_is_guarded until it is guarded or classified.
  • Proof: backends/crates/sms-api/tests/role_escalation_live_postgres.rs (the data-layer half) and backends/crates/sms-api/tests/owner_grant_live_postgres.rs (an_admin_cannot_*, signed, real HTTP, the incident replayed). Each guard was broken for real and the matching test failed; see AGENTS.md.

Service account scopes: sms:send, sms:read, webhook:manage, optout:read (reads every opted-out number of every tenant; an app's optout:manage scope reads nothing, see §5.2), and (#56/#57) job:read, job:enqueue, worker:read, and (#49) dashboard:read — the same literals operator's own perms carry, reused verbatim rather than invented separately, since require_permission (§5.1) checks either claim for the identical string. Registered per AppClient and enforced verbatim — scopes are rejected rather than filtered, and an omitted scope yields scope: None, which your check must treat as denial.

Implementation, #56/#57. Job's own @@allow("list"/"detail", ...) admits auth().kind == "app" unscoped (no appId on Job to filter by, unlike Message) — the same shape SenderId/OptOut already use. That makes the Layer 2 scope check the real perimeter for the admin console's Jobs screen, not defense in depth: router::JOB_READ_ROUTES gates GET /jobs/GET /jobs/{id} on job:read, and requeueJob's own require_permission(ctx, "job:enqueue") gates the re-enqueue write. workerLocks is gated the same way, on worker:read. Any provisioned app client that never requested these scopes gets a 403, exactly like an app client without sms:send cannot call sendMessage — the admin console's own client is provisioned with all three (scripts/demo.sh, docs/runbooks/getting-started.adoc).

Implementation, #49. dashboardSummary is a type-only procedure (no model, no DDL) whose own @allow admits any auth().kind == "app" caller unconditionally, the identical shape workerLocks/simulateRoute already have — so require_permission(ctx, "dashboard:read") inside Procedures::dashboard_snapshot is the real perimeter, not defense in depth. The console is provisioned with this scope alongside the others above.

Implementation, message reads (reading a message needs sms:read). Message's list and detail policy admitted auth().kind == "user" with no role test, and a policy cannot read Role.permissions (hasRole takes one literal), so a person in any role read every app's msisdn and, while it is retained, body (an OTP's code): a paired console key of any role, a password session (VSA-4b), a single-sign-on user created as support (VSA-4c), and a custom role holding only app:read alike. A role could limit what a person did and never what they read. Two things close it, and the second is the one that binds:

  • A claim. auth Principal gained messageRead and optOutRead, set in GatewayAuth::context_for from the same perms / effective scope that require_permission checks (auth::Grants::of): messageRead is granted when the caller holds sms:read (a person's role, an app key's scope) and optOutRead when it holds optout:read (either door) or, for a person only, optout:manage; each is denied otherwise, and denied on any context built by Principal::into_context (the system's, a test's). The policies of Message, MessagePart, DeliveryReceipt and WebhookAttempt (its payload is message data: messageId, clientRef, state, operator, costXaf and to, the full number when the endpoint's maskRecipient is false) and the listMessagesPage procedure ask for auth().messageRead == "granted"; an app keeps its appId == auth().appId clause on top of it, and the system context is admitted by role. A policy is evaluated wherever the model is read, so ?include=messages on App, ?include=message on a receipt, a relation filter (?messages.some.msisdn=) with its totalCount, and a nested include (/consent_records?include=app,app.messages, run as a caller that loads app: a custom role gets app: null there and would prove nothing) get no message from a caller without sms:read (cratestack 0.16 applies the related model's read policy to each; tests/message_read_live_postgres.rs runs every one of them, with a positive control, and each fails when Message's policy is reverted). An app keeps reading its own endpoints' attempts, with no scope.
  • A route gate in front. router::MESSAGE_READ_ROUTES (a fifth independent enforce_route_permission instance) asks sms:read of GET /messages, /delivery_receipts and /message_parts and their {id} forms, so those answer a 403 naming the permission instead of an empty page. A requirement on a GET gates the HEAD as well (RoutePermission::applies_to): axum answers a HEAD with the GET handler and keeps the full Content-Length, and the envelope layer resolves a signed HEAD to the GET contract, so before this a HEAD /messages walked past a gate that refused the GET, and HEAD /messages?body__startsWith=… was a length oracle on an OTP. The same walk-past existed on JOB_READ_ROUTES and PROVIDER_ROUTE_READ_ROUTES, the real perimeter for an app key, and is closed by the same change.
mermaid
sequenceDiagram
    participant P as caller, person or app
    participant L2 as MESSAGE_READ_ROUTES, Layer 2
    participant L1 as message policies, Layer 1
    P->>L2: GET or HEAD on messages, receipts, parts
    alt sms:read is not held
        L2-->>P: 403 missing required permission sms:read
    else sms:read is held
        L2->>L1: the request goes on unchanged
        L1-->>P: rows the policy admits, an app's own only
    end
    P->>L1: GET apps with include messages, or a relation filter
    alt messageRead is denied
        L1-->>P: the app rows, with no message in them and none matched
    else messageRead is granted
        L1-->>P: the messages the policy admits
    end

The contract digest does not move (a policy and an auth claim are not wire shape; 140 lines, identical before and after), there is no DDL (migrations-current passes untouched) and sms:read is an existing permission. That change took nothing from a built-in role (all six carried the permission); the role table was changed after it, by 0013_role_read_scope (above). Both schema copies carry the change (sdk-schema-check).

Model or procedure (what it admitted)DecisionReason
Message (any person, the app's own, system)claim and route gateThe subject: number and body.
DeliveryReceipt (any person, system)claim and route gaterawPayload is the provider's callback, which listMessageReceipts leaves out on purpose.
MessagePart (any person)claim and route gateDelivery state only, but anything about a message asks for the same permission.
listMessagesPage (any signed caller, a stub today)claimIt returns a page of messages: the day it is implemented it must not be a door.
App, AppClient, WebhookEndpointnothing of their ownThey reach a message only through a relation, which the Message policy now binds.
ConsentRecord (any person, the app's own, system)claim for a personEach row is a raw number. A person needs optout:read or optout:manage (no consent:read); an app keeps its own records whatever its scope, and an optout:read key does not read another tenant's.
OptOut (any person, any app key)claim: optout:read or optout:manage (person), optout:read (app)Each row is a raw number and the model has no owning app, so any app key of any tenant listed every opted-out number, and any person (whatever the role) listed them too. An app's optout:manage scope does not count: recordOptOut admits person roles only, so it would turn a scope that does nothing into a read of every tenant's numbers.
WebhookAttempt (any person, the endpoint's app)claim for a personThe payload is message data, and the number is whole when the endpoint's owner chose maskRecipient false. An app still reads its own endpoints' attempts, and system (the outbox, hooks) by role.

Who loses what: a person in a custom role that lacks sms:read (and, since 0013_role_read_scope, developer and support: see "Who loses what from 0013_role_read_scope" below; such a role also sees no webhook attempt, its history on the Webhooks screen included); an app key or machine token whose scope lacks sms:read, such as the send-only otp-sender in docs/runbooks/deployment.adoc, which could read back its own messages and now cannot (a token that asked for sms:send alone is narrowed the same way); an app key without optout:read reading opt-outs (it now sees an empty list, the way Layer 1 filters everywhere, and a 404 for one row); and, since 0014_optout_read_scope, a person holding neither optout:read nor optout:manage (developer, pending, a custom role with neither), who now gets the same empty list and 404 for opt-outs and consent records, a HEAD counted as the GET (no route gate: a requirement names one permission, so the answer is a 200 with nothing in it, not a 403); a person who can cancelMessage but whose role lacks sms:read (the procedure's @authorize reads the message under the caller's own policy); and a custom role with dashboard:read and no sms:read, whose dashboardSummary message tiles read as zero (the procedure counts messages as the caller, so it counts what the caller may read). A person's token and their console key are the same person and get the same answer; owner and admin still act only from a paired key.

The role table is now what decides who reads messages, and 0013_role_read_scope made it say who the owner meant (his decision, after #469: owner, admin, operator and auditor keep sms:read; developer and support lose it; a zero-permission pending role is added; nothing else changes). Who loses what from it:

  • developer reads no message, receipt, part or webhook attempt through the console, and a ?include=messages on /apps (which developer passes, because App's read policy names the role) returns the apps and no message. A developer reads its own app's messages through an app key whose scope holds sms:read (the appId clause on Message is unchanged). developer also can no longer replay a webhook attempt from the console: replayWebhookAttempt reads the attempt as the caller first (@authorize), the attempt's policy asks for the claim, and a developer has none; owner and admin replay. A developer cannot list attempts either, so the loss is consistent, and webhook:manage is deliberately not made to satisfy the attempt's policy (that would reopen the payloads, which are message data). Webhook endpoints, secret rotation and the delivery mode are unaffected (webhook:manage).
  • support loses the Messages, Receipts and Parts screens and the webhook attempts' history; it keeps optout:manage (so it still records and searches opt-outs) and delivery:read, which lookupDeliveryStatus checks: "Implementation, delivery lookups (delivery:read)" below is how support still answers "did it arrive?".
  • A custom role is untouched: the migration updates only the built-in rows named developer and support, so a role an owner made keeps what it holds, sms:read included.
  • A deployment whose overlay says human_login.sso.jit_role: support keeps creating first sign-ins as support, which now reads no message; it needs no change to stay safe, and should name pending to stop creating people with a permission at all.

pending is the role a first sign-in waits in. It holds no permission (' ', the empty packed set) and nothing in the schema names it (hasRole('pending') appears nowhere; device_only_roles.rs holds that, and puts pending on its allow-list with the reason), so a person in it reads no message and changes nothing. It does not read nothing at all: a person in it passes the policies that admit any user. Since 0014_optout_read_scope that no longer includes a phone number: what is left is the role table (message_read_live_postgres.rs lists it with its reason), the person's own paired keys, and previewMessage, and also SenderId, SenderIdRegistration and RouteValidation, which a person of any role reads and which hold no subscriber number (listed here, not ruled on: the owner may want them gated too). Every other GET collection is a 403 or an empty page for it, run over every route the schema generated. pending is not in REFUSED_JIT_ROLES (owner, admin, system, app) and it is the recommended human_login.sso.jit_role (the example in §4.10 and config/application.yml); the key stays required, with no Rust-side default. An owner or admin promotes the person in the console under Users and roles; the new role takes effect within the 60-second standing cache. serve checks the role row exists at boot, which 0013 guarantees for any database that migrated.

Left for later, by the owner's decision: a body-free replay (a ReplayResult checked through the endpoint, so a developer could replay without reading the payload); a route gate for opt-outs and consent records that answers a 403 naming the permission (the claim answers an empty list and a 404 today, and a RoutePermission names one permission, so it cannot say "either of two"); and auditLog showing DeliveryReceipt.rawPayload unredacted to an auditor.

Implementation, delivery lookups (delivery:read). lookupDeliveryStatus answers "did this message arrive?" for a person who may not read the message: support holds delivery:read and, since 0013_role_read_scope, no sms:read. It is a procedure with types of its own (DeliveryStatusQuery, DeliveryStatus, DeliveryStatusResult) and not a route on Message, because every route on Message answers with the model, and the model carries the body. Six rules, each held by a test that reads a file (tests/delivery_status_guards.rs) or a database (tests/delivery_status_live_postgres.rs):

  • The answer has no body. A DeliveryStatus is the masked recipient (+237*******56, null once the message is purged), the operator, sender, class, encoding, segment count, state, attempts, the lifecycle timestamps (acceptedAt is Message.createdAt) and lastReceipt, the newest receipt as the existing DeliveryReceiptSummary. It never carries body, bodyHash (an OTP is brute-forceable from its hash if the pepper leaks), bodyLength, msisdnHash, the full number (not even the one the caller typed), costXaf, stateReason (free text a provider's error put there), clientRef or idempotencyKey, providerMessageRef*, the routing columns or a receipt's rawPayload. The type is pinned field for field, and the forbidden names are checked on their own.
  • One key per call. A message id; an app's client reference (a person names the app with appId, an app key's own is implied); or the whole recipient number. A number is matched by equality on Message.msisdnHash (the keyed hash, messages_msisdn_hash_idx), so the procedure cannot be asked "which numbers have you messaged": a prefix, a suffix or a partial number is a 422, and the national form of a number is the same number. As with searchOptOutByMsisdn, a rotated pepper finds nothing and the answer cannot say so. A number reaches only unpurged rows inside the last 90 days, and that is the rule that keeps the lookup from undoing a purge: purge_retention keeps msisdnHash (so opt-outs keep matching), the rows are kept forever, and the keyed hash is what de-identifies a purged row, so a number that matched one would link a plain number to purged history in every app. Two guards hold it (the window is cut at 90 days wide, which alone would not: until can be any past date): the window of a lookup by number may not start before now minus 90 days (a 422 with a fixed message, a lookup by number may not start more than 90 days ago), and the number filter carries purgedAt IS NULL, so a row purged early (an erasure) is not found by number either. A reference matches nothing purged (purge_retention nulls clientRef), and an id is exact: it still answers for a purged message, with recipient: null and purged: true. The window is the last seven days for a number or a reference and at most 90 days wide; an id is unbounded unless the caller names a window. limit is 1 to 50, default 10, newest first, and truncated says more matched.
  • No new claim, no @authorize. Message's @@allow asks for messageRead and must not learn a deliveryRead: a claim there would let everyone holding delivery:read list GET /messages, bodies included (delivery_status_guards.rs fails on any policy that says delivery, and on a claim in auth.rs). @authorize(Message, detail, ...) would run the caller's Message policy, which delivery:read does not satisfy. The procedure's @allow admits a person or an app key; the gate is the first statement of its body, require_permission(ctx, "delivery:read"), so a custom role holding it works too.
  • The read is the body's to scope. It runs under system_context("sms-api:delivery-lookup"), so the model's policy decides nothing. An app key's every query therefore carries appId == its own, an app key naming another app is a 422 (refused, not ignored), and an app key with no app is a 403. A foreign id is an empty answer, as if it did not exist. An app key reaches this with delivery:read in its scope (docs/runbooks/deployment.adoc).
  • Owner, admin and operator do not hold delivery:read (the owner's decision: delivery status is a support tool). auditor and support do, and a custom role may be given it. A test pins operator as refused: it holds sms:read and still cannot ask.
  • Every lookup is accounted for, and the call fails if it cannot be. The procedure writes one DeliveryLookup row (@@internal("all"), @@audit, system-only, kept indefinitely, forward migration 0015_delivery_lookups) before it answers: who asked (actorSub, kind, role, app), which kind of key, how it ended (outcome: pending, answered or over_budget) and, if answered, how many messages came back (resultCount, null otherwise). keyValue is the message id, or the keyed hash of the number or of the client reference, never the thing typed: a client reference is hashed because purge_retention nulls Message.clientRef at 90 days as personal data, and this table is kept. Answering "who looked up this number?" therefore needs the pepper (hash the number, then match keyValue), the same limit msisdnHash has. A write that fails fails the call and nothing is answered. A repeated Idempotency-Key is answered by the idempotency layer with the first answer, without running the procedure (measured: one row for two calls), so it adds neither a row nor a count; the bytes are the same, so nothing new is learned.
  • The same rows are the budget, and the order of the two statements is what makes it hold under concurrency. What a caller can guess is counted: a lookup by number and, for a person, a lookup by client reference (integrators' references are guessable, and a person reaches every app). 60 an hour per caller (BUDGETED_LOOKUPS_PER_HOUR, self-imposed), keyed on the caller's sub and kind, so an app key and a person that happen to share a subject string do not share an hour. An app key's own client references (its own, nobody else's) and every message id are not counted. A budgeted lookup inserts its pending row first, then counts the caller's rows in the last hour, its own included, and is admitted only if the count is at most 60. Every write autocommits, so when an admitted row counts, every row admitted before it is visible and still counted (an admitted row is never marked over_budget, so none drops out of the count), and at most 60 are ever admitted, however many arrive at once, with no lock. Counting first and inserting after, as this once did, lets a parallel burst (the rate limiter allows about 120 a replica) count before any insert and overshoot: measured against live Postgres, 80 concurrent lookups were all answered. The cost of the safe order is that a burst larger than the budget can be refused more than it must be (a row another call counted may be marked over_budget a moment later): in the same burst of 80 it can admit as few as none in the worst interleaving (8 to 45 of 80 measured across 18 runs), rather than 60. It fails closed, never open. A refused attempt is a 403 whose message is delivery lookup budget exhausted: 60 lookups by number or reference per hour (the framework has no 429) and is recorded honestly: its row is marked over_budget, with no count, and is not counted again, so a client retrying in a loop is not held out longer than its admitted lookups warrant. An admitted one is marked answered with its count once it has the answer. Those two edges out of pending are the only updates the model allows (@@allow("update", hasRole('system'))), and the delivery_lookups_update_guard trigger in 0015 refuses every other change with SM001: a settled row is final, and no column but outcome and resultCount ever moves. A row left pending is a lookup that was admitted and never answered (the call failed after it), and counts against the budget. A lookup that is not budgeted is written once, as answered, so it never needs the update.
mermaid
sequenceDiagram
    autonumber
    participant C as Caller
    participant R as Router
    participant P as lookupDeliveryStatus
    participant D as Database
    C->>R: sealed POST with one key
    R->>P: Layer 1 admits a person or an app key
    P->>P: require_permission delivery:read
    P->>P: validate the query and scope an app key to its own app
    alt a number, or a person's client reference
        P->>D: create a pending DeliveryLookup row
        P->>D: count this caller's budgeted rows in the last hour, its own included
        D-->>P: used
        opt used is above 60
            P->>D: settle the row over_budget
            P-->>C: 403 budget exhausted
        end
    end
    P->>D: read messages as system by id, client reference or msisdnHash
    P->>D: read the newest receipt of each
    alt the row was claimed above
        P->>D: settle the pending row answered with the count
    else a message id, or an app key's own reference
        P->>D: create one answered DeliveryLookup row
    end
    D-->>P: stored, or an error that fails the call
    P-->>C: statuses without body and truncated
mermaid
stateDiagram-v2
    [*] --> Received
    Received --> Refused: delivery:read missing
    Received --> Validating
    Validating --> Rejected: not exactly one key, a partial number, a foreign app, or a bad window
    Validating --> Rejected: a number whose window starts over 90 days ago
    Validating --> Claiming: a number, or a person's client reference
    Validating --> Reading: a message id, or an app key's own reference
    Claiming --> OverBudget: more than 60 counted, own row included
    Claiming --> Reading: 60 or fewer counted
    OverBudget --> [*]
    Reading --> Settling: messages and receipts read
    Settling --> Answered: the DeliveryLookup row is stored
    Settling --> Failed: the row cannot be stored, nothing is answered
    Refused --> [*]
    Rejected --> [*]
    Answered --> [*]
    Failed --> [*]
mermaid
stateDiagram-v2
    [*] --> pending: a budgeted lookup claims its place
    [*] --> answered: any other lookup is written once
    pending --> answered: the answer is ready, with its count
    pending --> over_budget: more than 60 counted
    answered --> [*]
    over_budget --> [*]

Digest and contract: cratestack contract digest gains exactly one line (POST /$procs/lookupDeliveryStatus) and no other moves; DeliveryLookup is @@internal("all"), so it has no route, no RPC arm and no client stub (140 digest lines become 141, and sms-gateway routes grows by that one procedure). vsms keeps no contracts lock. The console has no screen for this yet: API and both SDKs only; a "Delivery lookup" page next to the opt-outs page is the follow-up, and so is an auditor-facing "who looked up this number" procedure (it would hash the number server-side, as searchOptOutByMsisdn does).

system deserves emphasis. It's the context the send procedure and the worker bind internally — bind_auth(json!({"sub":"system","kind":"system","role":"system","appId":""})) — to write past the hasRole('system') gates. role must be "system"; hasRole never looks at kind. It must be constructible only inside a process: system is not a row in roles, and the signer directory refuses a Role keyed system or app (§2.10).

5.3 The principal ​

For each request the signer directory (backends/crates/sms-api/src/auth.rs) turns the verified credential into a CratestackContext that the policies and Layer 2 read. The credential is a verified key or, since VSA-3b/VSA-4b, a verified token whose standing is read from rows; nothing is read from a token as a grant:

FieldUser principal (a ConsoleKey, or a person's token)App principal (a ClientKey, or a machine's token)
subUser.idAppClient.clientId
kinduserapp
rolethe person's Role.keyapp (matches no hasRole(...) clause, on purpose)
appIdemptyAppClient.appId
permsRole.permissions, unpackedempty
scopenone (a person's OIDC scope is never read)AppClient.scopes, as the packed string

A person's token is the same user principal as their console key, with one difference: DEVICE_ONLY_ROLES (owner, admin) are refused behind a token (403, DEVICE_REQUIRED, §4.10) and act only from a paired key. Every other role acts through either.

kind is what the schema's policies compare and role is what hasRole(...) compares; they are set together or a policy silently denies. system is never produced here: system_context (§5.2) is the only constructor.

Roles and scopes resolve per request, through the 60-second standing cache, so a changed role or a retired client takes effect within one TTL. A human session has no refresh token to outlive anything: a console key lasts 30 days and is revoked by row.

5.4 Server keys ​

Answers are signed with the gateway's current Ed25519 key and verified by clients against the two public keys they pin, current and next (§4.0, sms-gateway server-keys). There is no JWKS endpoint, no kid selection to get wrong and no algorithm negotiation: the algorithm is fixed per signer kind (Ed25519 for apps and for the server, ESP256 for consoles). The same key seals webhook deliveries (§4.4), so a receiver pins the same pair.

6. Provider abstraction ​

6.1 The trait ​

rust
#[async_trait]
pub trait SmsProvider: Send + Sync + 'static {
    fn key(&self) -> &str;
    fn capabilities(&self) -> Capabilities;

    /// Submit one message. Returns the provider's reference for DLR correlation.
    async fn submit(&self, req: &SubmitRequest) -> Result<SubmitAck, ProviderError>;

    /// Decide whether an inbound callback is authentic, before `parse_dlr` sees
    /// it. Defaults to `Ingest`: Orange and MTN document no signature scheme.
    /// An `Err` means the check could not run, not that the callback is forged.
    async fn authenticate_callback(
        &self,
        _raw: &RawCallback,
    ) -> Result<CallbackDisposition, ProviderError> {
        Ok(CallbackDisposition::Ingest)
    }

    /// Parse a provider-specific DLR callback into the canonical shape.
    fn parse_dlr(&self, raw: &RawCallback) -> Result<Vec<DeliveryUpdate>, ProviderError>;

    /// Optional: poll for status where the provider has no push DLR.
    async fn poll_status(&self, _refs: &[String]) -> Result<Vec<DeliveryUpdate>, ProviderError> {
        Err(ProviderError::Unsupported)
    }

    async fn health(&self) -> Health;
}

pub enum CallbackDisposition {
    /// Authentic (or nothing to check): hand it to `parse_dlr`.
    Ingest,
    /// Authentic but carries no delivery update, e.g. a subscription handshake
    /// the adapter already handled. The gateway answers 200 and parses nothing.
    Acknowledged { detail: String },
    /// Failed authentication. The gateway answers 403 and parses nothing.
    Refused { reason: String },
}

pub enum ProviderError {
    /// Do not retry on this provider. Try the next route.
    Permanent { code: String, message: String },
    /// Retry on this provider after `retry_after`.
    Transient { retry_after: Duration, message: String },
    /// Provider is down. Mark degraded, fail over, open the circuit.
    Unavailable { message: String },
    /// Caller's fault. Fail the message; no failover.
    Rejected { code: String, message: String },
    Unsupported,
    /// The request reached the provider, or may have — a post-connect
    /// timeout, an interrupted read, or a `2xx` we can't make sense of.
    /// Do not retry, do not fail over, do not fail the message: any of
    /// those risks a duplicate SMS. `routed -> uncertain` (§7.4); a later
    /// DLR or the grace-period expiry resolves it.
    Indeterminate { message: String },
}

The error taxonomy is the important part. Most gateway failover bugs are really error-classification bugs: a provider returns a 400 that actually means "your sender ID isn't approved" and the router faithfully retries it on three more providers, burning credit each time. Six variants, each with exactly one routing consequence — five of the six map onto §7.4's routed failure edges the way you'd expect; Indeterminate is the odd one out, because none of "retry," "fail over," or "fail the message" is safe when the provider might have already sent the SMS. See backends/crates/sms-provider/src/error.rs for the full reasoning and backends/crates/sms-provider-http/src/transport.rs's classify_transport_error — shared by every HTTP adapter, not duplicated per crate — for how a real transport failure gets sorted into Unavailable vs Indeterminate.

authenticate_callback exists because POST /dlr/{providerKey} is public and unauthenticated (a provider webhook carries no bearer token), and parse_dlr is synchronous and pure while verifying an Amazon SNS signature needs a certificate fetch. The gateway calls it before parsing: Refused is 403, Acknowledged is 200, Err is 503 (so the provider redelivers rather than a genuine receipt being lost), Ingest continues to ingestion. Only aws_sms overrides it.

6.2 Adapters ​

OrangeCmProvider — build this first. Genuinely self-service: register an app on the Orange developer portal, get client ID/secret, start sending within the hour.

  • OAuth2 client_credentials against https://api.orange.com/oauth/v3/token, TTL 3600s. Cache and refresh at 80% of life; do not fetch a token per message.
  • POST https://api.orange.com/smsmessaging/v1/outbound/tel%3A%2B2370000/requests with an outboundSMSMessageRequest body. Cameroon's country sender number is tel:+2370000.
  • Two products: SMS Cameroon 2.0 (all operators, ~16–22 FCFA/SMS) and on-net (Orange only, ~8–11 FCFA/SMS), selected by ?resource_type_parameter_management=SMS_OCB2. Prefer on-net for Orange-prefixed destinations — that single routing rule is most of the cost optimisation available to you.
  • 201 + a resource_id is your DLR correlation key. It is the last path segment of the response's resourceURL, and its format is not a contract — Orange's own getting-started guide says only "a string with the following typical format: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx". Typical, not guaranteed. Never validate it, parse it, or reason about it being a UUID; take the trailing segment and store it verbatim.
  • The DLR callback contract, from Orange's own docs (this was guessed from generic OneAPI prose until it was read properly, and the guess was wrong in two places — see backends/crates/sms-provider-orange-cm/src/dlr.md):
    • The body is {"deliveryInfoNotification":{"callbackData":"<resource_id>","deliveryInfo": {"address":"tel:+<msisdn>","deliveryStatus":"<status>"}}}. deliveryInfo is a single object, not an array — the original array assumption made every real DLR fail to parse.
    • callbackData carries Orange's own resource_id, not a caller-supplied token. The send body Orange documents has no receiptRequest field at all, so there is nothing to supply. That is why Message.providerMessageRef alone correlates a DLR, and why an Indeterminate submit — accepted by Orange, response never read, resource_id never learned — can never be correlated and is resolved by expire_stale instead.
    • deliveryStatus is exactly five values: DeliveredToNetwork, DeliveryUncertain, DeliveryImpossible, MessageWaiting, DeliveredToTerminal. Orange's own caveat matters for how you treat them: DeliveredToTerminal "you can rely on", but DeliveryImpossible may still be delivered later (a handset off-network for 24h, or a flat battery), and is also what a landline or deactivated number returns.
    • Your endpoint must answer 200 OK to acknowledge. Not 202 — see §3.2's own 202-never-200 rule, which is about this API's send endpoint, not about acknowledging an inbound provider webhook whose published contract demands 200.
  • Hard 5 TPS cap. ~18,000/hour ceiling, unbuyable self-service.
  • Sender name whitelisted via a support form; unapproved names return 400. Max 11 alphanumeric chars plus spaces.
  • Admin endpoints for balance and expiry: /sms/frontends/apps/admin/v1/contracts, /statistics, /purchaseorders.

MtnProvider — MTN's own platform (MADAPI), self-service, and the Swagger is real: downloaded 2026-09-17 and vendored at backends/crates/sms-provider-mtn/mtn-sms-v3-swagger.yaml. This paragraph previously said MTN "publishes no pricing, no sender-ID policy and no DLR spec" — the last of those was simply wrong, and cost this repo an invented placeholder contract that sat in the tree for a milestone. There is a full DLR spec. Pricing and sender-ID policy genuinely are still absent, and go-live is still a commercial timeline.

  • OAuth2 client_credentials against https://api.mtn.com/v1/oauth/access_token/accesstoken?grant_type=client_credentials, scope SEND-SMS. Not a static API key.
  • POST https://api.mtn.com/v3/sms/messages/sms/outbound. Success is 200, not 201 — and a 200 is not enough on its own: the body carries a 4-character MADAPI statusCode where '0000' means success, so a 200 with any other code is an application-level failure.
  • Request: receiverAddress is an array of E.164 strings, plus message, serviceCode (the approved short code) and clientCorrelatorId (maxLength: 36 — a truncated correlator makes every DLR for that message unmatchable). senderAddress is optional and takes precedence over serviceCode when set.
  • Correlation is the mirror image of Orange's. MTN's DLR echoes back clientCorrelatorId — the value you supplied — while transactionId is MADAPI's own tracing id. Orange does the opposite, reporting its own resource_id and accepting no caller token at all. This is exactly what providerMessageRefAlt is for, and the reason that column survived the Orange cleanup that stopped populating it.
  • DLRs do not arrive by default. You must send requestDeliveryReceipt: true and register a deliveryReportUrl through POST /messages/sms/subscription. Unlike Orange's manual support-ticket whitelisting, that registration is an API call, so it can be automated.
  • deliveryStatus is exactly eight values: ACCEPTD, DELETED, DELIVERED, ENROUTE, UNKNOWN, EXPIRED, REJECTED, UNDELIVERED — SMPP message_state names (ACCEPTD is the five-character truncation). Note ACCEPTD and ENROUTE are progress, not verdicts, and map to DeliveryOutcome::InFlight; treating them as uncertain costs the message its retry path (§7.4, and that variant's own doc).
  • The DLR carries no delivering-network field, so delivering_network is always None here — §6.4's grey-route detection gets nothing from MTN DLRs.

Caveat on provenance: developers.mtn.com's TLS certificate was expired when the Swagger was fetched (Let's Encrypt, notAfter Sep 16 06:02:41 2026 GMT), so verification was bypassed for that one download at the maintainer's direction, with the chain otherwise intact. Re-download over a valid certificate and diff against the vendored copy before trusting it for anything commercial.

AwsSmsProvider (backends/crates/sms-provider-aws-sms, key aws_sms) — AWS End User Messaging SMS, the go-live route while the Orange and MTN contracts are pending and a standing second route for the same numbers afterwards. Operator procedure: docs/runbooks/aws-sms.adoc.

  • Sends with SendTextMessage (the pinpoint-sms-voice-v2 API), signed with a hand-written SigV4 so the AWS SDK's HTTP stack — and aws-lc-rs with it — stays out of the tree. Long-lived IAM user keys only.
  • Why not classic SNS Publish: it reports delivery only to CloudWatch Logs, so every message would sit in submitted until expire_stale marked it expired. End User Messaging is the same SMS backend with an API that can publish per-message events to an SNS topic.
  • Receipts come back through a configuration set whose event destination is an SNS topic, which SNS delivers, signed, to POST /dlr/aws_sms. That route is public, so the adapter authenticates each call itself through SmsProvider::authenticate_callback (a hook with a default of "ingest", so Orange and MTN are unchanged), in this order: the TopicArn must equal the configured topic; SignatureVersion must be 2; the signing-certificate URL must be exactly https://sns.<topic-region>.amazonaws.com/SimpleNotificationService-<32 hex>.pem with no query or fragment; then the RSA-SHA256 signature must verify. Certificate fetches that are not already cached are rate-limited, so a forged request can cause at most an occasional fetch. A refusal is 403; a check that could not run (certificate fetch failed or was rate-limited) is 503, so SNS redelivers. A topic left on SignatureVersion 1 therefore has every receipt refused and, because SNS treats a 403 as a permanent failure, dropped for good; the gateway logs that case at error.
  • Subscriptions are confirmed by the operator, not by the gateway. On an authentic SubscriptionConfirmation the gateway logs, at warn, the topic ARN, the token and the command to run (aws sns confirm-subscription --topic-arn <arn> --token <token> --authenticate-on-unsubscribe true). Visiting the SubscribeURL instead would confirm with AuthenticateOnUnsubscribe left false, and every notification carries an UnsubscribeURL (stored in DeliveryReceipt.rawPayload), so anyone able to read a receipt could then unsubscribe the endpoint and silence all receipts. An UnsubscribeConfirmation means the subscription has been removed and is logged at error.
  • SNS gives up quickly on HTTP/S endpoints. The default delivery policy retries three times about 20 seconds apart, and any status other than 5xx or 429 is a permanent failure. A gateway outage longer than about a minute therefore loses receipts unless the subscription has a longer delivery policy (SNS caps the total retry time of an HTTP/S policy at one hour) and a dead-letter queue. The runbook recommends both.
  • The message TTL is set explicitly. providers[code=aws_sms].settings.time_to_live_seconds (5 to 259200) maps to SendTextMessage's TimeToLive. Unset, AWS keeps trying for its default of 72 hours, so an OTP can reach a handset days after vsms expired it (default_validity for otp is 15 minutes). The runbook recommends 900.
  • Two halves, so the gateway holds no AWS credentials: sms-worker builds the sender (region, key, configuration set, TPS ceiling, price); sms-gateway builds a receiver from the topic ARN alone.
  • A final TEXT_SUCCESSFUL (carrier accepted, isFinal) is recorded as Delivered, because many carriers never return a handset receipt to AWS; DeliveryReceipt.rawStatus keeps the original value. TEXT_UNKNOWN (no final event within 72 hours) is Uncertain. Any event that is not final (isFinal false) and is not TEXT_DELIVERED is InFlight, whatever its type: acting on a non-final failure would move the message toward a retry while the carrier may still deliver the original, which sends a duplicate SMS. The full table is backends/crates/sms-provider-aws-sms/src/events.md.
  • Error classes worth knowing. A ConflictException with reason DESTINATION_PHONE_NUMBER_NOT_VERIFIED means the account is still in the AWS SMS sandbox and the number is not a verified destination; it is Permanent, so the message fails over to the next route instead of failing. A ConflictException for a number that opted out at AWS stays Rejected.
  • Seeding for an AWS-only go-live. sms-gateway bootstrap seeds an orange_cm Provider plus a catch-all Route unless it is run with --skip-seed-dispatch; left alone in an AWS-only deployment that route would take about half of all traffic to a provider the worker holds no credentials for. Run bootstrap --skip-seed-dispatch, then seed-dispatch --key aws_sms --kind aggregator_http .... The gateway refuses to start if providers[code=aws_sms].settings.events_topic_arn is set and no Provider row keyed aws_sms exists.
  • No ProviderKind variant exists for it: the Provider row uses aggregator_http, since adding a variant would be a migration for a column nothing reads (adapters are chosen by Provider.key).
  • Data residency: sending recipient numbers and bodies to AWS is a cross-border transfer of personal data (§10 and §13, item 1). The runbook treats clearing it as a prerequisite, not a footnote.

Routing between aws_sms and the operator routes needs no new code. A priority band per operator holds that operator's direct route and an aws_sms route with weights, and a lower-priority wildcard aws_sms route catches everything else and serves as the failover target (§6.3; worked table in the runbook).

AggregatorHttpProvider — config-driven generic adapter so adding an aggregator is a config row, not a crate. Covers Nexah, Africa's Talking, Infobip, Twilio.

SmppProvider — SMPP 3.4, deferred but scaffolded, and the reason the worker has a dedicated singleton smpp role. bind_transceiver, enquire_link every 30s, sliding window (start at 10 outstanding), submit_sm with registered_delivery = 1, DLR as deliver_sm with esm_class = 0x04. data_coding 0 for GSM-7, 8 for UCS-2. Concatenation via UDH with a rotating reference. The classic trap: some SMSCs return message_id in hex on submit_sm_resp and in decimal in the deliver_sm receipt. That's what providerMessageRefAlt is for.

6.3 Routing ​

Ordered rule evaluation: filter Route rows where every non-null match* field matches, sort by priority then weighted-random within a priority band, then filter by provider health, sender-ID approval on that provider, capability fit, and remaining TPS/daily budget. First survivor wins; failoverRouteId is the explicit next hop.

Circuit breaker per provider: five consecutive Unavailable opens it for 60s, then half-open with a single probe. Failover capped at two hops — beyond that you're not routing, you're spraying.

Implementation, #62 (backends/crates/sms-routing, backends/crates/sms-worker/src/routing.rs), landed against the above — the matching/ranking half only, not the circuit breaker or the two-hop failover cap, both #63:

sms_routing::select_route is a pure function: (&[RouteRow], &HashMap<String, ProviderRow>, &RoutingCandidate, &ExcludedRouteIds, draw: f64) -> Decision. No I/O, no RNG, no clock — backends/crates/sms-worker/src/routing.rs::decide is the one place that fetches Route/Provider rows and draws draw (rand::random()), immediately before calling into the pure crate. This is the resolution to "weighted implies random" vs. "deterministic and explainable" (#62's own framing, driven by #54's admin simulator needing to replay a decision without sending anything): production draws once per decision; a replay supplies the same draw and gets the identical Decision back, because the function is a pure fold over its inputs.

NULL on any match* column means "matches anything," not "matches only a NULL value" — a route with no matchOperator routes every operator. Priority is the only cross-band ordering; there is no independent "more specific route wins" tiebreak — an operator wanting that has to set a higher priority explicitly. matchPrefix compares against sms_msisdn::Msisdn::national(), the same national-digit convention OperatorPrefixRule.prefix already uses, not a hand-rolled E.164 comparison.

Decision carries one RouteEvaluation per input route (matched, excluded, disabled, predicate-failed with which predicate and why, or provider-unavailable with why) plus, when the winning priority band had more than one eligible member, a TieBreak showing the exact cumulative-weight ranges draw was compared against and which one it landed in — the full "which routes were considered, which predicates each failed on, how ties were broken, and what the final weighted choice was" #62's own acceptance criteria asked for.

Provider health, sender-ID approval, capability fit, and remaining TPS/daily budget are not yet part of this filtering — one exception. "Provider health" narrows to exactly Provider.state == active (backends/crates/sms-worker/src/routing.rs::convert_provider) — the same check the M2 placeholder this replaced used, since Provider.healthy has no writer yet (§7.5's probe_providers job is still out of scope). Sender-ID approval, capability fit (UCS-2 support, alphanumeric sender), and TPS/daily budget are deliberately not modelled at all: no second provider with materially different capabilities existed to make any of them concrete when this landed (#61 added sms-provider-mtn's crate in parallel, not its wiring into a provider registry). WorkerContext gained a provider registry (ProviderRegistry, keyed by Provider.key/SmsProvider::key()) so a routed message resolves to whichever adapter its winning route's provider actually names, rather than one hardcoded field — the natural place this filtering will eventually plug in.

No Route rows configured at all refuses to dispatch, loudly — every accepted message goes to rejected with a stateReason naming why (routing::explain_no_route), rather than a silent fallback to "any active provider." A deliberate cutover from the M2 placeholder's implicit behaviour, not an oversight: see AGENTS.md's own #62 section for the reasoning and the demo-seeding consequence.

Route.failoverRouteId and select_route's own exclude: &ExcludedRouteIds parameter exist for #63 to build the two-hop-capped failover chain and the circuit breaker described above — calling select_route again with a failed route's id added to exclude finds the next-best route with no changes needed to this crate.

Implementation, #63 (backends/crates/sms-worker/src/dispatch.rs): failover and the provider circuit breaker, landed against sms_provider::ProviderError::routing()'s own compiler-checked mapping, not a second hand-derived one.

Permanent (TryNextRoute) and Unavailable (OpenCircuitAndTryNextRoute) are the two variants that trigger a failover attempt; Transient (RetryThisProvider), Rejected/Unsupported (FailMessage), and Indeterminate (HoldIndeterminate) never do — see backends/crates/sms-provider/src/error.rs's own routing() table and dispatch.rs::handle_submit_error's doc for the full mapping, and this section's own header paragraph for why Indeterminate failing over would risk a duplicate SMS. Route.failoverRouteId itself is not read by the failover mechanism — per this section's own #62-era paragraph above, the actual mechanism is select_route called again with the failed route's id added to exclude; the field is carried through Winner purely for an operator's own explanation trail.

"Failover must not double-send: the claim loop's lease is what prevents it" (the issue's own words) — a failover reroute never calls SmsProvider::submit a second time inline. dispatch.rs::attempt_failover writes routed -> queued with a new providerId/routeId stamped on the same if_match(version)-CAS'd row and returns; the actual resubmit happens on a later claim, under the ordinary queued -> routed -> submit path every other reclaim in this crate already relies on. This is only safe because TryNextRoute/OpenCircuitAndTryNextRoute both mean nothing was ever accepted by the provider — Indeterminate, the one variant that can't make that claim, never reaches this function.

Message.excludedRouteIds (a new nullable column, sentinel-packed via sms_core::pack/unpack) is each message's own accumulated exclude set, capped at two entries (§6.3's own "two hops"). Necessary, not just convenient: a Permanent failure never opens a provider's circuit breaker (backends/crates/sms-provider/src/error.rs's own permanent_never_opens_the_circuit_breaker test) — it is specific to this message (an unapproved sender ID, say), not a provider-wide outage — so nothing else marks that route ineligible for a second attempt. Without remembering it per-message, a second failover hop could pick the exact same already-failing route right back.

The provider-side circuit breaker (Provider.consecutiveFailures/circuitOpenUntil, two new columns) deliberately mirrors WebhookEndpoint's own shape (#40/#41/#59) — same two fields, same reset-on-success-and-on-trip discipline, same if_match-CAS'd best-effort writes — rather than inventing different semantics for a second breaker in the same codebase. The constants differ because the spec differs: five consecutive failures, 60s, not webhook delivery's 20/15min. backends/crates/sms-worker/src/routing.rs::convert_provider is the one reader: an open circuit is treated exactly like state != active (unavailable, with a reason), so every future routing decision — not just the message whose failure tripped the breaker — naturally skips that provider. This is the literal mechanism behind the ticket's own second acceptance clause, "must not fail a message a healthy alternative could carry": a fresh accepted message never even attempts a provider whose circuit is open, proven live by backends/crates/sms-worker/tests/dispatch_live_postgres.rs's own an_open_circuit_routes_new_messages_to_the_alternative_instead_of_rejecting, which asserts the broken provider's own call count stays flat across a sixth, brand-new message reaching submitted entirely through the healthy alternative.

Deliberately not the literal "half-open with a single probe" this section's own header paragraph describes — Provider's breaker, like WebhookEndpoint's, fully reopens the moment circuitOpenUntil passes, with no rate-limited single-probe admission. A true single-probe half-open state needs coordination this codebase has no mechanism for yet (something has to guarantee exactly one in-flight probe at a time across however many dispatch processes exist), and a second, differently-shaped breaker for that one property would cost more in consistency than it buys in fidelity to this paragraph's own prose. Revisit if a real incident shows the simpler reopen is too aggressive.

Found live, not by review: Provider's own update @@allow didn't admit hasRole('system'). dispatch.rs's new circuit-breaker writes run under this crate's internal sys() context, and without that clause every one of them returned Forbidden("update policy denied this operation") — silently absorbed by the same best-effort "log and drop" handling hooks::record_endpoint_failure already established, so the breaker never actually opened despite every surrounding assertion (submit call counts, message states) passing. Caught by an_open_circuit_routes_new_messages_to_the_alternative_instead_of_rejecting itself asserting the effect (circuitOpenUntil genuinely set), not just that the write didn't panic. Fixed the same way every prior instance of this gap shape was: schema.cstack only, confirmed byte-identical DDL via cratestack migrate diff before and after. Safe to grant: GatewayAuth::authenticate (backends/crates/sms-api/src/auth.rs) constructs role: "system" nowhere — it is minted exactly once, inside Procedures::sys(), never from a real bearer token — so this change adds no new HTTP-reachable capability to PATCH /providers/{id} (router::PROVIDER_WRITE_ROUTES). Implementation, #54 (the admin route simulator #62 was already built to support) — a new simulateRoute procedure plus the Providers/Routes/Simulator screens (§11's own "Admin console screens" list: "Providers & routes with a route simulator"):

backends/crates/sms-api/src/route_simulator.rs is the whole mechanism: fetch Route/Provider rows under a sys() context (mirroring backends/crates/sms-worker/src/routing.rs::decide's own query — necessarily a second copy, not a shared call, since sms-api cannot depend on sms-worker, the dependency runs the other way), classify the candidate's operator the same way sendMessage does (Procedures::classify_operator), call sms_routing::select_route with either a caller-supplied draw or a fresh rand::random(), and render the returned Decision onto SimulateRouteResult — every field a straight copy or a lossless flattening of the engine's own enum, never a second implementation of matching. SimulateRouteInput.draw is the load-bearing detail: omit it for a realistic sample of what would happen right now, or supply the exact value the engine reported back (TieBreakInfo.draw) to replay a specific tie-break deterministically — the property §6.3's own select_route doc names as the reason the draw is injected rather than generated internally.

noRoutesConfigured is a distinct field from "zero eligible for this candidate," not inferred by the caller from an empty evaluations list — the same "dispatch refuses, loudly" state §6.3 already documents, surfaced here as its own banner rather than a table that merely looks empty.

Two schema policy changes, not one. Provider.read and Route.read both gained auth().kind == "app" alongside their existing human roles — the identical shape #56/#57 gave Job's own @@allow (neither model carries an appId to scope a row-level predicate by), needed so the admin console's machine credential can list either model at all; router::PROVIDER_ROUTE_READ_ROUTES gates the four generated REST routes (GET /providers, GET /providers/{id}, GET /routes, GET /routes/{id}) on provider:read/route:read at Layer 2, the real perimeter today. simulateRoute itself never needed this — it already reads under sys(), admitted by the pre-existing hasRole('system') clause — but gates itself on route:read too via require_permission, for the identical reason workerLocks/requeueJob do: any auth().kind == "app" caller passes the procedure's own Layer 1 @allow unconditionally, so the scope check is the thing actually standing between a provisioned app client and every Route/Provider row in the system.

Write access is a separate question this PR does not close — and #194 (human login) landed mid-PR, which narrows why, not whether. Provider.update and Route.create/update/delete are untouched — still hasRole('owner') || hasRole('admin') (plus operator for Provider.update), no auth().kind == "app" clause at all. §4's own #194 section is the source of truth that GatewayAuth now genuinely resolves a real hasRole(...)-meaningful context for a human authorization_code token — that no longer hardcodes role: "app" for every real token, only for the client_credentials realm. What still stands between a logged-in owner/admin/operator and these two write paths is one layer up: every admin → sms-gateway call this console's own frontends/packages/gateway package makes — including the Providers/Routes screens' own — still authenticates as the shared SMS_CONSOLE_CLIENT_ID machine credential (§4's own "Not wired up in this PR" paragraph), never the logged-in human's own session token. Confirmed directly against #194's own diff, not inferred: frontends/packages/gateway/ doesn't appear in its changed-file list at all. The admin console's edit/create/delete forms are real, tested code (frontends/packages/gateway/src/providers.ts/routes.ts, wired through #59's own fetchWithEtag/updateWithIfMatch) that 403s against a real gateway — proven live (just demo), body missing required permission "provider:update" — not hidden behind a flag, stated on screen instead, matching this doc's own "no dormant code" convention. Forwarding a per-session human token through frontends/packages/gateway (replacing or supplementing its one static credential, for every screen, not just these two) is real, separate follow-up work this PR's own scope doesn't include.

6.4 Grey routes ​

Orange's published wholesale interconnect floor is 3.5–8 XAF/SMS depending on volume; its retail all-operator API is 16–22 XAF. Some local aggregators advertise 7–12 XAF all-network at low volume — at or below the wholesale floor. Sustained pricing below roughly 8 XAF all-network is hard to reconcile with a legitimate direct interconnect, and the usual explanation is SIM-farm or international grey routing.

Both MNOs run SMS firewalls. Grey routes get blocked without warning, typically at the worst moment. For OTP, a blocked message is a failed login, not a missed advert.

Recommended posture: Orange Developer API as primary for Orange-prefixed traffic, a reputable aggregator with verifiable MTN sender-ID pre-registration for MTN, and a second aggregator as failover only. Validate every route with real handsets on each network before trusting it, and re-validate monthly. Symptoms of a grey route: sender ID silently replaced with a numeric string, intermittent total non-delivery on one network only, DLRs reporting success while handsets receive nothing.

Implementation, #64. The monthly handset-validation half is RouteValidation (§2.10) plus docs/runbooks/grey-route-validation.adoc — a human runs the check and records it with sms-gateway record-route-validation; nothing server-side can perform or infer the check itself, since a grey route's whole effect is invisible past the wire (DeliveryReceipt still reports delivered). The divergence-alert half is crate::jobs::grey_route_watch (backends/crates/sms-worker/src/jobs/grey_route_watch.rs): "routes that should behave identically" is read directly off this section's own two-line reasoning above — the same operator, the same message class — grouped by Message.operator/Message.class, gated on sample size (≥30 terminal messages per side), a two-proportion z-test, and a practical-significance floor (≥15 points), so a route with a handful of messages and a bad ratio is never flagged on that basis alone. See OPEN_QUESTIONS.md §2.4 for what this closes and — more importantly — what it still does not: neither half produces a trusted observation of what a handset actually displayed; the divergence alert is a statistical proxy, and the validation record is only as current as its own performedAt.

7. The worker node ​

One binary, sms-worker, running one or more roles. Roles are the unit of scaling and the unit of failure isolation.

bash
sms-worker --roles dispatch,drain,hooks,jobs,scheduler
sms-worker --roles hooks,jobs          # a second node, scaled out

7.1 Roles ​

mermaid
flowchart LR
    subgraph W["sms-worker"]
        direction TB
        subgraph S["singleton — advisory lock"]
            direction TB
            D["dispatch"]
            DR["drain"]
            SC["scheduler"]
            SM["smpp"]
        end
        subgraph N["scale to N"]
            direction TB
            HK["hooks"]
            JB["jobs"]
        end
    end

    LOCK[("pg_advisory_lock")]
    MSG[("messages")]
    OUT[("cratestack_event_outbox")]
    ATT[("webhook_attempts")]
    JOBT[("jobs table")]

    S -. "pg_try_advisory_lock" .-> LOCK
    D -- "claim + lease → submit" --> MSG
    SM -- "binds · deliver_sm" --> MSG
    DR -- "drain → subscribers" --> OUT
    SC -- "enqueue due" --> JOBT
    JB -- "claim + lease → run" --> JOBT
    HK -- "claim + lease → signed POST" --> ATT
RoleCardinalityLoopWhy
dispatchsingletonclaim messages → route → submitOrange's 5 TPS cap is per contract. Two instances each politely limiting to 5 TPS send 10 and get blocked.
drainsingletondb.events().drain() every 5sThe framework runs no background drain worker. Multiple drainers multiply duplicate delivery (§8.2).
schedulersingletonenqueue due recurring Job rowsTwo schedulers double-enqueue; jobs_dedupe_idx catches it, but cleanly avoiding it is better.
smppsingleton per providerhold binds, pump submit_sm/deliver_smSMPP binds are stateful, sequence-numbered, and contractually count-limited.
hooksscale to Nclaim webhook_attempts → signed POSTSlow customer endpoints are the bottleneck; parallelism is the fix.
jobsscale to Nclaim jobs → execute by kindGeneric background work; nothing shared between rows.

The split is not aesthetic. Every singleton is a singleton because of an external constraint — a provider's rate contract, a stateful protocol session, or the framework's own delivery semantics — not because concurrency would be hard to write.

7.2 Leader election by advisory lock ​

Singleton roles don't need a coordinator, a lease table, or a consensus library. Postgres already has exactly the primitive: a session-scoped advisory lock, released automatically when the connection drops.

rust
const NS: i32 = 0x534d_5300;   // "SMS\0"

pub struct RoleLease {
    conn: PoolConnection<Postgres>,
    key: i32,
}

impl RoleLease {
    pub async fn try_acquire(pool: &PgPool, key: i32) -> sqlx::Result<Option<Self>> {
        let mut conn = pool.acquire().await?;
        let acquired: bool = sqlx::query_scalar("SELECT pg_try_advisory_lock($1, $2)")
            .bind(NS).bind(key)
            .fetch_one(&mut *conn).await?;
        Ok(acquired.then(|| Self { conn, key }))
    }

    /// MUST be called. See the note below.
    pub async fn release(mut self) -> sqlx::Result<()> {
        sqlx::query("SELECT pg_advisory_unlock($1, $2)")
            .bind(NS).bind(self.key)
            .execute(&mut *self.conn).await?;
        Ok(())
    }
}

If the process is killed, its connection closes and Postgres releases the lock; a standby node acquires it on its next attempt (retry every 5 seconds). Failover is bounded by that retry interval plus TCP timeout detection — set tcp_keepalives_idle low on the worker's connection so a hard kill is noticed in seconds rather than minutes.

The one real trap: advisory locks are held by the connection, and sqlx returns a PoolConnection to the pool on drop rather than closing it. A dropped RoleLease therefore leaves the lock held on a pooled connection that is now serving unrelated queries — and no other node can ever take that role until the process restarts. Either call release() explicitly on shutdown, or take the lock connection from a dedicated single-connection pool that you close outright. The Drop impl cannot save you here, because releasing requires an await.

A worker holding no lock for a singleton role isn't idle — it runs its scalable roles normally and retries the lock in the background, so a standby is a fully useful node rather than a cold spare.

7.3 Claim loops ​

Every claim in the system is the same shape: select candidates, take a lease by compare-and-swap, do the work, transition. Per R1 it goes through delegates:

rust
use chrono::{Duration, Utc};
use cratestack::{CratestackError, FilterExpr};
use cratestack_schema::{message, MessageState, inputs::UpdateMessageInput, models::Message};

async fn claim_batch(
    db: &Cratestack, sys: &CratestackContext, worker: &str, budget: i64,
) -> Result<Vec<Message>, CratestackError> {
    let now = Utc::now();

    let candidates = db.message().find_many()
        .where_expr(
            FilterExpr::from(message::state().in_([MessageState::accepted, MessageState::queued]))
                .and(message::expiresAt().gt(now))
                .and(FilterExpr::from(message::scheduledAt().is_null())
                        .or(message::scheduledAt().lte(now)))
                .and(FilterExpr::from(message::leaseUntil().is_null())
                        .or(message::leaseUntil().lt(now))),
        )
        .order_by(message::priority().desc())
        .order_by(message::createdAt().asc())
        .limit(budget)
        .run(sys)
        .await?;

    let mut claimed = Vec::with_capacity(candidates.len());
    for m in candidates {
        let taken = db.message().update(m.id.clone())
            .set(UpdateMessageInput {
                state:      Some(MessageState::routed),
                attempts:   Some(m.attempts + 1),
                leaseOwner: Some(Some(worker.to_owned())),
                leaseUntil: Some(Some(now + Duration::minutes(2))),
                ..Default::default()
            })
            .if_match(m.version)          // the compare-and-swap
            .run(sys)
            .await;

        match taken {
            Ok(row) => claimed.push(row),
            // Someone else claimed it between our read and our write.
            Err(CratestackError::PreconditionFailed(_)) => continue,
            // Ambiguous: policy denied, or the row is gone. Should not happen
            // under the system context — log it, it means a policy bug.
            Err(CratestackError::Forbidden(_)) => {
                tracing::warn!(message_id = %m.id, "claim forbidden");
                continue;
            }
            Err(e) => return Err(e),
        }
    }
    Ok(claimed)
}

budget derives from the provider's remaining TPS allowance, not a fixed constant. The leaseUntil predicate does double duty: it excludes rows another worker is mid-flight on, and it reclaims rows abandoned by a crashed worker once the lease expires. No separate reaper for the happy path. The job and webhook claims are the same function with different types.

Two details in that code carry the whole design.

if_match(m.version) is the claim. The framework renders it as WHERE id = $1 AND version = $2, and a zero-row result becomes CratestackError::PreconditionFailed. That is exactly the semantics you want from a competing-consumer queue — the loser of a race learns it lost, cheaply, and moves to the next row. It's also why @version is mandatory on every claimable model.

Forbidden is ambiguous and must not be swallowed as "lost the race". The framework returns it both when the update policy denies and when the row is invisible or gone, because both produce zero rows. Under a system principal neither should ever happen, so treating it as a warning rather than a continue-in-silence is what surfaces a policy regression instead of hiding it as mysterious throughput loss.

On SKIP LOCKED. The framework cannot express it — verified by grep across all nine crates and by compile error on skip_locked(). .for_update() exists and appends a real FOR UPDATE, but that blocks rather than skips, which for a claim loop means workers queue behind each other instead of moving on.

Optimistic CAS is the right answer here anyway, for reasons that have nothing to do with the framework's limits. dispatch is a singleton, so its claim loop has no competitor at all. hooks and jobs do scale out, but they contend only in the window between reading a candidate list and writing the lease — microseconds — and the loser pays one wasted UPDATE, not a blocked transaction. And crucially, no lock is held across the provider HTTP call, which with FOR UPDATE would mean holding a row lock for the duration of an SMS submission.

If profiling ever shows real contention — the signal is a rising PreconditionFailed rate on the hooks role — the fixes in order are: give each worker a random offset into the candidate window; raise the batch size so workers overlap less; and only then consider SKIP LOCKED, which means adding a fourth row to R1's exceptions table with the benchmark that justified it.

Both loops depend on the partial indexes in §2.10. Without messages_dispatch_idx the candidate query degrades to a sequential scan as the table grows, and it is the hottest query in the system.

7.4 The message state machine ​

Declared as data in message_state_transitions and enforced by a BEFORE UPDATE trigger (§2.10). The Rust code proposes a transition; Postgres decides whether it's legal.

mermaid
stateDiagram-v2
    [*] --> accepted

    accepted --> queued: passes routing
    accepted --> rejected: no route / suppressed
    accepted --> cancelled: operator
    accepted --> expired: validity elapsed

    queued --> routed: claimed by dispatch
    queued --> cancelled: operator
    queued --> expired
    queued --> failed: max attempts

    routed --> submitted: provider accepted
    routed --> queued: transient error, backoff
    routed --> failed: permanent error
    routed --> expired
    routed --> cancelled: operator
    routed --> uncertain: indeterminate outcome, do not retry

    submitted --> delivered: DLR success
    submitted --> uncertain: DeliveryUncertain
    submitted --> undelivered: DLR failure, retryable
    submitted --> failed: DLR permanent
    submitted --> expired: no DLR in window

    uncertain --> delivered: late DLR
    uncertain --> failed: late DLR
    uncertain --> expired: 6h timer

    undelivered --> queued: retry
    undelivered --> failed: max attempts
    undelivered --> expired

    delivered --> [*]
    failed --> [*]
    expired --> [*]
    rejected --> [*]
    cancelled --> [*]

Terminal states have no outgoing rows in the transition table, so terminality is data, not a code branch that someone can forget. An UPDATE moving a delivered message anywhere raises SM001 and the transaction aborts.

Why enforce this in Postgres rather than only in Rust. Three reasons, in ascending order of how much they'll matter to you:

Every writer is covered, not just the ones that went through your state-machine module. A migration script, a psql session at 2am, an admin CRUD route you forgot to lock down, a future service written by someone who hasn't read this document — all of them hit the same trigger. Business rules that live only in application code are enforced only for the callers that happen to use that code.

It closes the races that optimistic locking alone leaves open. @version tells you the row changed under you; it doesn't tell you the new state is one you're allowed to move on from. A cancel and a submit racing on the same message can both pass their version check in different transactions; only one can pass the trigger.

And it makes the state machine legible and diffable. SELECT * FROM message_state_transitions is the authoritative answer to "can a message go from X to Y", and adding an edge is a migration with a review, not a code change buried in a match arm.

Backoff on retryable failures: 5s, 30s, 2m, 10m, 30m, capped by maxAttempts and hard-stopped by expiresAt. Default validity 15 minutes for otp (a code that arrives after the user gave up is worse than no code), 24 hours for notification. This is the schedule undelivered -> queued: retry uses (sms_api::dlr::undelivered_retry_backoff, #122): dlr::ingest_one stamps it onto Message.leaseUntil the moment a retryable DLR failure lands a message in undelivered, and backends/crates/sms-worker/src/claim.rs's Claimable for Message::candidates() — the same shared lease filter every other claim uses — is what actually holds the row back until it elapses. A row whose expiresAt runs out first (the "hard-stopped by expiresAt" half) is excluded from further retries by that same candidates() filter and reaped to expired by expire_stale instead of retried past its own validity window.

uncertain gets its own 6-hour timer, then expired. Never retried automatically — a retry on uncertain is exactly the double-send you're avoiding. undelivered is different in kind, not just in name: it's the state a DLR reports when Orange itself says the failure is retryable, not an ambiguous outcome this system merely suspects might be safe to retry.

Billing note that shapes reconciliation: Orange bills on submission, not delivery, and does not refund DeliveryImpossible. Your cost ledger increments at submitted, not delivered, so the delivery-rate dashboard and the spend dashboard measure genuinely different things.

7.5 The job state machine ​

Same mechanism, simpler graph:

mermaid
stateDiagram-v2
    [*] --> pending: enqueued
    pending --> running: claimed, lease taken
    pending --> cancelled: operator
    running --> succeeded
    running --> failed: handler error
    running --> pending: lease expired, reclaimed
    failed --> pending: backoff elapsed, attempts < max
    failed --> dead: attempts exhausted
    failed --> cancelled: operator
    dead --> pending: operator requeues (#56)
    succeeded --> [*]
    cancelled --> [*]

running → pending is the crash path: a worker dies holding a lease, the lease expires, the next claim cycle moves it back to pending and increments attempts. Because that edge exists in the transition table, reclaiming is a legal move rather than a special case someone has to remember to allow.

Implementation, #56. dead → pending is the one edge requeueJob (backends/crates/sms-api/src/procedures.rs, admin Jobs screen) is allowed to propose — an operator's explicit "try this again," resetting attempts to 0 and clearing lastError/leaseOwner/leaseUntil so the job gets a fresh run at its full maxAttempts budget rather than resuming an already-exhausted counter. Nothing in the automatic pipeline ever writes this edge; jobs::apply_failure only ever reaches dead from failed, never leaves it. dead is therefore no longer a true terminal state, the same way #43 made WebhookAttempt's dead non-terminal — only succeeded/cancelled are, for Job.

Job kinds, all enqueued by the scheduler role with a dedupeKey:

kindCadenceDoes
expire_stale1 minsubmitted/uncertain past validity → expired
poll_balance5 minOrange /contracts; emit balance.low
probe_providers1 minPer-provider health → Provider.healthy
reap_outbox1 hDelete delivered cratestack_event_outbox rows >24h; alarm on high-attempts rows
reconcile_clients1 hOP client rows with no matching AppClient → orphan alert
purge_retentiondaily#67, done. Terminal Message rows past 90 days: null body/clientRef/idempotencyKey/stateReason, overwrite msisdn with a placeholder (kept NOT NULL), stamp purgedAt. msisdnHash survives — see §10 — and no lookup by number matches a purged row (lookupDeliveryStatus excludes purgedAt rows and refuses a window older than 90 days, §5.2): the keyed hash is what de-identifies the row. Delete DeliveryReceipt rows past their own receivedAt + 90 days
cleanup_secrets1 hDeactivate AppClient past retiredAt. Not built for the other half of its original job: the job that would clear an hmac endpoint's prevSecret after the overlap window; until it exists the overlap ends at the next rotation (§4.4)
anchor_auditdailyMerkle root of the day's audit rows → append-only store
verify_backupdailyRestore last night's dump into a scratch database and count rows

The backup-verify job is not optional. An untested backup is a hypothesis.

7.6 The Orange TPS ceiling ​

Worth restating because it constrains the whole worker design. Orange's 5 TPS cap is per contract, not per connection — hence the singleton dispatch role. If you outgrow one instance: move the token bucket to Redis so the budget is genuinely shared, or shard dispatch by provider so exactly one worker owns each provider's bucket. Sharding is simpler and probably enough for a long time, and it maps naturally onto the advisory-lock scheme — one lock key per provider instead of one for the role.

A second, non-obvious ceiling: Orange top-ups are capped at 100,000 FCFA/day per SIM. At all-operator rates that's roughly 4,500–6,000 SMS/day of self-service headroom regardless of TPS. Past that you need a commercial contract or a second channel — a procurement lead time, not a config change.

8. Events via Subscriptions ​

8.1 What @@emit actually is ​

On the Postgres backend, @@emit is a transactional outbox — not the in-process bus the docs imply. From cratestack-sqlx/src/query/write/create.rs:

rust
let mut tx = self.runtime.pool().begin().await?;
if emits_event { ensure_event_outbox_table(&mut *tx).await?; }
let record = create_record_with_executor(&mut *tx, ...).await?;
if emits_event {
    enqueue_event_outbox(&mut *tx, schema_name, ModelEventKind::Created, &record).await?;
}
tx.commit().await?;                                   // <-- boundary
if emits_event { let _ = self.runtime.drain_event_outbox().await; }

The event row lands in cratestack_event_outbox inside the mutation's transaction, so it's atomic with the write and survives process death — an undelivered row is picked up by the next drain. run_in_tx rollback unwinds row, audit and outbox together. That's a genuine at-least-once outbox, which is why there's no hand-rolled OutboxEvent model in §2.

Subscription is a callback registry:

rust
db.events().on_message_updated(|event: MessageUpdatedEvent| async move {
    // event.data is the full post-update Message row
    enqueue_webhook_attempts(&pool, &event).await
});

db.events().on_opt_out_created(|event| async move { ... });

ModelEvent<T> carries event_id, model, operation, occurred_at, and data: T — the full row, not just the PK.

8.2 What it is not ​

Four properties that shape everything downstream. All verified in source; none are in the docs.

Delivery is synchronous and blocks the mutation. drain_event_outbox sequentially awaits each envelope, and CratestackEventBus::emit sequentially awaits each handler, inside the caller's future, after commit. There is no tokio::spawn anywhere. A slow subscriber adds its latency directly to sendMessage.

Handlers are not panic-isolated. No catch_unwind. A panicking subscriber unwinds out of run() into the caller's task after the row has committed — turning a successful write into an apparent API failure.

emit short-circuits on the first failing handler, and a failed row retries from the top of the handler list on the next drain. Handlers registered after the failing one never see that event on that pass; handlers before it get it twice. Idempotency is mandatory, not defensive.

There is no FOR UPDATE SKIP LOCKED on the drain, and delivered_at is only set after handlers complete. Two drainers both read the same undelivered rows and both deliver them. Across processes this is a competing-consumer queue, not a fan-out: an event reaches one arbitrary drainer, or several, but never reliably all of them.

Also: no background drain worker exists — drain is triggered only by the tail of a mutation on an emitting model, or an explicit events().drain(). And attempts/last_error are recorded but never read: no retry cap, no backoff, no dead-letter. A permanently failing handler retries that row forever and the table grows without bound.

8.3 The design that follows ​

mermaid
sequenceDiagram
    autonumber
    participant W as sms-worker<br/>dispatch
    participant PG as Postgres
    participant SUB as sms-api<br/>subscriber
    participant H as sms-worker<br/>hooks
    participant CUST as Customer endpoint

    rect rgb(238, 244, 252)
        note over W,PG: one transaction
        W->>PG: UPDATE message → delivered<br/>(trigger validates edge)
        W->>PG: INSERT cratestack_event_outbox
    end

    note over PG,SUB: drain role, every 5s
    SUB->>PG: SELECT undelivered outbox rows
    SUB->>SUB: map post-update state → event type
    SUB->>PG: INSERT webhook_attempts<br/>ON CONFLICT DO NOTHING
    SUB->>PG: mark outbox row delivered

    note over H,CUST: independent loop, scales to N
    H->>PG: claim attempts, SKIP LOCKED + lease
    H->>CUST: POST signed payload
    alt 2xx
        CUST-->>H: 200
        H->>PG: attempt → succeeded
    else failure
        CUST-->>H: 5xx / timeout
        H->>PG: attempt → failed, backoff<br/>or → dead at max attempts
    end

Subscribers do the minimum possible and nothing that can block or fail. A webhook subscriber makes no HTTP call — it inserts WebhookAttempt rows and returns. The hooks role does the slow, failure-prone part on its own schedule. Putting an HTTP call in a subscriber would mean a customer's slow endpoint adds latency to your send API.

Idempotency is enforced in the database, not the handler. The unique index on (endpoint_id, aggregate_id, event_type) absorbs all three duplicate sources at once: concurrent drains, retry-after-failure re-delivery, and Message.updated firing on every update — including ones that don't change state — each with a fresh event_id. Keying on event_id would produce a webhook per touch; keying on aggregate + derived type produces exactly one message.delivered per message per endpoint.

ON CONFLICT DO NOTHING isn't expressible through the delegates — upsert doesn't exist on a model whose @id has a default, and it would overwrite rather than ignore anyway. The delegate-native form is create plus a catch on the unique violation, which is precisely what db_sqlstate() is for:

rust
match db.webhook_attempt().create(input).run(sys).await {
    Ok(_) => {}
    // 23505 unique_violation on webhook_attempts_dedupe: already enqueued
    // by another drain, or by an earlier update to the same message.
    Err(e) if e.db_sqlstate() == Some("23505") => {}
    Err(e) => return Err(e),
}

Branching on the SQLSTATE rather than substring-matching the message is the whole reason CratestackError::DatabaseTyped carries DbErrorInfo { detail, sqlstate, constraint }. Check constraint too if more than one unique index can fire on this table.

Handlers cannot panic. Every subscriber body is wrapped so a bug becomes a logged error and an Err return, never an unwind. An Err leaves the row undelivered for retry, which is what you want; a panic corrupts the caller's response, which is not.

The drain role is a singleton for the same reason. Nothing breaks if two drain — the unique index catches it — but every duplicate is wasted work and a wasted index probe.

Opt-out cache invalidation does not use Subscriptions. This is where competing-consumer semantics bite: an OptOut.created event reaches one process, so using it to invalidate an in-memory suppression cache leaves the other sms-api replicas stale. The subscriber only enqueues the optout.created webhook. Cache coherence uses a 30-second TTL plus Postgres LISTEN/NOTIFY, which genuinely fans out.

8.4 Event catalogue ​

text
message.accepted     message.submitted    message.delivered
message.failed       message.expired      message.uncertain
message.cancelled
provider.degraded    provider.recovered
balance.low          quota.exceeded
sender_id.approved   sender_id.rejected
optout.created

Derived by mapping the post-update state — updated events carry no before-image, only the post-update row, so "what changed" is inferred from the new state. Where the previous value genuinely matters, read cratestack_audit, which does capture before/after.

json
{
  "id": "c8f2a1b3d4e5f60718293a4b5",
  "eventId": "6f1d6e0e-5a59-4d8c-9b5a-3e8f0c1d2a4b",
  "type": "message.delivered",
  "occurredAt": "2026-07-28T14:03:11Z",
  "appId": "c9c1eb3d4e5f60718293a4b5c",
  "endpointId": "cd7e0f1a2b3c4d5e6f7a8b9c0",
  "data": {
    "messageId": "c4f2a1b3d4e5f60718293a4b5",
    "appId": "c9c1eb3d4e5f60718293a4b5c",
    "clientRef": "otp-login-4471",
    "to": "+2376xxxxx89",
    "state": "delivered",
    "operator": "orange",
    "segments": 1,
    "deliveredAt": "2026-07-28T14:03:09Z",
    "costXaf": "22.00"
  }
}

id (WebhookAttempt.id) is the dedupe key, eventId the id of the event that produced it, and appId and endpointId name the endpoint the delivery was made through: all four sit in a sealed delivery's signed body (§4.4), and a receiver compares endpointId (the binding) and appId with its own. (An hmac delivery carries id, type, occurredAt and data; its event id is the signed X-Sms-Event-Id header, and it holds its endpoint's own secret, so it has no appId or endpointId to compare.) to is masked unless WebhookEndpoint.maskRecipient is false. Most consumers correlate on clientRef and don't need the number.

8.5 Outbound delivery ​

Backoff: 1s, 5s, 25s, 2m, 10m, 1h, 6h, 24h — eight attempts, then dead. Timeout 10s per attempt. 2xx is success; 410 Gone deactivates the endpoint immediately; everything else retries.

Per-endpoint circuit breaker: 20 consecutive failures sets circuitOpenUntil = now + 15min and stops attempting that endpoint. Rows are still created as pending, so nothing is lost — they're just not attempted. Alert the app owner.

Replay from the admin console re-queues any attempt with a fresh counter, so a correctly-implemented receiver dedupes it. In practice (#43) this is a same-row reset, not a new row: WebhookAttempt's own dedupe index (endpoint_id, aggregate_id, event_type) means a second row for the same event can never exist, so replay finds the existing row and resets it in place — WebhookAttempt.id (the envelope's own id, §8.4) and sourceEventId are therefore both unchanged by a replay, not just sourceEventId.

Deliver in order per endpoint where practical, but document that ordering is not guaranteed. Receivers must tolerate message.delivered arriving before message.submitted. Saying so in the docs is much cheaper than the support thread you'll otherwise have.

Implementation, #40 (backends/crates/sms-worker/src/hooks.rs), landed against the above:

mermaid
stateDiagram-v2
    [*] --> pending: enqueued by a subscriber (#38)
    pending --> delivering: claimed
    failed --> delivering: claimed, backoff elapsed
    delivering --> succeeded: 2xx
    delivering --> failed: retryable — backoff scheduled
    delivering --> dead: max attempts exhausted, or 410 Gone
    failed --> pending: replayed by an operator (#43)
    dead --> pending: replayed by an operator (#43)
    succeeded --> [*]

attempt_state_transitions + attempts_guard_transition (§2.10) enforce this the same way messages/jobs are enforced — the table shipped with #38/#39's AttemptState enum, but nothing drove the column until this PR, so no transition table existed to decide against. A crash-abandoned delivering lease is reclaimed the same way Message's routed state is: a same-state write that renews the lease without incrementing attempts, needing no row in the table (same-state writes bypass the guard).

maxAttempts is read from WebhookEndpoint, per candidate, not a constant — the backoff schedule above is fixed and shared, but how many entries into it a given endpoint gets before dead is that endpoint's own column. An endpoint's attempts counter is incremented once, at claim time (pending/failed → delivering); the crash-reclaim same-state write does not increment it again, since it resumes an attempt already counted rather than starting a new one.

What resets WebhookEndpoint.consecutiveFailures. Two things, both explicit: a successful delivery resets it to zero (and clears circuitOpenUntil, defensively); crossing the 20-failure threshold that opens the circuit also resets it to zero, so the cool-down period is followed by a fresh 20-failure allowance rather than the very next post-cooldown failure reopening the circuit immediately. circuitOpenUntil is otherwise left untouched by an ordinary failure below the threshold — by construction, hooks never attempts a candidate whose circuit is already open (see below), so a failure is never recorded against an endpoint the breaker had already tripped.

Candidate selection excludes both an inactive endpoint and an endpoint whose circuit is currently open, application-side (backends/crates/sms-worker/src/claim.rs's Claimable for WebhookAttempt::candidates) rather than as a join in the claim query itself — WebhookEndpoint has no @version, so WebhookAttempt.leaseUntil/attempts are what the CAS claim actually contends on. The endpoints are read first and such an endpoint is left out before any attempt is read: a window of the oldest due rows that was filtered afterwards would be all of one unhealthy endpoint's backlog whenever it held as many due attempts as the window (nothing purges them), and the claim would take nothing for every app. A last re-read of the chosen attempts' endpoints, just before the leases, catches a change in between. This is exactly "rows are still created as pending, so nothing is lost — they're just not attempted": the rows of a left-out endpoint are simply not selected, and are reconsidered as soon as the endpoint is claimable again (backends/crates/sms-worker/src/claim.md has the two query shapes and their measurements).

A sealed delivery's body is the full {id, eventId, type, occurredAt, appId, endpointId, data} envelope §8.4 shows (§4.4), and an hmac delivery's the four-field {id, type, occurredAt, data} its receivers have always verified (eventId rides in the signed X-Sms-Event-Id header). Either is built at delivery time from the stored WebhookAttempt row — data is payload (already masked per maskRecipient at insert time by #38, §2.7), id is WebhookAttempt.id (the only value that has existed since the row was created — see §2.7's own note on why the outer id can't be anything else), eventId is sourceEventId, type is eventType, appId and endpointId are the endpoint's own. occurredAt is a documented approximation, not the original event's timestamp: WebhookAttempt carries no creation timestamp (no @use(Timestamps)), and the framework's own ModelEvent::occurred_at is read and discarded by #38's subscriber before the row exists to store it on. hooks therefore stamps occurredAt with the time of this delivery attempt — accurate for a first attempt seconds after the event, increasingly approximate under retries, and wrong by up to the full backoff schedule's span (up to 24h) for an attempt that only succeeds on its last try. Closing this properly needs WebhookAttempt to gain a stored event timestamp (a real schema decision touching a model #38 owns, not a mechanical follow-on) or #38's subscriber to start threading ModelEvent::occurred_at through to a new column — flagged here as a real, tracked gap, not silently accepted.

Sealing (sms_webhook::Sealer::seal) always covers the exact bytes sent as the request body — the envelope is serialized once and that same String's bytes are what goes inside the COSE_Sign1 message, which is what reqwest sends, never two independent serializations of the same logical value (the "signing something other than what you send" bug class #41's module doc warned about, which the seal makes structurally impossible: the sealed bytes are the request body).

Implementation, #43 (replayWebhookAttempt, backends/crates/sms-api/src/procedures.rs), landed against the above:

The design question this story has to resolve before writing any code: given webhook_attempts_dedupe forbids a second row for the same (endpoint_id, aggregate_id, event_type), replay cannot insert a fresh attempt — it can only reset the existing row in place. That is exactly what makes the "same event id, same signature semantics" requirement free rather than something to engineer: nothing about WebhookAttempt.id (the envelope's own id, §8.4) or sourceEventId ever changes, because no new row is ever created.

Two new edges, failed -> pending and dead -> pending (§2.10) — and, deliberately, no succeeded -> pending. A failed attempt would eventually retry on its own once its backoff elapses; replay's value there is forcing that retry now, on an operator's schedule rather than the backoff schedule's. A dead attempt never retries on its own at all — replay is the only way back for it. Both land the row in the same place: pending, with attempts reset to 0 (a fresh counter, per this section's own prose above) and lastStatusCode/lastError/leaseOwner/leaseUntil cleared. succeeded -> pending is not added: a receiver that already got this event and (most likely) already acted on it is a materially different, riskier thing to re-fire than one that never got through, and this story is scoped to the latter — "re-fire a failed delivery," not "re-send a delivered one." Both the procedure and the transition table enforce this the same way twice over: replayWebhookAttempt only ever proposes failed -> pending or dead -> pending, and even if it proposed anything else, no other edge exists for Postgres to accept — attempting to replay a pending, delivering, or succeeded row is a 409 Conflict (SQLSTATE SM001, mapped by backends/crates/sms-api/src/errors.rs::map_database_error — the first procedure in this codebase to actually call it; every earlier write path either cannot reach an illegal edge or, like cancelMessage, isn't implemented yet).

What a receiver sees on a replayed request: the same envelope id and eventId, in a new sealed request — a new nonce and a new timestamp, as for every retry (§4.4). The dedupe burden this puts on the receiver is exactly what §8.5's opening line already asks of it — deduplicate on the envelope's id, not on the nonce, the timestamp or the bytes. A receiver that dedupes correctly (by id) treats a replay as the redelivery it is; one that dedupes on the sealed bytes would wrongly treat every retry and every replay as a distinct event.

Circuit breaker: replay clears it, not just the one attempt. An operator invoking replay is, by definition, asserting "I fixed the receiving end" — exactly the condition the breaker exists to wait for, and the case named explicitly in this story's own brief as the one where a stale breaker must not silently swallow the operator's action. Rather than teach claim.rs's candidate filter a one-off "except this specific attempt" bypass (touching the shared claim path #40 and #44 both depend on, for one procedure's benefit), replayWebhookAttempt resets the endpoint's consecutiveFailures to 0 and clears circuitOpenUntil in the same transaction, whenever either is currently set — the same reset hooks.rs::reset_endpoint_failures already performs on an ordinary successful delivery, just performed eagerly on the operator's word rather than waiting for proof. This unblocks every other pending/failed attempt against that endpoint too, not only the one replayed — deliberate: if the endpoint is fixed, every message stuck behind its breaker should get another chance, not just the one an operator happened to click on. It does not touch WebhookEndpoint.active — an endpoint deactivated by a 410 Gone stays deactivated, and replaying an attempt against it lands the row in pending where it sits, uninspected by claim.rs's health filter, until an operator separately reactivates the endpoint (the existing PATCH /webhook_endpoints/{id} route). Reactivation is a distinct, more consequential decision than "retry this one delivery," and conflating the two would mean a replay click silently reviving an endpoint nobody asked to revive.

Who may replay. Same three human roles that manage the endpoint itself (owner/admin/developer, §5.2). On top of that, replayWebhookAttempt calls require_permission(ctx, "webhook:manage") (Layer 2, §5.1) before doing anything else. webhook:manage is developer's own permission per §5.2's table; owner/admin hold it implicitly as part of "everything"/"all". Since 0013_role_read_scope a developer cannot replay from the console: @authorize(WebhookAttempt, detail, ...) reads the attempt as the caller and its policy asks for the messageRead claim, which a role without sms:read does not have (§5.2); owner and admin replay. As with PATCH /providers/{id} (#24), this deployment's GatewayAuth never issues a real token carrying anything but role: "app"/"system" — no human-login path exists yet — so Layer 1 alone already closes this procedure to every token this deployment can currently issue; Layer 2 is real and tested, but defense in depth rather than the thing actually stopping a live request, until a role-bearing token exists.

#193: rotateWebhookSecret carries the identical require_permission(ctx, "webhook:manage") gate, and refuses a sealed endpoint (VS-8R, §4.4).

Scope boundary. This is the API-side capability only — a permission-gated procedure and the state-machine edges it needs. The admin console's own webhooks screen (#55, milestone 4) is not built here; a minimal API surface is all this story owns.

9. Operations ​

9.1 Observability ​

Landed in #70/#71, corrected against what actually shipped rather than left as the original aspirational prose — the five bolded items below are real (backends/crates/sms-metrics, a Prometheus text endpoint on both binaries, deploy/prometheus/alerts.yml); everything else in this section is still the M6 target, not yet built, and the two paragraphs after the lists are the specific corrections. #64 added two more of each — delivery-rate-by-pair, one of the items this section's own "still aspirational" list originally named — see that section's own note below.

  • Tracing: tracing, no OpenTelemetry (no collector infrastructure exists anywhere in deploy/, and one was not needed for what shipped — see below). cratestack_request_id (an inbound X-Request-Id header, honoured verbatim, or freshly minted — backends/crates/sms-api/src/auth.rs::request_id_from) ties every cratestack_*-logged line together for one HTTP request, within sms-gateway. It does not propagate into sms-worker or back through the DLR — those are separate processes with no shared span context, and the request that created a message is long finished by the time a worker submits it or a DLR arrives. Message.id is the join key across that boundary instead: backends/crates/sms-api/src/procedures.rs::send, backends/crates/sms-worker/src/dispatch.rs::submit_one, and backends/crates/sms-api/src/dlr.rs::ingest_one each log a message_id-carrying event. See docs/runbooks/alerting.adoc's "Correlating a message end to end" section for the worked example — this is real, grep-able correlation, not a claim of distributed tracing in the OpenTelemetry sense.
  • Metrics: five landed with #70/#71, two more with #64 — SM001 rejection count by from/to pair (sms_sm001_total, entity/from_state/to_state labels), advisory lock holder per singleton role (sms_worker_singleton_lease_held{role}), concurrent in-flight dispatch submits per provider (sms_dispatch_in_flight_submits{provider}), webhook outbox oldest-undelivered age (sms_webhook_outbox_oldest_undelivered_age_seconds), poison event-outbox row count (sms_event_outbox_poison_rows), route delivery-rate divergence count (sms_route_delivery_divergence_flagged, #64 — see §6.4 and backends/crates/sms-worker/src/jobs/grey_route_watch.rs), and overdue handset-validation route count (sms_route_validation_overdue, #64). Everything else in this original list — submit rate/latency by provider, time-to-delivery percentiles, queue depth, DLR lag, jobs pending depth, webhook success rate, provider balance, segments per message, UCS-2 ratio, /token failure rate — is still aspirational; #70's own five named alert conditions plus #64's own named one are what metrics work has scoped to, deliberately, not this full list.
  • Alerts that matter: seven metrics, seven real, loadable Prometheus rules in deploy/prometheus/alerts.yml — any singleton role unheld for more than 30s; a non-zero SM001 rate, which means code is proposing transitions the machine forbids; unexpected concurrent dispatch submits (a fleet-wide sum above 1, sustained); outbox oldest undelivered above 2 minutes, not the 60s this line originally said (backends/crates/sms-worker/src/drain.rs::STALLED_THRESHOLD is the implemented, real threshold — see docs/runbooks/alerting.adoc for why the alert matches the code, not this stale prose figure); any row with attempts > 5 (reap_outbox's own threshold, #42); a route's delivery rate diverging, past a sample-size and statistical-significance gate, from its best-performing operator/class peer (#64); an enabled route with no real-handset validation in 30 days (#64). Everything else in this original line — OTP p95, queue oldest-age, provider balance, DLR silence, webhook dead-letter rate — has no metric behind it yet and therefore no rule either.

The SM001 metric is the highest-signal one in the list. In a correct system it is flat zero — the trigger is a backstop, not a control path. Any non-zero rate means application logic and the transition table disagree, and it will tell you that before a customer does.

"Alerting" does not mean this repository can page anyone. No Alertmanager, no receiver, no Slack/PagerDuty integration exists anywhere in this tree — a real Prometheus (the prometheus service in deploy/docker-compose.yml) genuinely evaluates the five rules and shows firing state on its own /alerts page, and an operator wires a receiver on top of that themselves. Building a bespoke in-process alerting engine was considered and rejected as the wrong shape of deliverable for #70.

The UCS-2 ratio deserves its own tile. A sudden jump means someone shipped a template with a ç or a smart apostrophe, and it will show up in your bill before anyone notices in the UI.

That tile — and the rest of #49's Dashboard screen — landed separately from #70/#71's Prometheus work, deliberately. Throughput, queue depth, job backlog, outbox depth, delivery rate by operator, and the UCS-2-ratio trend (as a real six-hour bucketed trend, not a bare percentage — see backends/crates/sms-api/src/procedures.rs's own doc on dashboard_snapshot) are a new dashboardSummary procedure reading Message/Job/WebhookAttempt through cratestack's own aggregate().count() (no GROUP BY exists in cratestack-sqlx =0.7.10, so this is ~26 small, policy-scoped COUNT queries per snapshot, cached 15s server-side), not new Prometheus metrics — the two systems answer different questions: Prometheus's five metrics are pager-facing invariants ("is anything on fire"), the dashboard is an operator's situational-awareness screen ("how's traffic looking"), and the two never recompute the same number a second, possibly-disagreeing way (the dashboard's own outbox-depth tile is explicitly a different table from sms_webhook_outbox_oldest_undelivered_age_seconds/sms_event_outbox_poison_rows — see that tile's own caption). Provider balance remains genuinely unavailable — poll_balance (§7.5) was never built — and the dashboard says so rather than showing a fabricated number.

9.2 Deployment ​

mermaid
flowchart LR
    subgraph vm["Single VM to start"]
        CADDY["caddy"]
        subgraph scale["scale to N"]
            API1["sms-api ×2"]
            HOOKS["sms-worker<br/>--roles hooks,jobs"]
        end
        subgraph single["singletons"]
            W1["sms-worker<br/>--roles dispatch,drain,<br/>scheduler,hooks,jobs"]
        end
        ADMINN["admin (Next.js)"]
        PGN[("postgres 16")]
        PROM["prometheus<br/>#70/#71"]
    end
    S3[("object storage<br/>WAL archive + dumps")]

    CADDY --> API1
    CADDY --> ADMINN
    API1 --> PGN
    W1 --> PGN
    HOOKS --> PGN
    PGN -.->|"WAL + nightly dump"| S3
    PROM -.->|"scrapes /metrics,<br/>never fronted by caddy"| API1
    PROM -.->|"scrapes /metrics,<br/>never fronted by caddy"| W1

Docker Compose on one well-specified VM. Postgres with WAL archiving to object storage plus nightly pg_dump. Kubernetes only when you have a reason that isn't résumé-driven.

Note the second worker runs only scalable roles. It also attempts the singleton locks in the background and will take over within seconds if the first node dies — so it's a warm standby and a scale-out node at the same time, with no separate failover configuration.

Hosting location is a legal question here, not a latency question. See §10.

9.3 CI gates ​

yaml
- cargo clippy --workspace --all-targets -- -D warnings
- cargo test --workspace
# R1: no raw sqlx outside the three named exceptions
- ./ci/assert-no-raw-sqlx.sh
# migrate diff has NO --check mode: regenerate into a temp dir, fail if non-empty
- ./ci/assert-no-schema-drift.sh
# every edge in the mermaid state diagram exists in the transition table, and vice versa
- cargo test -p sms-core --test state_machine_parity
- cratestack diff schema/schema.prev.cstack schemas/vsms.cstack
- cratestack generate-typescript --schema schemas/vsms.cstack --out frontends/packages/sms-client \
    --package-name @sms/client --base-path /v1 --full-selection --check
- npm --prefix admin run build && npm --prefix admin run test

assert-no-raw-sqlx.sh is the R1 lint, and the allowlist lives in the script rather than in #[allow] attributes scattered through the code — the point is that the exception list is short, visible in one place, and reviewed as a unit:

bash
#!/usr/bin/env bash
set -euo pipefail
hits=$(grep -rn --include='*.rs' -E 'sqlx::(query|query_as|query_scalar|raw_sql)\b' crates/ \
       | grep -vE 'sms-worker/src/(lease|notify)\.rs|sms-api/src/cache\.rs' || true)
if [ -n "$hits" ]; then
  echo "R1 violation — raw sqlx outside the named exceptions:" >&2
  echo "$hits" >&2
  exit 1
fi

assert-no-schema-drift.sh exists because migrate diff has no check mode: run it into a temp --out-dir and fail if it emitted anything.

state_machine_parity is worth the twenty lines. It loads message_state_transitions from the migration and compares it against the Rust enum's declared edges, failing if either side has an edge the other doesn't. Without it the trigger and the code drift, and the symptom is a production SM001 on a transition everyone assumed was legal.

--full-selection matters: by default every scalar in the generated TS interfaces is optional, because partial fields/include projection makes absence a real state. The admin fetches full objects, so you want the non-optional shape. --check generates in memory and diffs, failing on drift.

9.4 Configuration (VSC-1) ​

sms-gateway serve, sms-gateway mtn-subscribe-dlr and sms-worker read one YAML document, config/application.yml plus a profile overlay config/application-<profile>.yml, through the sms-config crate. It is modelled on vpay's vpay-config and draws the same line:

Process plumbing: stays a flag or a variableDeployment behaviour: the YAML
DATABASE_URLdatabase.gateway_max_connections, database.worker_max_connections
--listen / SMS_LISTEN_ADDR, --metrics-listen / SMS_METRICS_LISTEN_ADDR / SMS_WORKER_METRICS_LISTEN_ADDRsecurity.hash_pepper
SMS_ENVELOPE_SIGNING_KEY_FILE, SMS_ENVELOPE_NEXT_SIGNING_KEY_FILE (paths)api.idempotency_ttl_secs, api.rate_limit, api.source_rate_limit
SMS_WORKER_ROLES, SMS_WORKER_ID, SMS_WORKER_HEALTH_FILEproviders: every provider's host, settings and credentials
RUST_LOGdeployment.name, .livemode, .public_base_url
SMS_CONFIG, SMS_PROFILE

The one-shot operator subcommands (register-client, bootstrap, create-app, provision-user, pair-console-key, seed-dispatch, record-route-validation) take their arguments as arguments and read no deployment configuration.

mermaid
sequenceDiagram
    participant Main as binary main
    participant Flags as retired-flag check
    participant Loader as Config::load
    participant Env as process environment
    Main->>Flags: argv, before clap parses
    Flags-->>Main: refuse a retired flag, naming its YAML key
    Main->>Loader: --config, --profile
    Loader->>Loader: merge application.yml with application-profile.yml
    Loader->>Env: is a retired variable set and not read as a placeholder?
    Env-->>Loader: refuse it, naming its YAML key
    Loader->>Env: resolve every ${VAR}
    Env-->>Loader: an unset one is fatal, never an empty string
    Loader->>Loader: typed model, unknown keys refused, then the cross-field rules
    Loader-->>Main: Config, or the first rule it broke
    Main->>Main: only now open key files, the pool and the listeners

Profiles. --profile / SMS_PROFILE (default dev) names a file and selects no code path. The images bake config/ at /config and set SMS_CONFIG=/config/application.yml; the profile is never baked. A profile that names no application-<profile>.yml is an error that names the file it expected and says to set SMS_PROFILE, so a typo (prodd) cannot boot on the base alone and look like a choice; the default profile is the one exemption, and may run on the base file alone. The default is dev and stays so (vpay defaults to sandbox on the same reasoning: the default points at fakes and is never livemode). Its price is that a release image run without SMS_PROFILE loads the dev overlay, and the gateway then stops on the first thing in it a release build refuses; that message names the loaded profile and SMS_PROFILE. Maps merge key by key; a list in the overlay replaces the base's list wholesale, which is why each profile lists its providers in full.

ProfileUsed byProviders
devcompose.dev.yaml, and the defaultorange_cm and aws_sms against the two local fakes
democompose.demo.yamlorange_cm against the fake Orange
proddeploy/docker-compose.yml (mounted over the baked file)orange_cm, livemode: true; mtn_cm and aws_sms as commented examples that a test enables and loads

The Helm chart ships no profile: it renders config.overlay into a ConfigMap and mounts it over application-<profile>.yml in the gateway and the worker (docs/runbooks/deployment.adoc, "Kubernetes (Helm)").

Keys. Every block refuses unknown keys, so a misspelt key is an error naming it, not a setting silently never applied. The defaults are the base file's, and are the defaults the clap flags had (backends/crates/sms-config/tests/defaults.rs).

KeyDefaultNotes
deployment.namevsmslabel for logs; must not be blank
deployment.livemodefalseturns on the rules below; selects no code path
deployment.public_base_urlhttp://localhost:8080validated and logged; reserved for the issuer a future token realm would derive from it
auth.methodsabsent (all four)which ways of authenticating are permitted (§4.9)
security.hash_pepper${SMS_HASH_PEPPER}at least 32 bytes, checked by sms_api::HashPepper
api.idempotency_ttl_secs86400at least 1
api.rate_limit.{burst, refill_per_second}120, 2.0burst at least 1, refill positive
api.source_rate_limit.{burst, refill_per_second}1200, 10.0
database.gateway_max_connections10at least 1
database.worker_max_connections10at least 1
providers[]see below

Providers. An entry's presence configures the provider, so an incomplete entry is refused naming the missing key, where the flags used to be "all of these or none". A deployment that configures none cannot serve (nothing to parse receipts with) or dispatch.

codehost.urlsettingscredentials
orange_cmdefault https://api.orange.comsender_number (E.164, quoted)client_id, client_secret
mtn_cmdefault https://api.mtn.comservice_code (quoted), tps_ceiling, cost_per_segment_xaf (quote it: money is never a float), sender_id?, supports_alphanumeric_sender (default false)client_id, client_secret
aws_smsendpoint override, optionalsending: region, configuration_set, tps_ceiling, cost_per_segment_xaf, origination_identity?, message_type (default transactional), supports_alphanumeric_sender, time_to_live_seconds?; receiving: events_topic_arn; demo builds: fake_sns_cert_baseaccess_key_id, secret_access_key

tps_ceiling and cost_per_segment_xaf have no default: they are commercial terms of a contract or account, and an invented one throttles a paid contract or misprices estimatedCostXaf. aws_sms has two independent halves, as its two binaries do: the sending keys are all-or-none (naming any configures sending), and the gateway needs only events_topic_arn; an entry with neither is refused. fake_sns_cert_base exists in a fake-aws gateway build only; a release build refuses the key as it refused the flag (which did not parse).

What is refused, one rule each (sms_config::ConfigError, a fixture per rule in backends/crates/sms-config/tests/fixtures):

  • the file is missing, malformed or has a malformed ${...}, or a ${VAR} names a variable the environment does not set;
  • an unknown key anywhere, an unknown code, a duplicate provider, a blank name, a pool of zero connections, a non-positive rate, a negative price, an unparseable URL, an unknown message_type;
  • an incomplete provider entry, an aws_sms entry that sends and receives nothing, and sending-only settings on an entry with no sending keys;
  • under livemode: a URL that is not https, a stub host (localhost, mock, stub, wiremock, fake), a secret written as a literal, and fake_sns_cert_base;
  • a retired flag or variable (below).

The literal-secret rule reads the text as written, before ${VAR} resolves, because afterwards a literal and a resolved placeholder are the same string. cargo xtask config-literal-secrets is its static half: it refuses a committed livemode profile that carries a literal, without building the workspace. It has no YAML parser (xtask depends on regex alone), so it reads lines: a secret key in a block or in a flow mapping (credentials: {client_secret: x}, quoted or not, across lines), and livemode: true in any spelling YAML reads as true (True, TRUE). What it cannot see is a value folded into a literal by YAML itself (a block scalar, a multi-line plain scalar); the loader parses the real document at boot and refuses those, and is the authority.

The worker does not load security. sms-worker never computes a hash, so sms_config::Binary::Worker removes the security block after the retired-variable and literal-secret checks have read it and before placeholders resolve: the worker neither requires nor receives SMS_HASH_PEPPER, and the chart and the three compose stacks do not give it one. sms-gateway mtn-subscribe-dlr loads as Binary::ProviderTool for the same reason: it registers a URL with MADAPI and hashes nothing. Binary::Gateway keeps the block and refuses to start without the pepper. A retired variable is still refused in the worker, and a literal pepper under livemode is still refused there too: those rules read the shared file, not the binary's use of it.

An error is a log line, so it never repeats a secret. A malformed placeholder is reported by YAML key and the offending ${...} fragment (capped), never the string around it, which may be a sandbox literal secret. A secret that YAML reads as a number or a boolean (client_secret: 123456 unquoted) is refused by key before serde sees it, because serde's own message for it would print the value; converting it to text instead would run the process on a secret nobody typed. Other Shape errors echo what serde could not read (an unknown key, a wrong type), which is never one of the three secret keys.

The hard cutover. The flags and variables below are gone. A retired flag is refused before clap parses (clap would otherwise say "unexpected argument" and name no key). A retired variable that is set while the YAML does not read it as ${VAR} is refused by the loader, because clap ignores a variable it no longer declares and a deployment that still set ORANGE_CM_CLIENT_ID would boot with Orange silently missing. A variable set to the empty string counts as set. Both errors name the YAML key. A retired name the YAML does read as a placeholder is a secret's source, which is how the pepper and the provider secrets are still delivered.

Retired flag / variableYAML key
--max-connections / SMS_DB_MAX_CONNECTIONSdatabase.gateway_max_connections
--db-max-connections / SMS_WORKER_DB_MAX_CONNECTIONSdatabase.worker_max_connections
--hash-pepper / SMS_HASH_PEPPERsecurity.hash_pepper (${SMS_HASH_PEPPER})
--idempotency-ttl-secs / SMS_IDEMPOTENCY_TTL_SECSapi.idempotency_ttl_secs
--rate-limit-burst, --rate-limit-refill-per-second / SMS_RATE_LIMIT_BURST, SMS_RATE_LIMIT_REFILL_PER_SECONDapi.rate_limit.{burst, refill_per_second}
--source-rate-limit-burst, --source-rate-limit-refill-per-second / SMS_SOURCE_RATE_LIMIT_BURST, SMS_SOURCE_RATE_LIMIT_REFILL_PER_SECONDapi.source_rate_limit.{burst, refill_per_second}
--orange-client-id, --orange-client-secret / ORANGE_CM_CLIENT_ID, ORANGE_CM_CLIENT_SECRETproviders[code=orange_cm].credentials.{client_id, client_secret}
--orange-sender-number, --orange-base-url / ORANGE_CM_SENDER_NUMBER, ORANGE_CM_BASE_URLproviders[code=orange_cm].settings.sender_number, .host.url
--mtn-client-id, --mtn-client-secret / MTN_CLIENT_ID, MTN_CLIENT_SECRET (also mtn-subscribe-dlr)providers[code=mtn_cm].credentials.{client_id, client_secret}
--mtn-service-code, --mtn-sender-id, --mtn-tps-ceiling, --mtn-cost-per-segment-xaf, --mtn-supports-alphanumeric-sender / MTN_SERVICE_CODE, MTN_SENDER_ID, MTN_TPS_CEILING, MTN_COST_PER_SEGMENT_XAF, MTN_SUPPORTS_ALPHANUMERIC_SENDERproviders[code=mtn_cm].settings.*
--mtn-base-url / MTN_BASE_URLproviders[code=mtn_cm].host.url
--aws-sms-region, -configuration-set, -tps-ceiling, -cost-per-segment-xaf, -origination-identity, -message-type, -supports-alphanumeric-sender, -time-to-live-seconds / AWS_SMS_* of the same nameproviders[code=aws_sms].settings.*
--aws-sms-access-key-id, --aws-sms-secret-access-key / AWS_SMS_ACCESS_KEY_ID, AWS_SMS_SECRET_ACCESS_KEYproviders[code=aws_sms].credentials.*
--aws-sms-endpoint-url / AWS_SMS_ENDPOINT_URLproviders[code=aws_sms].host.url
--aws-sms-events-topic-arn / AWS_SMS_EVENTS_TOPIC_ARNproviders[code=aws_sms].settings.events_topic_arn
--aws-sms-fake-sns-cert-base / AWS_SMS_FAKE_SNS_CERT_BASEproviders[code=aws_sms].settings.fake_sns_cert_base

The authoritative list is sms_config::RETIRED. Each binary's tests fail if its clap definition declares any entry again. mtn-subscribe-dlr reads MTN's credentials from the mtn_cm entry, so it loads every block but security: run it where the gateway's environment is, because every ${VAR} it writes must be set, the pepper's excepted.

Where the document goes. Compose: the images bake it; deploy/docker-compose.yml additionally mounts config/application-prod.yml into both services as a top-level configs: entry, so an edit needs a restart and not a rebuild, and forwards the variables that file reads. Kubernetes: config.overlay, as above. A test reads the compose files and the chart's values as data and fails when a stack stops forwarding a variable its profile reads, or sets a retired one.

Adding a key. Add the field to the model with its default in config/application.yml (not in Rust), a fixture for each rule it has, a row above, and, if it replaces a flag, an entry in RETIRED. Nothing may be added that no code reads.

auth.methods (VSA-1, §4.9) is the one key whose absence means something: absent permits every method, so the base file carries it as a comment and a test (defaults.rs) fails if it is ever written there. A misspelt auth_methods:, or an auth: nested under another block, is an unknown key, so a deployment cannot believe it narrowed something it did not.

10. Compliance — read this before choosing a host ​

Law No. 2024/017 of 23 December 2024 (data protection). The 18-month transition expired around 23 June 2026; it should be fully enforceable now. Three provisions reshape the architecture:

  • Consent is opt-in and mandatory, and "legitimate interest" is not a recognised lawful basis — stricter than GDPR. In practice you need explicit consent for essentially all processing, including merely holding a phone number to notify it.
  • Prior registration with the Authority before processing personal data.
  • All cross-border transfers require prior authorisation. Hosting the gateway and its database outside Cameroon is a cross-border transfer. So is routing through an offshore aggregator, or sending through a cloud SMS service such as AWS End User Messaging (§6.2): every recipient number and body leaves the country. Host in Cameroon, or obtain the authorisation.

Sanctions run to 100,000,000 FCFA, suspension, withdrawal of authorisation, and criminal penalties up to ten years. Whether the supervisory Authority is operational and accepting registrations as of today is unverified — confirm with local counsel.

Law No. 2010/012 of 21 December 2010 (cybersecurity). Article 48(1) prohibits marketing messages that conceal the sender's identity or omit a valid address for opt-out — your legal basis for a registered sender ID and a working STOP path. Article 48(2) prohibits sending by usurping another's identity (sender-ID spoofing). Article 25(1) requires operators and electronic communications service providers to retain connection and traffic data for ten years, in direct tension with the 90-day minimisation in §2.5.

Resolved by the maintainer, 2026-08-11 (issue #5): 90-day minimisation, no split ledger. A parallel ten-year traffic-metadata table (timestamp, hashed MSISDN, operator, segments, state) — kept alongside a 90-day purge of content and plaintext MSISDN — was the recommendation this section used to carry, and it was a real, considered option, not a strawman: it is the shape most likely to satisfy Article 25(1) and Law No. 2024/017's minimisation at once. It was not taken. vsms purges content and plaintext MSISDN both at 90 days (purge_retention, #67, backends/crates/sms-worker/src/jobs/purge_retention.rs) and does not carry a second, longer-horizon ledger anywhere in schema.cstack. If Article 25(1) compliance requires holding traffic data past 90 days, that is now an infrastructure concern for whoever operates the deployment — database backup/archival retention configured outside this application — not something vsms's own schema does on their behalf. Nothing in this repository fails if nobody configures that; see issue #5's own resolution comment for the tradeoff stated in those terms. msisdnHash is the one thing that survives every purge specifically so that opt-out matching and dedupe keep working against a purged row — see §2.5 and sms_api::pepper's module doc for the pepper-rotation caveat that decision makes sharper: a row whose plaintext msisdn has already been purged can never be rehashed.

ART licensing. Orange's VAS interconnection catalogue requires an ART title — a licence or a récépissé de déclaration préalable — plus an ART short-code allocation document, before it will interconnect you. ART has enforced this: in 2018 it announced it would dismantle unlicensed VAS providers' networks, with penalties of 100–500 million FCFA. Whether a pure API consumer buying capacity from a licensed aggregator needs its own title is unverified. Safe reading: direct MNO interconnection or a short code unambiguously requires one. Settle this before committing to SMPP.

Operational compliance regardless — landed in #72:

  • A delivery-status lookup by number never reaches purged history. msisdnHash survives the purge so opt-outs keep matching; lookupDeliveryStatus is the one read that takes a plain number, so it must not use that survival to link the number to a purged row. It reaches unpurged rows inside the last 90 days only (§5.2, "Implementation, delivery lookups"), and records every lookup, the refused ones too, under a keyed hash of the number.
  • Opt-out honoured at send time, before routing, for marketing and notification. OTP and transactional are exempt. backends/crates/sms-api/src/consent.rs::requires_recipient_consent_controls is the single, exhaustively-matched source of truth for this class split — before #72 the opt-out check ran unconditionally for every class, more restrictive than specified (an opted-out recipient couldn't receive an OTP either), not a compliance gap but not what this section asked for.
  • Consent records: who consented, when, through what channel, to what scope. ConsentRecord (§2.7) — msisdnHash/msisdn, scope MessageClass, channel ConsentChannel, consentedAt, evidenceRef. sendMessage refuses a marketing/notification send with no matching record on file (Procedures::ensure_consent_on_file). Read §2.7's own prose for what this does and does not prove about the classification itself, and for the deliberate, unresolved retention tension (no @@retain here yet — proof of consent and 90-day content minimisation want different answers).
  • Inbound STOP requires two-way SMS. Twilio supports neither two-way SMS nor short codes in Cameroon — inbound needs a direct MNO short code or an aggregator provisioning one locally. Until then, the Article 48 opt-out address is an email or URL in the body.
  • Self-imposed quiet hours, 08:00–20:00 WAT, for marketing only. backends/crates/sms-api/src/consent.rs::{MARKETING_QUIET_HOURS_START_WAT,MARKETING_QUIET_HOURS_END_WAT,is_within_marketing_quiet_hours} — named and documented as a policy knob, not a statute, per this bullet's own original wording ("No Cameroon-specific statutory rule was found"). Enforced at sendMessage (accept time), not dispatch (hold-and-release) — see that module's own doc for the tradeoff. notification is consent-gated but not quiet-hours-gated; only marketing is.

11. Repository layout ​

text
vsms/
├── schema/
│   ├── schema.cstack
│   ├── schema.prev.cstack          # CI wire-contract baseline
│   └── migrations/postgres/
│       ├── 0001_init/              # cratestack migrate diff output
│       └── 0002_bootstrap/         # generated from §2.10 by ci/gen-bootstrap-sql.py
├── crates/               # libraries only
│   ├── sms-core/         # domain types, transition tables, error taxonomy
│   ├── sms-encoding/     # GSM-7/UCS-2 — build this first
│   ├── sms-msisdn/       # E.164 every country, +237 default; operator inference
│   ├── sms-provider/     # SmsProvider trait, Capabilities, ProviderError
│   ├── sms-provider-http/          # shared reqwest transport/status classification for every HTTP adapter
│   ├── sms-provider-orange-cm/
│   ├── sms-provider-mtn/
│   ├── sms-provider-aggregator/
│   ├── sms-provider-smpp/          # scaffold only, milestone 7
│   ├── sms-api/          # include_server_schema! + AuthProvider + procedures + subscribers
│   │   ├── src/auth.rs             # Principal → CratestackContext; the only place field names live
│   │   ├── src/procedures.rs       # ProcedureRegistry
│   │   ├── src/router.rs           # generated router assembly
│   │   └── src/cache.rs            # R1 exception: LISTEN for opt-out invalidation
│   └── sms-worker/       # the worker as a library; the binary is backends/apps/sms-worker
│       ├── src/lease.rs            # R1 exception: pg_try_advisory_lock
│       ├── src/notify.rs           # R1 exception: NOTIFY
│       ├── src/claim.rs            # the shared candidate + CAS lease loop
│       └── src/roles/
│           ├── dispatch.rs  drain.rs  scheduler.rs
│           ├── hooks.rs     jobs.rs   smpp.rs
├── app/                  # binaries only
│   ├── sms-gateway/      # the API server: clap CLI, serve / routes
│   └── sms-worker/       # role-selectable worker, milestone 2
├── frontends/packages/
│   └── sms-client/       # generated TS, do not edit, checked in CI
├── frontends/apps/admin/                # Next.js 15 App Router + shadcn/ui
└── deploy/               # compose, Caddyfile, sops secrets

crates/ is libraries, app/ is binaries, and the dependency arrow only ever points from app/ into crates/. The split is worth the extra directory for two reasons: a library that cannot declare a main cannot quietly grow process-level concerns like signal handling or CLI parsing, and it makes "what do we actually ship" a directory listing rather than a grep for [[bin]]. Internal crates are wired through [workspace.dependencies] path entries, so a version or a feature flag is set once at the root rather than per consumer.

The .cstack file stays at schema/, not inside sms-api. include_server_schema! resolves against CARGO_MANIFEST_DIR and the conventional layout is to keep the schema in the crate that expands it — but three other things already read this one (the migration diff, ci/gen-bootstrap-sql.py, and sms-worker), and separating the schema from its own migrations to satisfy a macro's default path resolution is the wrong trade. sms-api reaches back out with ../../schemas/vsms.cstack.

sms-api depends on cratestack = { package = "cratestack-pg", version = "=0.5.0" } — the rename is mandatory, because generated code emits absolute ::cratestack::* paths. JsonCodec comes from the separate cratestack-codec-json crate.

Admin console screens ​

Dashboard (throughput, delivery rate by operator, queue depth, outbox depth, job backlog, balance, UCS-2 ratio) · Messages (filterable list, detail with a state timeline and raw provider payloads) · Composer with live encoding preview · Apps & service accounts (provision, show-secret-once, retire with overlap) · Sender IDs and per-provider registration status · Providers & routes with a route simulator · Webhooks (endpoints, attempts, replay, secret rotation) · Jobs (queue, failures, re-enqueue) · Workers (which node holds which singleton lock) · Opt-outs · Users & roles · Audit log · Settings.

Providers/Routes/Simulator landed in #54. List/detail for both models is real over real HTTP; editing (Provider.update, Route.create/update/delete) is real, tested code, but still 403s against a real gateway — not because #194 (human login) hasn't landed (it has), but because this console's own upstream seam (frontends/packages/gateway) hasn't been rewired to forward a human session's own token — see §6.3's own #54 implementation note for the full mechanism and why.

The Workers screen is small and disproportionately useful: pg_locks joined against your role-key table answers "is dispatch running, and where" without shelling into a box.

Two things to check before writing screens. Run cratestack studio init / studio run first — it ejects into a customisable crate and already does typed forms, SQL preview, drift detection and CSV/JSON export; it may cover the internal-operator half. And enumerate cratestack_schema::axum::ROUTE_TRANSPORTS at build time rather than hardcoding paths, because pluralize() is naive.

12. Milestones ​

#DeliverableGate
0Schema + migrations + state machine triggers + generated router + sms-encoding + sms-msisdnThe schema expands (not just parses — see below); every create input carries the fields its procedure sets; migration applies clean on an empty DB; an illegal transition raises SM001; previewMessage correct on a French corpus incl. ç and ’
1(historical: the OP was deleted in VS-4) sms-auth: OP with RS256, custom ClientStore, /token rate limiting, service-account provisioning, RBAC layers 1–2A persisted client_credentials client actually gets a token; a developer token cannot reach a provider:write route. Superseded by the signed transport (§4.0): a persisted ClientKey signs a call against a restarted process and a developer console key is refused (m1_acceptance_gate_live_postgres.rs)
2sms-worker skeleton: lease, claim loop, dispatch + jobs roles, Orange CM adapterReal SMS to a real Orange handset, delivered in under 15s; kill -9 the worker mid-submit and the lease reclaims the message
3Subscriptions → WebhookAttempt → hooks role, signing, retries, replay, drain roleSignature verifies in a sample Node receiver; killing sms-api mid-drain loses no event; two workers produce exactly one attempt per event
4Next.js admin, all screens, ETag/If-Match threaded through every editAn operator can diagnose a failed message without touching SQL
5MTN via aggregator, routing rules, failover, circuit breakersKill Orange in staging; MTN unaffected, Orange fails over cleanly
6Retention, audit anchoring, backup verification, alertingRestore drill passes; purge verified against a seeded old dataset
7(conditional) SMPP + direct interconnectOnly after the ART licensing question is settled

(Historical, VS-4: the OP and its gate are gone; the signed-transport equivalent is in the row above.) Milestone 1's gate is deliberately blunt. The GrantType bug means a stock integration looks correct in every in-memory test and fails the moment you persist a client — so "a persisted client gets a token" is the only gate that proves anything.

Milestone 2's kill -9 gate is the same idea applied to the worker: lease reclamation is the kind of code that's easy to write and easy to never actually exercise.

Milestone 0's gate says expands, not parses, and the distinction is not pedantic. cratestack-parser and cratestack-migrate both accept a scalar list field and the emitter writes a happy TEXT[]; include_server_schema! then panics with unsupported SQLx value type for this slice. A schema can be fully valid by two of the three tools and still not compile. The same gate catches the @default trap: a create input missing a field its procedure needs is a compile error, not a runtime surprise.

Milestone 0 still comes first. The encoding crate has the highest ratio of business impact to line count, it's pure and trivially testable, and getting it wrong is invisible until the invoice arrives.

13. Risks and open questions ​

Needs a decision from you:

  1. Where does this host? Law 2024/017's cross-border authorisation requirement makes this a legal question with an architectural answer. Cameroon-hosted is the safe default.
  2. Do you need your own ART title? Determines whether direct SMPP is on the table, and therefore whether milestone 7 exists.
  3. 10-year traffic retention vs 90-day minimisation. Resolved 2026-08-11 (issue #5). 90-day minimisation, no split ledger — see §10. Kept here, struck through, so this list stays the record of what was asked rather than silently shrinking.
  4. Does authkestra-op stay? Resolved 2026-10-27 (VS-4): no. The signed transport (§4.0) replaced the token plane, and the OpenID Provider, authkestra and sms-auth were deleted. Kept here, struck through, so this list stays the record of what was asked.

Technical risks:

RiskMitigation
Raw SQL creeping in — bypasses policy, audit, events and version bumping at onceR1 + assert-no-raw-sqlx.sh in CI; the exception allowlist lives in one script
A scalar list field panics include_server_schema! — and the parser and migration emitter both accept it, so nothing catches it until the macro runsNo model declares a list; multi-values are sentinel-delimited String (§2.2). Milestone 0 gate is "the schema expands", not "the schema parses"
A @default on a caller-settable field makes it unwritable (excluded from create input)Verified by compile in CI via the create-input round-trip test; §2.0
@server_only on a field the server must write makes it unwritable entirelyR3; secrets are protected by model-level @@allow, or kept out of the database
Advisory lock leaked on a pooled connection → singleton role dead until restartExplicit release() on shutdown, or a dedicated single-connection pool that's closed outright. Drop cannot do this — releasing needs an await
Two workers both running dispatch → Orange TPS doubled → blockedAdvisory lock is the only gate; alert on "singleton unheld" and on unexpected concurrent submits per provider
Claim-loop Forbidden swallowed as "lost the race", hiding a policy bug as throughput lossMatch PreconditionFailed for the race; log Forbidden loudly (§7.3)
State machine and code drift → production SM001 on a legal-looking transitionstate_machine_parity test in CI; alert on any non-zero SM001 rate
Illegal transition surfaces as 500 DATABASE_ERRORMap sqlstate = 'SM001' → CratestackError::Conflict → 409
A leaked client private key signs as that client until revokedrevokeAppClientKey (effective on the next request on every replica), per-client scopes, App.ipAllowlist at the edge, per-principal rate limits
A webhook subscriber blocking or panicking breaks the send APISubscribers only insert rows; bodies wrapped so panics become Err; HTTP lives in the hooks role
Concurrent drains double-deliver (no SKIP LOCKED in the framework)Unique index on (endpoint_id, aggregate_id, event_type) + ON CONFLICT DO NOTHING; drain is a singleton
Poison outbox row retries forever, table grows unboundedreap_outbox job + alert on attempts > 5; nothing in the framework caps this
Outbox stalls when writes go quiet (no background worker)drain role every 5s; alert on oldest undelivered age
Worker crash strands a claimed messageleaseUntil + reclaim predicate in the claim query; kill -9 gate at milestone 2
Cuid ids with _/- break GET /messages?id=… with a 400cs_cuid() emits [a-z0-9]{23}, no prefix separator
Orange 5 TPS + 100k FCFA/day SIM cap ceilings you at ~5k SMS/dayStart MTN and commercial-contract conversations at milestone 2
No FK constraints, no column defaults, no non-unique indexes, no triggers from the emitter§2.10, applied at milestone 0, with an "applies clean on empty DB" gate
A typo'd @@allow action or @@use silently no-opsTest asserting the full generated policy set and expanded field list
System context sets kind but not role = "system" → all message writes denyIntegration test on the first send
SMPP hex/decimal message_id mismatchproviderMessageRef + providerMessageRefAlt, both indexed
Grey route silently replaces sender IDMonthly handset validation per route; alert on delivery-rate divergence
CrateStack pre-1.0, 23 releases in 11 weeksPin exactly (see the root Cargo.toml's cratestack line, or run cargo xtask cratestack-pin for the current value — this table doesn't hardcode a version that's certain to be stale by the next bump); cratestack diff CI gate catches wire breaks
Only in-memory rate-limit store shipsImplement RateLimitStore against Redis/Postgres before the second API replica

Sources ​

Frameworks — schema, attribute, policy, CLI, event and Rust-API claims were verified against published crate sources for cratestack-{parser,core,macros,cli,pg,migrate,axum,policy,sqlx} 0.4.16, including compiling a real include_server_schema! expansion.

CrateStack · scalars · field attributes · mixins · auth provider guide · auth support matrix · idempotency · optimistic locking · audit log · soft delete · rate limiting · transaction isolation · pagination · migrations · TypeScript client generation · banking readiness · current state · Authkestra

Operators & aggregatorsOrange Developer — SMS Cameroon · pricing · SMS Cameroon on-net · on-net pricing · getting started · Orange Cameroun — Catalogue d'interconnexion SVA 2020 (PDF) · MTN SMS v3 API · Twilio — Cameroon SMS guidelines · Twilio — Cameroon pricing · Vonage — Cameroon features & restrictions · Africa's Talking — Cameroon bulk SMS & sender ID · Nexah · Techsoft-SMS Cameroon pricing

Regulation & numberingLaw No. 2024/017 (data protection) · analysis — African Law & Business · Law No. 2010/012 (cybersecurity, PDF) · ART enforcement against unlicensed VAS · ITU — Cameroon numbering plan 2014 (PDF) · google/libphonenumber · ART — number portability

Released under the MIT License.