vsms

Runbook: sending through AWS End User Messaging SMS ​

aws_sms is a second SmsProvider beside orange_cm and mtn_cm. It sends through AWS End User Messaging SMS (the pinpoint-sms-voice-v2 API, SendTextMessage) and gets its delivery receipts back through Amazon SNS. It exists so the system can carry real traffic while the Orange and MTN contracts are pending, and it stays useful afterwards as a second route for the same numbers. The crate’s own description is in backends/crates/sms-provider-aws-sms/src/lib.md; the design-doc summary is §6.2. This page is the operator’s half: what to create in AWS, what to set in vsms, how to check it, and how to move traffic between providers.

Important

Read the data-residency section first. Sending through AWS means every recipient number and every message body leaves Cameroon. That may need an authorisation vsms cannot grant you, and it is not a detail to sort out after go-live.

Data residency: a blocker, not a footnote ​

§13 of the design doc (item 1; also AGENTS.md, "Open questions") records that Cameroon’s Law No. 2024/017 requires prior authorisation for cross-border transfers of personal data, and that "legitimate interest" is not a lawful basis for one. A recipient’s phone number is personal data, and an OTP or notification body usually is as well. Calling SendTextMessage hands both to AWS in whichever Region you configure, which is outside Cameroon in every case: AWS has no Cameroon Region, and the nearest one, af-south-1, is in South Africa.

vsms does not make this question go away and cannot answer it:

  • providers[code=aws_sms].settings.region only chooses which foreign Region receives the data.

  • The 90-day purge (purge_retention) clears vsms’s own copy of the number and body. It does nothing about what AWS retains. Check AWS’s own retention for End User Messaging in the Region you pick.

  • AWS keeps a separate opt-out list. A number that texts STOP to an AWS origination identity is blocked by AWS, and vsms’s own OptOut table does not know. See Known limits.

Before you send production traffic, someone with the authority to decide must have cleared the transfer (or decided the deployment is out of that law’s scope, for example because the recipients are not in Cameroon). Record that decision somewhere the next operator can find it. Traffic to Orange or MTN through their own APIs has a different, simpler position; that is one reason to move traffic back to them as soon as they are available (see Routing, now and later).

What gets built, and who talks to whom ​

text
sms-worker --SendTextMessage (SigV4)--> AWS End User Messaging SMS --> carrier --> handset
                                              |
                              configuration set (event destination)
                                              |
                                              v
                                     Amazon SNS topic
                                              |
                       signed HTTPS POST, SignatureVersion 2
                                              v
                              https://<gateway>/dlr/aws_sms

  • The worker sends. It needs AWS credentials and nothing about the topic.

  • The gateway receives. It needs the topic ARN and no AWS credentials, so a compromised gateway cannot send an SMS on your account.

  • /dlr/aws_sms is public and unauthenticated by design (a provider webhook carries no bearer token). The gateway authenticates every call itself: the TopicArn must equal the configured one, SignatureVersion must be 2, the signing-certificate URL must be https://sns.<topic-region>.amazonaws.com/SimpleNotificationService-<32 hex>.pem with no query or fragment, and the RSA-SHA256 signature must verify. Certificate fetches that are not already cached are rate-limited, so a forged request can cost the gateway at most an occasional fetch. A refused call is answered 403; a check that could not run (certificate fetch failed or was rate-limited) is answered 503 so SNS retries. See backends/crates/sms-provider-aws-sms/src/sns.md.

  • A subscription is confirmed by you, not by the gateway (step 6). The gateway logs the exact command to run.

Why End User Messaging and not plain SNS Publish: SNS can send an SMS directly, but it reports what happened to it only through CloudWatch Logs. Every message sent that way would sit in submitted until expire_stale marked it expired, firing message.expired for an OTP that arrived. End User Messaging is the same SMS backend with an API that can publish per-message events to an SNS topic.

Prerequisites ​

  • An AWS account, and an operator with permission to create IAM users, SNS topics and End User Messaging resources.

  • The AWS CLI v2 (aws), configured for that account. Every command below takes --region; use the same one throughout.

  • A public HTTPS URL for the gateway that SNS can reach, with a certificate a public CA signed. SNS will not post to a plain-HTTP or self-signed endpoint, so a laptop behind NAT cannot receive receipts. The Caddy edge in deploy/ is that URL in a normal deployment (Deployment).

  • A running sms-gateway and sms-worker with a migrated database (Deployment, or Getting started for a scratch setup).

  • The data-residency decision above.

1. Choose a Region ​

End User Messaging SMS is available in af-south-1 (Africa, Cape Town) and in eu-west-1 (Europe, Ireland), among many others; the full list is the "AWS End User Messaging SMS and Voice V2 API" table on https://docs.aws.amazon.com/general/latest/gr/end-user-messaging.html.

Two differences that matter in practice:

  • af-south-1 is an opt-in Region: it is disabled by default and must be enabled in the account (Account page, then AWS Regions, or aws account enable-region --region-name af-south-1; the operation is asynchronous and can take from minutes to hours). eu-west-1 is enabled by default. Source: https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-regions.html.

  • Everything is per Region: the sandbox, the spend limit, the sender ID, the configuration set, the SNS topic and the IAM policy ARNs. The sandbox exit request below has to be filed for each Region you send from, and the SNS topic must be in the same Region as the configuration set.

Which Region is better for Cameroon traffic is not something this page can decide: it depends on the transfer decision above and on AWS’s own pricing for the destination from each Region, which you should read on the AWS pricing page rather than take from here.

The rest of this page writes $REGION, $ACCOUNT (the 12-digit account id) and uses the names vsms-sms-events (topic), vsms-sms (configuration set) and vsms-sms-sender (IAM user). Rename freely.

bash
export REGION=eu-west-1
export ACCOUNT=123456789012

2. Leave the SMS sandbox and set the spend limit ​

New accounts start in the SMS sandbox. In it, the monthly spend limit is USD 1.00 and you can send only to verified destination numbers (up to 10). Source: https://docs.aws.amazon.com/sms-voice/latest/userguide/sandbox.html.

To leave it, open an AWS Support case (Account and Billing, service Service Quotas, category AWS End User Messaging SMS (Pinpoint), quota SMS Production Access, new value 1) and fill in the use-case fields: the site sending, the message types (an OTP is One Time Password), the destination country, how recipients opt in, and the message template. AWS says it answers within 24 hours and may ask for more. File one case per Region.

Until that is approved you can still test: add your own phone as a verified destination number in the console (Verified destination numbers), which AWS verifies with a code.

What the sandbox looks like from vsms. A send to a number that is not a verified destination is refused by AWS with a ConflictException whose reason is DESTINATION_PHONE_NUMBER_NOT_VERIFIED. vsms treats that as a problem with this route, not with the message (Permanent): the message fails over to the next route. If aws_sms is the only route there is nothing to fail over to and the message ends failed with a "no alternate route available" reason. So a deployment that sends to real customers while the account is still in the sandbox shows failed messages, not an outage and not an open circuit. A recipient who opted out at AWS is a different ConflictException and is Rejected: the message fails with no failover.

The monthly spend limit is separate and has two parts: the account’s ceiling (set by AWS, raised through the same Support route) and the limit you enforce under it.

bash
# View the limits the account currently has
aws pinpoint-sms-voice-v2 describe-spend-limits --region $REGION

# Enforce a monthly ceiling for text messages (must not exceed the account maximum)
aws pinpoint-sms-voice-v2 set-text-message-spend-limit-override \
  --monthly-limit 200 --region $REGION

(set-text-message-spend-limit-override is documented at https://docs.aws.amazon.com/cli/latest/reference/pinpoint-sms-voice-v2/set-text-message-spend-limit-override.html; it says the value must not exceed the maximum AWS sets and does not state the unit on that page. The quotas table lists the default threshold as USD 1.00 per account.) Pick the figure on purpose: when the limit is reached AWS refuses further sends, and vsms reacts to that as an outage of the provider (see Troubleshooting).

3. Sender ID for Cameroon ​

AWS’s country table (https://docs.aws.amazon.com/sms-voice/latest/userguide/phone-numbers-sms-by-country.html) lists Cameroon (CM, dialling code 237) as: short codes Yes, long codes No, sender IDs Yes (with no "registration required" note, unlike, for example, Kenya or the UK), two-way SMS Yes. So a sender ID works for Cameroon without a regulatory registration.

  • A sender ID is 1 to 11 characters, letters, digits and hyphens, at least one letter, no spaces, not starting or ending with a hyphen (https://docs.aws.amazon.com/sms-voice/latest/userguide/sender-id.html).

  • The display rule for a country that supports but does not require registration: if the message specifies a sender ID, it is sent from that ID. If it specifies none and the account has no dedicated number for that country, AWS sends it from the word NOTICE. Always set an origination identity (below) so recipients do not see NOTICE.

  • The sender ID is case-sensitive at the carrier. Use exactly the casing you intend to display, and do not copy the console’s upper-cased listing.

  • The same page’s section Request a sender ID describes requesting one in the console (Sender ID, Request originator). The pages read for this runbook do not say whether a sender ID in a no-registration country must be requested in the account before it can be used. If the first test send fails with ResourceNotFoundException or a sender-identity ValidationException, request it there and retry.

Then choose what vsms passes as the origination identity:

  • settings.origination_identity set (recommended): that value is used for every message. It can be the sender ID, a phone number, a pool, or one of their ARNs.

  • Not set: the message’s own approved vsms sender ID is passed instead, which then has to exist in the AWS account too.

A pool (Pools in the console) groups origination identities so you can add or remove a problem identity without redeploying. AWS recommends this for business-critical traffic and notes that SMS numbers in each country are provisioned through a single carrier partner, which is a single point of failure (same country-table page). That is a second reason to keep another provider configured.

4. Create the SNS topic ​

bash
aws sns create-topic --name vsms-sms-events --region $REGION
# note the TopicArn, then:
export TOPIC_ARN=arn:aws:sns:$REGION:$ACCOUNT:vsms-sms-events

Set the signature version to 2. SNS topics default to SignatureVersion 1 (SHA1). The gateway refuses version 1 outright, and if you skip this step every notification and the subscription confirmation will be answered 403. SNS treats a 403 as a permanent failure and does not retry it, so those receipts are not delayed, they are lost; the gateway logs each one at error with the reason. Messages sent in that window end expired.

bash
aws sns set-topic-attributes --region $REGION \
  --topic-arn $TOPIC_ARN \
  --attribute-name SignatureVersion --attribute-value 2

Source: https://docs.aws.amazon.com/sns/latest/dg/sns-verify-signature-of-message-configure-message-signature.html (it lists versions 1 and 2, defaults to 1, and gives this command). Attribute SignatureVersion is not among the attributes aws sns create-topic --attributes documents, so set it afterwards as above.

Let End User Messaging publish to the topic. The topic’s access policy must allow the service principal sms-voice.amazonaws.com to call sns:Publish, scoped to your account and to the configuration set. This is AWS’s own statement (https://docs.aws.amazon.com/sms-voice/latest/userguide/configuration-sets-sns-creating-role.html), with the placeholders filled in:

json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "sms-voice.amazonaws.com" },
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:REGION:ACCOUNT:vsms-sms-events",
      "Condition": {
        "StringEquals": { "aws:SourceAccount": "ACCOUNT" },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:sms-voice:REGION:ACCOUNT:configuration-set/vsms-sms"
        }
      }
    }
  ]
}

Save it as topic-policy.json with REGION and ACCOUNT replaced, then:

bash
aws sns set-topic-attributes --region $REGION --topic-arn $TOPIC_ARN \
  --attribute-name Policy --attribute-value file://topic-policy.json

set-topic-attributes replaces the whole Policy attribute. If the topic already has statements you want to keep (a freshly created topic carries a default owner statement), fetch them with aws sns get-topic-attributes and merge rather than overwrite. The console’s New Amazon SNS topic option for an event destination creates a topic "with all of the required permissions" (per the AWS page for creating the destination) and is an alternative to writing the policy by hand; you still have to set SignatureVersion on it afterwards.

If you enable server-side encryption with a customer-managed KMS key on the topic, AWS says the key policy also needs sms-voice.amazonaws.com allowed to kms:GenerateDataKey and kms:Decrypt. vsms does not need encryption on the topic and the simplest working setup leaves it off.

5. Configuration set and the SNS event destination ​

bash
aws pinpoint-sms-voice-v2 create-configuration-set \
  --configuration-set-name vsms-sms --region $REGION

aws pinpoint-sms-voice-v2 create-event-destination \
  --configuration-set-name vsms-sms \
  --event-destination-name vsms-sns \
  --matching-event-types TEXT_ALL \
  --sns-destination TopicArn=$TOPIC_ARN \
  --region $REGION

(create-event-destination flags and values: https://docs.aws.amazon.com/cli/latest/reference/pinpoint-sms-voice-v2/create-event-destination.html; the console route is Configuration sets, then Add destination event, type Amazon SNS, event types All SMS events, per https://docs.aws.amazon.com/sms-voice/latest/userguide/configuration-sets-sns-add.html.)

Which event types. Use TEXT_ALL. vsms maps every TEXT_* event it knows (backends/crates/sms-provider-aws-sms/src/events.md), and an event type you leave out is a receipt it never sees. Picking a narrow list such as only TEXT_DELIVERED leaves every message whose carrier never sends a handset receipt submitted until it expires. Two facts about AWS’s list:

A configuration set accepts up to 5 event destinations. If you add another (for example CloudWatch Logs, for your own audit), leave this one alone.

The configuration set name must match settings.configuration_set below. Without a configuration set that has an event destination, AWS emits no events at all and every message ends expired.

6. Subscribe the gateway URL to the topic ​

The gateway has to know the topic ARN before you subscribe, because it checks TopicArn on the confirmation message too. So set settings.events_topic_arn on the gateway first (step 8), after seeding the aws_sms Provider row (step 9, first command): the gateway refuses to start if that variable is set and no Provider row keyed aws_sms exists. Then restart the gateway and subscribe:

bash
aws sns subscribe --region $REGION \
  --topic-arn $TOPIC_ARN \
  --protocol https \
  --notification-endpoint https://GATEWAY_HOST/dlr/aws_sms \
  --return-subscription-arn

Do not set RawMessageDelivery. It defaults to off and must stay off: the signature SNS makes lives in the JSON envelope, and raw delivery strips the envelope (the AWS subscribe page describes it as removing the JSON formatting).

Confirm the subscription yourself ​

SNS posts a SubscriptionConfirmation to that URL. The gateway verifies it exactly as it verifies a notification, but it does not confirm the subscription. It logs, at warn, the topic ARN, the token and the command to run, and you run it:

bash
aws sns confirm-subscription --region $REGION \
  --topic-arn <topic arn from the log> \
  --token <token from the log> \
  --authenticate-on-unsubscribe true

Why it is manual: the gateway could confirm by fetching the message’s SubscribeURL, but a subscription confirmed that way has AuthenticateOnUnsubscribe set to false. Every notification SNS sends carries an UnsubscribeURL, and vsms stores each receipt’s raw payload (DeliveryReceipt.rawPayload), so anyone able to read one could unsubscribe the endpoint with a plain request and silence every future receipt. With --authenticate-on-unsubscribe true, AWS’s confirm-subscription page says an authenticated request is required, and then only the topic owner and the subscription owner can unsubscribe (https://docs.aws.amazon.com/cli/latest/reference/sns/confirm-subscription.html).

The pages read for this runbook do not state how long a confirmation token is valid. An older version of this page said two days; treat that as unverified. If the token has stopped working, delete the subscription and subscribe again.

Check that it worked:

bash
aws sns list-subscriptions-by-topic --region $REGION --topic-arn $TOPIC_ARN
# SubscriptionArn is a real arn:aws:sns:... value once confirmed,
# and the literal "PendingConfirmation" until then.

Keep the confirmed SubscriptionArn; the next section needs it. Every later notification either ingests quietly or logs a warn that names the refusal (see Troubleshooting). If the subscription is ever removed (by anyone, including AWS), SNS posts an UnsubscribeConfirmation; the gateway logs it at error and does nothing else. Receipts stop until you subscribe and confirm again.

Survive a gateway outage: delivery policy and dead-letter queue ​

SNS’s default for an HTTP/S endpoint is three retries about 20 seconds apart, and anything other than a 5xx or 429 reply is a permanent failure that is never retried. A gateway restart or deploy longer than a minute can therefore lose receipts, and a lost receipt leaves its message submitted until it expires. Two settings on the subscription reduce that. Both are subscription attributes, set with aws sns set-subscription-attributes; DeliveryPolicy and RedrivePolicy are among the attribute names that command accepts (https://docs.aws.amazon.com/cli/latest/reference/sns/set-subscription-attributes.html).

A longer retry policy. Save as delivery-policy.json. This is the retry block from AWS’s own HTTP/S example (50 retries: 3 immediate, 2 one second apart, 10 exponential from 1 to 60 seconds, 35 at 60 seconds); the retry phases there add up to well under the one-hour cap on total retry time for an HTTP/S policy (https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html):

json
{
  "healthyRetryPolicy": {
    "minDelayTarget": 1,
    "maxDelayTarget": 60,
    "numRetries": 50,
    "numNoDelayRetries": 3,
    "numMinDelayRetries": 2,
    "numMaxDelayRetries": 35,
    "backoffFunction": "exponential"
  }
}
bash
aws sns set-subscription-attributes --region $REGION \
  --subscription-arn <the confirmed SubscriptionArn> \
  --attribute-name DeliveryPolicy \
  --attribute-value file://delivery-policy.json

AWS states that the total retry time cannot exceed 3,600 seconds and that this is a hard limit, so a retry policy alone cannot cover an outage longer than an hour.

A dead-letter queue. After retries are exhausted, SNS discards the message unless the subscription has a dead-letter queue, which is an ordinary SQS queue in the same account and Region (https://docs.aws.amazon.com/sns/latest/dg/sns-dead-letter-queues.html). Create a standard queue, let SNS write to it, then attach it:

json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "sns.amazonaws.com" },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:REGION:ACCOUNT:vsms-sms-events-dlq",
      "Condition": {
        "ArnEquals": { "aws:SourceArn": "arn:aws:sns:REGION:ACCOUNT:vsms-sms-events" }
      }
    }
  ]
}
bash
aws sqs create-queue --queue-name vsms-sms-events-dlq --region $REGION
# set the queue policy above on it (aws sqs set-queue-attributes, attribute Policy), then:
aws sns set-subscription-attributes --region $REGION \
  --subscription-arn <the confirmed SubscriptionArn> \
  --attribute-name RedrivePolicy \
  --attribute-value '{"deadLetterTargetArn":"arn:aws:sqs:REGION:ACCOUNT:vsms-sms-events-dlq"}'

(Queue policy and RedrivePolicy shape: https://docs.aws.amazon.com/sns/latest/dg/sns-configure-dead-letter-queue.html.) AWS recommends a 14-day retention period on the dead-letter queue, and an alarm on the SQS metric ApproximateNumberOfMessagesVisible so you hear about anything that lands there.

What is not settled by the pages read: the form of a dead-lettered message body for an HTTP/S subscription, and whether a reply of 403 from the gateway sends the message to the queue. Inspect a real dead-lettered message before you build any replay. Nothing in vsms replays them; a receipt that was lost is not recovered, and its message resolves by expiry.

7. IAM user for the worker ​

The worker signs its own requests with a long-lived access key (no instance profile, IRSA or STS session credentials; see Known limits). Create a dedicated IAM user with nothing else attached.

AWS documents sms-voice:SendTextMessage with resource ARNs of three types (https://docs.aws.amazon.com/sms-voice/latest/userguide/permissions-actions.html):

text
arn:aws:sms-voice:REGION:ACCOUNT:phone-number/PHONE_NUMBER_ID
arn:aws:sms-voice:REGION:ACCOUNT:pool/POOL_ID
arn:aws:sms-voice:REGION:ACCOUNT:sender-id/SENDER_ID/ISO_COUNTRY_CODE

and the configuration set’s ARN, which the SNS policy page uses, is arn:aws:sms-voice:REGION:ACCOUNT:configuration-set/NAME. A least-privilege policy for a sender ID MYBRAND in Cameroon and the configuration set above:

json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sms-voice:SendTextMessage",
      "Resource": [
        "arn:aws:sms-voice:REGION:ACCOUNT:sender-id/MYBRAND/CM",
        "arn:aws:sms-voice:REGION:ACCOUNT:configuration-set/vsms-sms"
      ]
    }
  ]
}

If you send from a pool or a phone number instead, list that ARN in place of the sender-id one. If you pass no origination identity at all, you must also allow whatever identity AWS picks, which is the reason to always set one.

bash
aws iam create-user --user-name vsms-sms-sender
aws iam put-user-policy --user-name vsms-sms-sender \
  --policy-name vsms-send-text --policy-document file://send-policy.json
aws iam create-access-key --user-name vsms-sms-sender
# prints AccessKeyId and SecretAccessKey -- the only time the secret is shown

The pages read for this runbook do not state whether the configuration-set ARN is required in Resource for SendTextMessage (the permissions page lists only the three origination resource types for that action). It is harmless to list it. If the first send returns AccessDeniedException, read the message, which names the ARN that was refused, and add it. Verify the policy with a real send (step 9) before you rely on it.

Rotate the key on a schedule: create a second key, put it in the secret store, restart the worker, delete the old one. There is no hot reload.

8. Configure vsms ​

Two processes, two different halves of one aws_sms entry in the YAML configuration (VSC-1; docs/architecture.md, section 9.4). Both binaries read the same document, so the entry is written once; compose keeps it in config/application-prod.yml (the AWS block is a commented example there), and Helm keeps it in config.overlay. The secret is the one thing that is not written in the file: credentials.secret_access_key is a ${AWS_SMS_SECRET_ACCESS_KEY} placeholder, delivered through the environment, and livemode refuses it as a literal.

yaml
providers:
  - code: aws_sms
    credentials:
      access_key_id: <the IAM user's key id>
      secret_access_key: ${AWS_SMS_SECRET_ACCESS_KEY}
    settings:
      region: eu-west-1
      configuration_set: vsms-sms
      tps_ceiling: 5
      cost_per_segment_xaf: "30"
      time_to_live_seconds: 900
      events_topic_arn: arn:aws:sns:eu-west-1:123456789012:vsms-sms-events

sms-worker (sends). The sending half is all-or-none: naming any of the first six makes the rest required, and an entry that is missing one is refused at start, naming it, so a half-filled block is loud:

KeyMeaning

settings.region

The Region from step 1.

credentials.access_key_id, credentials.secret_access_key

The IAM user’s key from step 7. The secret is a ${AWS_SMS_SECRET_ACCESS_KEY} placeholder and never prints.

settings.configuration_set

vsms-sms — the set whose event destination is the SNS topic.

settings.tps_ceiling

Message parts per second the account may send. An account quota, so it has no default. Take it from AWS (it varies by destination country and origination identity) and, if unsure, start low.

settings.cost_per_segment_xaf

What one segment costs, converted to XAF by you. Write it as a quoted string ("30"), because it is money. AWS prices in USD; this value feeds estimatedCostXaf, so a wrong figure misprices that field. No default.

Optional:

KeyDefaultMeaning

settings.origination_identity

(unset)

Sender ID, number, pool or ARN used for every message. Strongly recommended (step 3).

settings.message_type

transactional

transactional or promotional; AWS applies different carrier handling. OTP and notification traffic wants transactional.

settings.supports_alphanumeric_sender

false

Whether the account sends under alphanumeric sender IDs to your destinations. Informational (it feeds the adapter’s advertised capabilities); it does not change what is sent.

settings.time_to_live_seconds

(unset)

How long AWS keeps trying to deliver one message, 5 to 259200 seconds. Unset means AWS’s default of 72 hours, so an OTP can reach a handset days after vsms expired it (an OTP’s default validity in vsms is 15 minutes). For OTP traffic set 900. It is a worker-wide setting, applied to every message sent through this provider whatever its class, so if the same provider also carries traffic that is valid for longer (the other classes default to 24 hours) a short value cuts those short; choose the value for the traffic you actually send.

host.url

(unset)

Overrides the AWS endpoint. For tests against a fake only; under livemode it must be https and not a stub host.

sms-gateway (receives):

KeyMeaning

settings.events_topic_arn

The topic ARN from step 4. Setting it enables POST /dlr/aws_sms. No AWS credentials go on the gateway. The gateway exits at startup if this is set and no Provider row keyed aws_sms exists, so seed that row (step 9) before you set it.

An entry with the topic and none of the sending keys is a receive-only entry, which the worker reads as "nothing to send through"; an entry with neither is refused. Quote values that look like numbers but are text; an unquoted +2370000 is a YAML integer, and the typed model refuses it.

The old AWS_SMS_* variables and --aws-sms-* flags are gone and are refused, not ignored: a process that still sets one stops at start and names the key above that replaced it.

Order: seed the Provider row (step 9, first command), then restart the gateway (so it knows the topic before you subscribe), then the worker.

9. Seed the provider and the route, then send a test ​

The worker builds the adapter from the YAML entry, but the routing layer and the database need a Provider row with the same key. seed-dispatch creates it, activates it, and creates a catch-all Route pointing at it. Run it before you start the gateway with settings.events_topic_arn set (step 8): it needs only the database.

bash
docker compose --env-file .env run --rm sms-gateway seed-dispatch \
  --key aws_sms \
  --kind aggregator_http \
  --display-name "AWS End User Messaging SMS" \
  --credential-ref env:AWS_SMS_SECRET_ACCESS_KEY \
  --max-tps 5 \
  --cost-per-segment-xaf 30

(docker compose run --rm sms-gateway …​ is how deployment.adoc runs the other one-shot commands; use your own wrapper if you deploy differently. --max-tps and --cost-per-segment-xaf here should match the entry’s tps_ceiling and cost_per_segment_xaf; the row is a record, the adapter’s own values are the ones the worker uses. Flags and defaults are in backends/apps/sms-gateway/src/commands/seed_dispatch.rs; --max-daily-submissions defaults to 100000.)

AWS-only go-live: do not seed Orange by accident. sms-gateway bootstrap seeds an orange_cm Provider and a catch-all Route as one of its steps, and so does a bare seed-dispatch. In a deployment where the worker has no Orange credentials that Route is live, and it shares priority 0 and weight 1 with the AWS catch-all, so about half of all messages are routed to a provider the worker cannot send through. Those messages are not failed: the worker finds no adapter for them and retries with backoff indefinitely (see the warning in Routing, now and later). The fix is to never create that Route:

bash
# first run on a fresh database: everything bootstrap does, except the orange_cm seed
# (add --console-redirect-uri / --owner-email / --owner-display-name if you run the console)
docker compose --env-file .env run --rm sms-gateway bootstrap --skip-seed-dispatch
# then the AWS provider and its catch-all route, with the seed-dispatch command above

How each deployment path avoids it (read from the files, not run):

  • Compose (deploy/docker-compose.yml): use the two commands above, and not bootstrap without the flag or a bare seed-dispatch. The file’s own comment next to the one-shot commands says the same.

  • Helm (deploy/charts/vsms): the chart does not run bootstrap; it uses granular pre-install and pre-upgrade hooks. The seedDispatch hook (Orange) is switched off by the chart when orange.clientId is empty, and the separate seedDispatchAws hook is switched on when awsSms.region or awsSms.eventsTopicArn is set. An AWS-only values file therefore leaves orange.clientId and mtn.clientId empty and sets the awsSms block. If you already seeded Orange by mistake, disable the orange_cm route and provider in the console (Routes, Providers) rather than leaving it.

Two things about the seed-dispatch command are deliberate and worth knowing:

  • There is no aws_sms ProviderKind. --kind takes one of orange_cm_http, mtn_http, aggregator_http, smpp, and the row uses aggregator_http. Adding a variant would be a schema change and a migration for a field nothing reads (the adapter is chosen by key, not kind), so it was avoided on purpose. If you want a real distinction, that is a future schema change.

  • The command is idempotent and always ensures a catch-all route. It creates a Route named catch-all (seed-dispatch) at priority 0, weight 1, every match* empty, pointing at this provider, and re-enables it if it was disabled. If you already run another provider with its own catch-all, you now have two wildcard routes at equal priority and weight, and traffic splits between them. That is the starting point for the split described below, but make it deliberate, and only once the other provider’s adapter is configured in the worker: see Routing, now and later.

Now send a message to a number you control (while in the sandbox, a verified destination number) and watch it:

  1. Send through the API or the console composer (just demo-style flows are in Getting started). Note the messageId that comes back.

  2. GET /messages/{id}, or open the message in the console, until it leaves accepted. The expected sequence is accepted, queued, routed, submitted, then delivered.

  3. Confirm a receipt arrived: the console’s message detail shows the DeliveryReceipt rows, and in SQL:

    sql
    SELECT received_at, outcome, raw_status
FROM delivery_receipts
WHERE message_id = '<messageId>'
ORDER BY received_at;

    You should see TEXT_QUEUED / TEXT_PENDING rows as in_flight and a final TEXT_SUCCESSFUL or TEXT_DELIVERED. raw_status keeps AWS’s own event name, so you can always tell a carrier acceptance from a handset confirmation (see Known limits).

  4. On the gateway, a warn line with provider_key="aws_sms" means a callback was refused or failed, and an error means receipts are being lost (topic on SignatureVersion 1, or the subscription was removed); none means every call passed.

Routing, now and later ​

Routing is data: rows of Route (priority, weight, matchOperator, matchClass, matchAppId, matchPrefix, providerId, enabled), evaluated by the pure engine in backends/crates/sms-routing (§6.3).

How a decision is made, in order:

  1. Disabled routes and routes whose provider is unavailable are dropped. A provider is unavailable if it is not active or its circuit is open.

  2. Routes whose predicates do not match are dropped. A NULL predicate is a wildcard ("matches anything"), not "matches only a NULL value".

  3. Of what remains, the highest priority wins as a band. Priority is the only cross-band ordering; there is no "more specific route wins". A route with three predicates set does not beat a wildcard at the same priority.

  4. Inside the winning band a weighted draw picks one route: each route gets a share of [0, 1) proportional to its weight (all weights 0 falls back to an even split).

  5. If the chosen provider’s submit fails in a way that means nothing was accepted, the message fails over: the engine runs again over the whole route set with the failed route excluded, so the other routes in the same band, or the next band down, get their turn. At most two hops.

What fails over and what does not (from backends/crates/sms-provider/src/error.rs and dispatch.rs):

Provider outcomeEffect

Permanent (e.g. sender identity or country refused, or the account still in the SMS sandbox and the number unverified)

Fail over to the next route.

Unavailable (connect failure, 5xx, spend limit, credential or access error)

Count towards that provider’s circuit breaker; fail over. Five consecutive opens it for 60 seconds and routing then skips that provider for all new messages.

Transient (throttled)

Retry on the same provider after a short delay. No failover.

Rejected (the message itself is bad, e.g. recipient opted out at AWS)

Fail the message. No failover.

Indeterminate (timed out after connecting; response unreadable)

uncertain. Never fails over or resubmits, because AWS may have sent it. Resolved by expiry: its MessageId was never learned, so no later receipt can match it (see Known limits).

Now: AWS is the only provider ​

One wildcard route. This is exactly what seed-dispatch --key aws_sms leaves you with:

namepriorityweightmatchOperatormatchClassprovider

catch-all (seed-dispatch)

0

1

(any)

(any)

aws_sms

Every message goes to AWS. If AWS’s circuit opens (for example the spend limit is hit) and there is no other route, accepted messages find no eligible route and go to rejected with a stateReason naming why, rather than queue up behind an outage. That is by design; see Troubleshooting.

Later: MTN-or-AWS, Orange-or-AWS, AWS alone elsewhere ​

Once the MTN and Orange adapters are configured and their Provider rows exist (mtn_cm, orange_cm), the routes become:

namepriorityweightmatchOperatorproviderpurpose

mtn direct

100

70

mtn

mtn_cm

most MTN numbers go direct

mtn via aws

100

30

mtn

aws_sms

a standing share on AWS, and the failover target for the row above

orange direct

100

70

orange

orange_cm

most Orange numbers go direct

orange via aws

100

30

orange

aws_sms

the same for Orange

everything else

0

1

(any)

aws_sms

Camtel, Nexttel, unknown and foreign numbers, and the fallback

Reading it:

  • An MTN number matches the two mtn rows (priority 100) and the wildcard (priority 0). Priority 100 is the winning band, so the wildcard is never drawn for it; weights 70 and 30 split MTN traffic roughly 70/30 between MTN direct and AWS.

  • If the chosen route fails with Permanent or Unavailable, the engine re-runs with that route excluded. If MTN direct failed, "mtn via aws" is the only route left in that band and wins; if that fails too, the priority-0 wildcard (also AWS) is next, then the two-hop cap ends the attempts.

  • A number classified unknown, camtel or nexttel matches no mtn or orange row and only the wildcard, so it goes to AWS alone.

  • To send all MTN traffic direct and use AWS only as a failover, there is no "failover only" flag. Give "mtn via aws" weight 0 in the same band as "mtn direct": a weight-0 route is never drawn while a heavier route is eligible, but it is still a candidate once the direct route is excluded. Simulate it before you trust it (below).

  • To drain AWS entirely, set the aws_sms routes' enabled to false; to change the split, edit weights. Both take effect on the next claimed message.

  • matchClass and matchAppId narrow a route further, for example to send only otp traffic of one app through a particular provider.

Create and edit them in the admin console (Routes), or over REST (POST /routes, PATCH /routes/{id} with If-Match). Both require an owner or admin session; see Getting started for how a console session is obtained. Before changing live routes, use the console’s Simulator screen (or the simulateRoute procedure) with an MSISDN, class and app: it runs the same engine with a chosen or random draw and shows, per route, why it matched, was dropped, or won, and the exact tie break ranges, without sending anything.

Warning

Never create a Route to a provider whose credentials the worker does not have. dispatch resolves the adapter by Provider.key at submit time. If the winning route’s provider has no registered adapter in this worker process (you created the mtn_cm route but the worker has no MTN_* variables), the message is not failed: it is retried with backoff, indefinitely, until someone fixes the mismatch. Configure the adapter in the worker first, restart it, then add the route.

Warning

Operator classification is a hint. matchOperator matches the operator vsms guessed from the number’s prefix (OperatorPrefixRule), and number portability means a prefix stops identifying a network once a subscriber ports; some prefix rows are marked contested or unverified. A mis-classified number goes to the wrong provider’s band. That is safe when both bands include AWS (as above), because every band can still reach a working provider, but do not build a route set that depends on the guess being right for correctness. Foreign numbers have no operator at all (multi-country design, decision 3), which is why the final wildcard matters.

Validate each route with real handsets on each network before trusting it, and re-validate monthly (Monthly handset validation). That applies to aws_sms routes too: AWS routes through a carrier partner per country, and a sender ID that is rewritten downstream is invisible to every delivery receipt.

Troubleshooting ​

SymptomLikely causeWhat to do

POST /dlr/aws_sms answers 404

settings.events_topic_arn not set on the gateway, or the gateway was not restarted

Set it, restart the gateway.

403 on every call, gateway log "refusing an unauthenticated provider callback"

The log’s reason says which check failed. Usually TopicArn differs from settings.events_topic_arn (different Region, account or name), or the topic is still on SignatureVersion 1 (logged at error), or RawMessageDelivery is on, or the signing-certificate URL is not the expected https://sns.<topic-region>.amazonaws.com/SimpleNotificationService-<32 hex>.pem

Compare the ARN in the reason with the env value byte for byte. Re-run step 4’s SignatureVersion 2 command (aws sns get-topic-attributes shows it; absent means 1). Remove raw delivery from the subscription. Receipts refused with 403 are not retried by SNS and are lost; their messages end expired.

503 on calls, log "could not authenticate a provider callback; asking for redelivery"

The gateway could not complete the check: it could not fetch the signing certificate from sns.<topic-region>.amazonaws.com (egress blocked, DNS, TLS), or the fetch was refused because uncached certificate fetches are rate-limited

Fix outbound HTTPS from the gateway to that host. The rate limit only matters under a burst of uncached certificate URLs (normally a flood of forged requests, since genuine traffic reuses one cached certificate); it clears by itself. SNS retries 503, so receipts are not lost while it is fixed, within the retry policy in step 6.

Gateway exits at startup, "no Provider row has key aws_sms"

settings.events_topic_arn is set but the aws_sms Provider row does not exist

Run the seed-dispatch --key aws_sms …​ command from step 9, then start the gateway. For an AWS-only deployment, use bootstrap --skip-seed-dispatch, not plain bootstrap.

Subscription stays PendingConfirmation

The confirmation was refused (see the 403 row), never arrived (URL not publicly reachable, wrong path), or arrived and nobody ran aws sns confirm-subscription (the gateway never confirms)

Look for the gateway warn with the topic ARN, token and command. Run it. If there is none, or the token has stopped working, fix the cause, delete the subscription, subscribe again, and confirm with --authenticate-on-unsubscribe true.

Receipts stopped arriving; the gateway logged an UnsubscribeConfirmation at error

The subscription was removed, by someone with the topic, or by anyone holding an UnsubscribeURL if it was confirmed without --authenticate-on-unsubscribe true

aws sns list-subscriptions-by-topic to confirm it is gone. Subscribe and confirm again (step 6), this time with --authenticate-on-unsubscribe true, and re-apply the delivery policy and dead-letter queue to the new SubscriptionArn. Messages sent while it was missing end expired.

Messages reach submitted and stay there, then become expired

No receipts are arriving: configuration set has no SNS event destination (AWS then emits no events at all), the destination’s event types are too narrow, the topic policy does not allow sms-voice.amazonaws.com, the subscription is unconfirmed or was removed, receipts are being refused with 403 (see the first row), or settings.configuration_set names a different set

Walk the chain in order: the set exists, its destination exists (aws pinpoint-sms-voice-v2 describe-configuration-sets), the topic policy allows publish from that set, the subscription is confirmed, the gateway receives. A message sent before the fix is not rescued; it expires.

Every send fails and the provider’s circuit opens (new messages route elsewhere, or are rejected when there is no other route)

AWS refused with a quota or access error, which vsms treats as the provider being down: ServiceQuotaExceededException (monthly spend limit reached), AccessDeniedException (IAM policy, or the account is paused), ResourceNotFoundException (wrong configuration set or origination identity)

Read the worker log for AWS’s message. Raise the spend limit and the account maximum (step 2), fix the policy, or correct the names. The breaker reopens after 60 seconds on its own; there is nothing to reset.

A message is rejected immediately with a "no route" reason

Zero eligible routes: the only route’s provider is not active or its circuit is open

Check Provider.state and circuitOpenUntil; fix the cause above.

Messages failed while the account is new, "no alternate route available"

The account is still in the SMS sandbox and the destination is not a verified number (ConflictException, DESTINATION_PHONE_NUMBER_NOT_VERIFIED). vsms fails over to another route, and with none the message fails

Leave the sandbox (step 2), or send only to verified destination numbers meanwhile.

An OTP arrived on a handset long after it should have expired

settings.time_to_live_seconds is unset, so AWS keeps trying for its default 72 hours

Set it (step 8); 900 for OTP traffic. It only affects messages sent afterwards.

A message is failed with a sender-identity or country error

ValidationException about the origination identity, country or price that vsms treats as permanent for that route

Check the origination identity exists and is allowed for Cameroon in this Region (step 3). With another route available the message fails over to it first.

A recipient never gets messages and vsms shows failed/Rejected

The number opted out on AWS’s own list (AWS returns a conflict)

Check AWS’s opt-out list (aws pinpoint-sms-voice-v2 describe-opted-out-numbers, needs the list name). vsms’s OptOut table is separate.

uncertain messages after timeouts

A send timed out after connecting; AWS may have sent it

Expected, and a deliberate trade (a possibly lost message over a duplicate). It is not retried. Check AWS’s own logs if you need to know.

Without an AWS account: sms-fake-aws ​

backends/crates/sms-fake-aws stands in for AWS the way sms-fake-orange stands in for Orange. It answers SendTextMessage (checking the worker’s Signature Version 4 signature against its own implementation, held to AWS’s published get-vanilla vector), and plays the SNS topic: for every accepted message it publishes signed Notification`s to `/dlr/aws_sms, retrying until acknowledged, with an RSA key and certificate it generates at start-up.

just demo runs it. compose.dev.yaml seeds an aws_sms provider and catch-all route beside Orange’s, at the same priority and weight, so messages split between the two fakes and each message’s detail page names its provider. The worker gets the fake’s fixed demo account (FAKE-AWS-SMS-DEMO-KEY, configuration set vsms-fake), and the gateway gets settings.events_topic_arn plus settings.fake_sns_cert_base (both in config/application-dev.yml).

That second key is honoured only by an sms-gateway built with the off-by-default fake-aws cargo feature. It changes where the signing certificate is fetched from and nothing else: the topic, the signature version, the certificate URL’s shape (https://sns.<region>.amazonaws.com/SimpleNotificationService-<hex>.pem, which is what the fake’s notifications carry) and the signature are checked exactly as in production. Every published image is built without the feature, so it cannot be told to trust the fake: it refuses the key at start, as it once refused the flag, which did not parse.

sms-fake-aws --fault-modeWhat every submit gets

happy (default)

Accepted; TEXT_PENDING, then TEXT_DELIVERED after --delivery-delay-ms.

carrier-only

Accepted; a final TEXT_SUCCESSFUL, as carriers without handset receipts report.

silent

Accepted, never reported: the message ends expired.

sandbox

Refused with DESTINATION_PHONE_NUMBER_NOT_VERIFIED: traffic fails over to Orange.

unreachable

Accepted; a final TEXT_CARRIER_UNREACHABLE.

bash
docker compose -f compose.dev.yaml stop sms-fake-aws
docker compose -f compose.dev.yaml run --rm --service-ports sms-fake-aws \
  --bind-addr 0.0.0.0:8095 \
  --dlr-endpoint http://sms-gateway:8080/dlr/aws_sms \
  --fault-mode sandbox

The same fake backs backends/crates/sms-fake-aws/tests/round_trip.rs (the real adapter against the fake over HTTP, no database: signing, every error class, tampering, redelivery, a subscription confirmation) and backends/crates/sms-worker/tests/aws_sms_fake_live_postgres.rs (dispatch to a signed receipt and delivered; the sandbox refusal failing over to Orange; a response without a MessageId held uncertain and never resent; a tampered receipt refused on the route).

Both sides were written from the same reading of AWS’s documentation, so all of this proves the code agrees with that reading, over real HTTP and through the real signature code. It does not prove AWS behaves that way; step 9 with a real account does.

Known limits ​

From backends/crates/sms-provider-aws-sms/src/lib.md and the notes above:

  • Long-lived IAM user keys only. No instance profile, IRSA or STS session credentials, because refreshing them is the AWS SDK’s credential chain, which this crate avoids (the SDK’s HTTP stack is what would bring aws-lc-rs back). Scope the user to the one policy above and rotate the key.

  • A timed-out send is never rescued by a later receipt. Its MessageId was never learned, so any event that arrives matches nothing and expire_stale resolves the message. Same trade-off as Orange (AGENTS.md, #119).

  • A final TEXT_SUCCESSFUL counts as Delivered. Many carriers never send a handset receipt to AWS; for those AWS sends TEXT_SUCCESSFUL with isFinal: true and nothing after it. Treating that as in-flight would expire every such message, so it is recorded as delivered. It is an overstatement: it means the network took it, not that the handset confirmed. DeliveryReceipt.rawStatus keeps TEXT_SUCCESSFUL.

  • A non-final event never moves a message toward a retry. Any event with isFinal false other than TEXT_DELIVERED is recorded as in-flight, even a failure-looking one, because the carrier may still deliver the original and acting on it would send a duplicate. The cost is that a message AWS will later report as failed waits for that final event (or for expiry) instead of being retried early.

  • No delivering-network field. Events carry mcc/mnc, but they are not mapped onto vsms’s operator codes, so grey-route divergence analysis gets nothing per-operator from AWS receipts.

  • The SNS certificate’s own chain is not walked. The gateway trusts the signing certificate because it was fetched over HTTPS from the SNS host of the topic’s Region (redirects disabled), the same trust model the AWS SDK validators use. AWS’s published guidance also lists confirming the chain of trust; this check is not implemented separately.

  • Timestamp freshness is not checked. Replaying a genuine receipt is harmless: ingestion is idempotent and an out-of-order transition is refused by the state machine.

  • A receipt SNS gives up on is lost. SNS drops a message after its retry policy is exhausted (or on the first reply other than 5xx or 429) unless a dead-letter queue holds it, and nothing in vsms reads the queue. Its message ends expired. The delivery policy and dead-letter queue in step 6 shrink the window; they do not close it.

  • The subscription’s protection depends on a command you run. Confirmed without --authenticate-on-unsubscribe true, anyone with a notification’s UnsubscribeURL can unsubscribe the endpoint. vsms cannot check how a subscription was confirmed.

  • AWS keeps its own opt-out list, separate from vsms’s OptOut.

  • One provider, one carrier partner per country. AWS says SMS numbers are provisioned through a single carrier partner in each Region and country. Keep another provider configured for traffic that matters.

  • Nothing here has run against a live AWS account. The request and event shapes come from AWS’s published documentation and botocore’s service model; the SigV4 signer is checked against two independent references. The first real send is the first real verification, so do step 9 with a phone you hold before pointing production traffic at it.

Sources ​

AWS documentation relied on for this page (read when it was written; AWS changes these, so check the page if a step disagrees with what you see):

Released under the MIT License.