Reliability and recovery

Secure File Transfer is designed to survive the loss of an Availability Zone automatically and the loss of the Tokyo region within an hour, without losing confirmed revocations. This page sets out the targets, the design behind them, and how each one will be proven.

#Targets

The requirements state each target with the way it will be measured. Design load is 500 tenants, about 10,000 transfers and upload requests a day, and about 50 API requests a second on average, with bursts to 300.

Time to shareable includes the malware scan and DLP inspection. Time a transfer waits for its sender's confirmation or an approver isn't counted. A real regional failure is expected to breach the availability target; the RTO bounds the recovery.

#Within the Tokyo region

Production runs across three Availability Zones in Tokyo.

• API pods run at least three replicas spread across the AZs, and autoscale on request rate, CPU and active download streams. The load balancer stops sending traffic to a failed pod.
• Aurora PostgreSQL keeps six copies of the data across three AZs, with a writer and a reader in different AZs. Losing an AZ loses no data, and failover to the reader typically takes under a minute (ADR-0011).
• S3 and SQS are regional services, not tied to one AZ.
• Karpenter replaces a lost AZ's nodes within minutes, and Kubernetes reschedules pods into the other two.

Interrupted transfers recover on their own. An upload resumes from its completed parts. A cut download resumes with a Range request under the same grant, without counting against the download limit. During a deploy, download streams get up to 120 seconds to finish before the browser resumes them.

#Regional failover to Osaka

Osaka (ap-northeast-3) runs a pilot light: the data is always there, and compute starts only on failover.

Failover is always an operator's decision. Health checks never move traffic on their own. The runbook fences Tokyo through Git first, so a returning Tokyo cluster converges to zero workloads. It then promotes the Osaka database, scales up compute, re-applies security-critical changes from the Osaka audit archive, re-publishes unprocessed work, switches DNS and verifies with the synthetic probe. The step budgets add up to 57 minutes, inside the 1-hour RTO. That total is a budget for the first drill to check, not a measurement.

Security-critical changes survive. Revocations, API key revocations, deactivations, lockouts and download grants reach the audit archive in both regions before the API confirms them, and workers stay at zero until they are re-applied. A revocation confirmed in Tokyo therefore holds in Osaka. The one stated exception is a download grant committed in roughly the last second before Tokyo is lost.

Files uploaded in the minutes before a failure may not have reached Osaka. The service shows them as unavailable, not as broken links. SSO members sign in through the standby pool as usual; members without SSO re-enrol by email, because Cognito can't export password hashes or MFA factors.

#Backups and restore

• Point-in-time restore. Aurora keeps 14 days of continuous backups, encrypted with the database key. A bad migration or data corruption is fixed by restoring to a new database at any chosen point, accurate to 5 minutes, within 2 hours (NFR-11). The recovery job then re-imports archived audit records and re-applies security-critical changes made after that point.
• No down-migrations. Migrations are expand-only, so a bad one is fixed forward or by restore, never by running it backwards.
• Files are versioned, not backed up. File buckets are deliberately kept out of AWS Backup and Object Lock, so deletion can reach every copy within 7 days, as the retention rules require. Versioning and the Osaka replica protect them instead. See Revocation and retention.
• The audit archive is immutable. In prod it uses S3 Object Lock in compliance mode with 365-day retention, and no restore or failover removes a record.

#Data residency

All customer data, including files, metadata, backups and logs, stays in Tokyo and Osaka (NFR-21). A service control policy denies every other region. The exceptions are AWS global services (CloudFront, IAM, Route 53, Budgets) and a short list of resources that AWS requires in us-east-1, such as CloudFront's certificate and web ACL, the CloudWatch alarms on CloudFront and Route 53 health checks, and the key that signs the DNS zone. None of them holds file contents. CloudFront serves only the web app, so file contents are never cached at an edge outside Japan.

#How the targets will be proven

The engineering definition of done for v1 is that every non-functional requirement is measured and the result published, not estimated.

• Drills with reports. AZ-failure, regional-failover and restore drills run first in staging, then in prod. Each report records the measured RTO and RPO. The failover drill revokes, locks out, deactivates and downloads just before the cut-over, then checks that each one holds in Osaka.
• Load tests with k6 at 1× and 2× design load, against the latency and scalability targets.
• A synthetic probe every 5 minutes, which feeds the availability SLI.
• SLO dashboards and alerts as code, each alert linked to a runbook.

These belong to milestone M7 and the operations phase. Known gaps that matter only in a failover, restore or failback, such as backup copies in Osaka, are tracked in the open in issue #42.

Source: PRD, Architecture, ADR-0011, ADR-0012, Delivery plan