Work / Landing zone
Landing zone, March 2025
From one AWS account to six
A SaaS company ran production, staging and development in one undocumented AWS account. We moved it to six accounts under Control Tower, with 12 permission sets, Terraform modules, canary deployments and fault injection tests.
The situation
The client is a growing SaaS company. Its whole operation ran in one AWS account with no documentation. We found four problems.
Mixed environments
Production, staging and development shared one account. Resource names such as "prod-maybe" and "test-final-v2" did not say which environment a resource belonged to.
No access control
Developers had administrator access to production. There was no audit trail. Credentials were shared in Slack channels.
More than 200 IAM policies
The policies overlapped. No one could say who had access to what.
Service quotas reached
The account had reached its Lambda, API Gateway and VPC quotas, so workloads could not scale.
ec2-prod-1rds-mainlambda-apiec2-stg-testrds-staging-2lambda-testec2-dev-johnrds-dev-finallambda-poc- A developer could delete the production RDS database by mistake.
- Backups had never been tested.
- Production ran in a single Availability Zone.
- Secrets were written into Lambda code.
- No monitoring or alerting was configured.
- Deployments were made by hand in the AWS console.
Six accounts under Control Tower
We designed a multi-account structure with one account per environment and per shared function. A management account holds Control Tower, single sign-on and billing. Six accounts sit under it.
Management account Control Tower, single sign-on, billingSecurity
- GuardDuty
- Security Hub
- CloudTrail
Shared services
- CI/CD pipelines
- Build artifacts
- DNS
Network
- Transit Gateway
- VPN
- Direct Connect
Production
- Workloads
- RDS
- ElastiCache
Staging
- Pre-production testing
- Quality assurance
Development
- Development workloads
- Sandbox
Fault containment
A fault in development cannot affect production.
Separate service quotas
Each account has its own AWS quotas.
Cost by environment
Each environment's cost is visible on its own.
Audit trail by environment
Each environment has its own CloudTrail record.
Access: 12 permission sets instead of 200 policies
We replaced more than 200 IAM policies with a least-privilege model. Users sign in through IAM Identity Center, formerly AWS SSO, with the client's identity provider. Access is granted through 12 standard permission sets.
- More than 200 custom IAM policies
- Inline policies on most roles
- Shared IAM users with access keys
- No MFA requirement
- Administrator access for convenience
- 12 standard permission sets
- IAM Identity Center with the identity provider
- Roles assumed through federation
- MFA required for every user
- Elevated access for a limited time, on request
IAM Identity Center identity provider, MFA for every user
Standard Terraform modules
We wrote reusable Terraform modules that apply the same high availability, security and observability settings in every environment.
vpc-standard
ecs-service
rds-cluster
elasticache
Single Availability ZonetoMulti-AZ
No failovertoAutomatic failover
1 NAT gatewayto1 NAT gateway per Availability Zone
Canary deployments
Deployments run through AWS CodeDeploy in four stages. CloudWatch alarms check each stage, and a failed alarm rolls the deployment back without manual action.
10 % of traffic
The new version receives 10 % of traffic.
30 % of traffic
If no alarm fires, traffic rises to 30 %.
60 % of traffic
Traffic rises to 60 %.
100 % of traffic
The new version receives all traffic.
Automatic rollback
The deployment rolls back when either alarm fires. No one has to act.
ManualtoAutomatic
30 minto15 min
5 %to0.1 %
Fault injection with AWS FIS
We used AWS Fault Injection Simulator (FIS) to test the new architecture against four failure cases. All four experiments passed.
| Experiment | Action | Result | Status |
|---|---|---|---|
| Availability Zone failure | Stop every EC2 instance and ECS task in Availability Zone A | Traffic moved to Availability Zones B and C in 45 s | Passed |
| RDS failover | Force a failover of the RDS primary instance | The application reconnected in 30 s | Passed |
| ElastiCache node failure | Stop the primary Redis node | The application continued without the cache and read from the database | Passed |
| Network latency | Add 200 ms of latency to a downstream service | The circuit breaker opened and the service degraded without errors | Passed |
Quarterly GameDay
We set up a quarterly GameDay, a planned exercise in which the team injects faults and practises the response.
Scheduled FIS experiments
Automated resilience tests run in staging every week.
Runbook validation
The on-call team practises incident response against the runbooks.
Recovery time tracking
Mean time to recovery is measured at each exercise.
Findings documented
Each weakness found is recorded and fixed.
Resilience in the application
We identified the services the application can run without, such as the cache, and added circuit breakers in the application code. Automated integration tests with fault injection confirm that the application switches to the primary data source within seconds when the cache fails or times out.
Cache answers the request
Database not read
Database answers the request
The circuit closes again when the cache returns.
The figures
| Measure | Before | After | Change |
|---|---|---|---|
| Production incidents per month | 12 | 2 | 85 % fewer |
| Deployment frequency | Weekly | Daily | 7 times more often |
| Mean time to recovery | 4 h | 15 min | 94 % shorter |
| Critical security findings | 47 | 0 | All resolved |
| Uptime, % | 99.5 | 99.99 | 4 nines |
What the client received
- Six AWS accounts under Control Tower, with a management account for single sign-on and billing.
- IAM Identity Center with 12 permission sets and MFA for every user.
- Four Terraform modules:
vpc-standard,ecs-service,rds-clusterandelasticache. - A CodeDeploy canary pipeline with automatic rollback.
- Four FIS experiments, a quarterly GameDay and circuit breakers in the application.
A landing zone for your estate
We assess your account first, then agree the target structure with your team, then move each environment into its own account with a rollback plan for each step. You receive the Terraform code and the documentation.
Talk to an engineer