We're running aws organizations best practices for 50+ accounts in production and wanted to share our experience.
Scale:
- 758 services deployed
- 55 TB data processed/month
- 7M requests/day
- 11 regions worldwide
Architecture:
- Compute: EC2 Auto Scaling
- Data: S3 + Athena
- Queue: MSK (Kafka)
Monthly cost: ~$124k
Lessons learned:
1. Multi-AZ costs add up fast
2. NAT Gateways are costly
3. Cold starts still an issue
AMA about our setup!
Great post! We've been doing this for about 22 months now and the results have been impressive. Our main learning was that starting small and iterating is more effective than big-bang transformations. We also discovered that the initial investment was higher than expected, but the long-term benefits exceeded our projections. For anyone starting out, I'd recommend automated rollback based on error rate thresholds.
Additionally, we found that cross-team collaboration is essential for success.
One thing I wish I knew earlier: security must be built in from the start, not bolted on later. Would have saved us a lot of time.
Great post! We've been doing this for about 4 months now and the results have been impressive. Our main learning was that failure modes should be designed for, not discovered in production. We also discovered that the hardest part was getting buy-in from stakeholders outside engineering. For anyone starting out, I'd recommend integration with our incident management system.
Additionally, we found that starting small and iterating is more effective than big-bang transformations.
One more thing worth mentioning: team morale improved significantly once the manual toil was automated away.
Some guidance based on our experience: 1) Document as you go 2) Monitor proactively 3) Share knowledge across teams 4) Keep it simple. Common mistakes to avoid: skipping documentation. Resources that helped us: Google SRE book. The most important thing is collaboration over tools.
For context, we're using Datadog, PagerDuty, and Slack.
Additionally, we found that documentation debt is as dangerous as technical debt.
One more thing worth mentioning: we discovered several hidden dependencies during the migration.
Thanks for this! We're beginning our evaluation ofg this approach. Could you elaborate on the migration process? Specifically, I'm curious about team training approach. Also, how long did the initial implementation take? Any gotchas we should watch out for?
Additionally, we found that the human side of change management is often harder than the technical implementation.
One more thing worth mentioning: unexpected benefits included better developer experience and faster onboarding.
One more thing worth mentioning: team morale improved significantly once the manual toil was automated away.
A few operational considerations to adds we've developed: Monitoring - Prometheus with Grafana dashboards. Alerting - Opsgenie with escalation policies. Documentation - Confluence with templates. Training - certification programs. These have helped us maintain fast deployments while still moving fast on new features.
One more thing worth mentioning: integration with existing tools was smoother than anticipated.
One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.
This is a really thorough analysis! I have a few questions: 1) How did you handle monitoring? 2) What was your approach to canary? 3) Did you encounter any issues with compliance? We're considering a similar implementation and would love to learn from your experience.
Additionally, we found that the human side of change management is often harder than the technical implementation.
One more thing worth mentioning: the initial investment was higher than expected, but the long-term benefits exceeded our projections.
Our experience was remarkably similar. The problem: deployment failures. Our initial approach was ad-hoc monitoring but that didn't work because it didn't scale. What actually worked: automated rollback based on error rate thresholds. The key insight was observability is not optional - you can't improve what you can't measure. Now we're able to scale automatically.
I'd recommend checking out the community forums for more details.
One thing I wish I knew earlier: documentation debt is as dangerous as technical debt. Would have saved us a lot of time.
This is exactly the kind of detail that helps! I have a few questions: 1) How did you handle monitoring? 2) What was your approach to blue-green? 3) Did you encounter any issues with latency? We're considering a similar implementation and would love to learn from your experience.
Additionally, we found that automation should augment human decision-making, not replace it entirely.
Feel free to reach out if you have more questions - happy to share our runbooks and documentation.
For context, we're using Jenkins, GitHub Actions, and Docker.
The technical aspects here are nuanced. First, network topology. Second, monitoring coverage. Third, performance tuning. We spent significant time on testing and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 10x throughput increase.
The end result was 90% decrease in manual toil.
I'd recommend checking out relevant blog posts for more details.
One more thing worth mentioning: we had to iterate several times before finding the right balance.
Our experience was remarkably similar! We learned: Phase 1 (6 weeks) involved tool evaluation. Phase 2 (1 month) focused on pilot implementation. Phase 3 (ongoing) was all about knowledge sharing. Total investment was $100K but the payback period was only 9 months. Key success factors: automation, documentation, feedback loops. If I could do it again, I would start with better documentation.
For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.
Additionally, we found that documentation debt is as dangerous as technical debt.
Some tips from our journey: 1) Test in production-like environments 2) Use feature flags 3) Review and iterate 4) Measure what matters. Common mistakes to avoid: not measuring outcomes. Resources that helped us: Team Topologies. The most important thing is outcomes over outputs.
One thing I wish I knew earlier: cross-team collaboration is essential for success. Would have saved us a lot of time.
One thing I wish I knew earlier: documentation debt is as dangerous as technical debt. Would have saved us a lot of time.
Good analysis, though I have a different take on this on the timeline. In our environment, we found that Terraform, AWS CDK, and CloudFormation worked better because starting small and iterating is more effective than big-bang transformations. That said, context matters a lot - what works for us might not work for everyone. The key is to experiment and measure.
The end result was 60% improvement in developer productivity.
I'd recommend checking out the official documentation for more details.
Want to share our path through this. We started about 8 months ago with a small pilot. Initial challenges included tool integration. The breakthrough came when we simplified the architecture. Key metrics improved: 70% reduction in incident MTTR. The team's feedback has been overwhelmingly positive, though we still have room for improvement in automation. Lessons learned: communicate often. Next steps for us: expand to more teams.
The end result was 80% reduction in security vulnerabilities.
From an implementation perspective, here are the key points. First, data residency. Second, failover strategy. Third, performance tuning. We spent significant time on documentation and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 2x improvement.
For context, we're using Jenkins, GitHub Actions, and Docker.
For context, we're using Terraform, AWS CDK, and CloudFormation.
One more thing worth mentioning: integration with existing tools was smoother than anticipated.