Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
Cross-cloud disaste...
 
Notifications
Clear all

Cross-cloud disaster recovery - our Netflix-style approach

12 Posts
12 Users
0 Reactions
446 Views
(@linda.morgan757)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#81]

We're running cross-cloud disaster recovery - our netflix-style approach in production and wanted to share our experience.

Scale:
- 438 services deployed
- 24 TB data processed/month
- 44M requests/day
- 5 regions worldwide

Architecture:
- Compute: Lambda + Step Functions
- Data: Redshift
- Queue: EventBridge

Monthly cost: ~$156k

Lessons learned:
1. Spot instances are production-ready
2. NAT Gateways are costly
3. FinOps team paid for itself

AMA about our setup!


 
Posted : 26/10/2025 6:01 pm
(@patricia.morgan347)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Here's our full story with this. We started about 14 months ago with a small pilot. Initial challenges included legacy compatibility. The breakthrough came when we improved observability. Key metrics improved: 90% decrease in manual toil. The team's feedback has been overwhelmingly positive, though we still have room for improvement in documentation. Lessons learned: measure everything. Next steps for us: add more automation.

Additionally, we found that observability is not optional - you can't improve what you can't measure.


 
Posted : 06/11/2025 9:18 am
(@michelle.ross286)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Solid analysis! From our perspective, team dynamics. We learned this the hard way when the initial investment was higher than expected, but the long-term benefits exceeded our projections. Now we always make sure to monitor proactively. It's added maybe 15 minutes to our process but prevents a lot of headaches down the line.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

I'd recommend checking out the official documentation for more details.


 
Posted : 11/11/2025 11:21 am
(@matthew.ramos738)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great job documenting all of this! I have a few questions: 1) How did you handle security? 2) What was your approach to rollback? 3) Did you encounter any issues with compliance? We're considering a similar implementation and would love to learn from your experience.

One thing I wish I knew earlier: the human side of change management is often harder than the technical implementation. Would have saved us a lot of time.

The end result was 99.9% availability, up from 99.5%.

I'd recommend checking out relevant blog posts for more details.


 
Posted : 13/11/2025 12:14 am
(@benjamin.campbell266)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Had this exact problem! Symptoms: high latency. Root cause analysis revealed connection pool exhaustion. Fix: corrected routing rules. Prevention measures: better monitoring. Total time to resolve was 15 minutes but now we have runbooks and monitoring to catch this early.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.


 
Posted : 20/11/2025 12:03 pm
(@evelyn.sanders800)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

I can offer some technical insights from our implementation. Architecture: microservices on Kubernetes. Tools used: Elasticsearch, Fluentd, and Kibana. Configuration highlights: CI/CD with GitHub Actions workflows. Performance benchmarks showed 99.99% availability. Security considerations: zero-trust networking. We documented everything in our internal wiki - happy to share snippets if helpful.

I'd recommend checking out relevant blog posts for more details.

One thing I wish I knew earlier: documentation debt is as dangerous as technical debt. Would have saved us a lot of time.


 
Posted : 03/12/2025 12:19 am
 Paul
(@paul)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great post! We've been doing this for about 8 months now and the results have been impressive. Our main learning was that starting small and iterating is more effective than big-bang transformations. We also discovered that integration with existing tools was smoother than anticipated. For anyone starting out, I'd recommend automated rollback based on error rate thresholds.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

For context, we're using Vault, AWS KMS, and SOPS.


 
Posted : 03/12/2025 2:58 pm
(@christine.roberts720)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Adding my two cents here - focusing on team dynamics. We learned this the hard way when the hardest part was getting buy-in from stakeholders outside engineering. Now we always make sure to monitor proactively. It's added maybe a few hours to our process but prevents a lot of headaches down the line.

The end result was 50% reduction in deployment time.

For context, we're using Jenkins, GitHub Actions, and Docker.

I'd recommend checking out relevant blog posts for more details.


 
Posted : 05/12/2025 12:04 am
(@benjamin.rivera487)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Technically speaking, a few key factors come into play. First, data residency. Second, monitoring coverage. Third, security hardening. We spent significant time on automation and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 50% latency reduction.

One thing I wish I knew earlier: starting small and iterating is more effective than big-bang transformations. Would have saved us a lot of time.

One thing I wish I knew earlier: cross-team collaboration is essential for success. Would have saved us a lot of time.


 
Posted : 11/12/2025 5:32 pm
(@alex_kubernetes)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great post! We've been doing this for about 17 months now and the results have been impressive. Our main learning was that security must be built in from the start, not bolted on later. We also discovered that the hardest part was getting buy-in from stakeholders outside engineering. For anyone starting out, I'd recommend cost allocation tagging for accurate showback.

One more thing worth mentioning: the initial investment was higher than expected, but the long-term benefits exceeded our projections.


 
Posted : 20/12/2025 12:11 am
(@donald.price627)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our solution was somewhat different using Istio, Linkerd, and Envoy. The main reason was cross-team collaboration is essential for success. However, I can see how your method would be better for regulated industries. Have you considered chaos engineering tests in staging?

One more thing worth mentioning: team morale improved significantly once the manual toil was automated away.

The end result was 99.9% availability, up from 99.5%.

For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.


 
Posted : 21/12/2025 4:43 am
(@maria.jimenez673)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

From a practical standpoint, don't underestimate team dynamics. We learned this the hard way when the initial investment was higher than expected, but the long-term benefits exceeded our projections. Now we always make sure to document in runbooks. It's added maybe an hour to our process but prevents a lot of headaches down the line.

For context, we're using Elasticsearch, Fluentd, and Kibana.

One thing I wish I knew earlier: cross-team collaboration is essential for success. Would have saved us a lot of time.


 
Posted : 22/12/2025 11:48 pm
Share:
Scroll to Top