Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
How we achieved 99....
 
Notifications
Clear all

How we achieved 99.99% uptime with chaos engineering

17 Posts
15 Users
0 Reactions
333 Views
(@david_jenkins)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#113]

Project: How we achieved 99.99% uptime with chaos engineering

Timeline: 14 months
Team: 2 engineers
Budget: $211k

Challenge:
We needed to migrate to cloud while maintaining 99.99% SLA.

Solution:
We implemented a blue-green deployment strategy using:
- Service mesh with Istio
- Chaos engineering
- Platform engineering team

Results:
✓ Lead time: 2 weeks → 2 hours
✓ Compliance audit passed first try
✓ Team can focus on features

Happy to discuss our approach and share learnings!


 
Posted : 28/08/2025 8:26 am
(@linda.morgan757)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

This level of detail is exactly what we needed! I have a few questions: 1) How did you handle monitoring? 2) What was your approach to canary? 3) Did you encounter any issues with availability? We're considering a similar implementation and would love to learn from your experience.

Additionally, we found that observability is not optional - you can't improve what you can't measure.

The end result was 80% reduction in security vulnerabilities.

I'd recommend checking out relevant blog posts for more details.


 
Posted : 31/08/2025 4:46 pm
(@jose.jackson593)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Technical perspective from our implementation. Architecture: hybrid cloud setup. Tools used: Elasticsearch, Fluentd, and Kibana. Configuration highlights: GitOps with ArgoCD apps. Performance benchmarks showed 50% latency reduction. Security considerations: container scanning in CI. We documented everything in our internal wiki - happy to share snippets if helpful.

For context, we're using Datadog, PagerDuty, and Slack.

For context, we're using Vault, AWS KMS, and SOPS.


 
Posted : 01/09/2025 12:15 am
(@scott.allen968)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

From beginning to end, here's what we did with this. We started about 12 months ago with a small pilot. Initial challenges included performance issues. The breakthrough came when we automated the testing. Key metrics improved: 3x increase in deployment frequency. The team's feedback has been overwhelmingly positive, though we still have room for improvement in testing coverage. Lessons learned: start simple. Next steps for us: optimize costs.

I'd recommend checking out the official documentation for more details.


 
Posted : 01/09/2025 11:07 pm
(@samuel.miller567)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We encountered this as well! Symptoms: frequent timeouts. Root cause analysis revealed network misconfiguration. Fix: fixed the leak. Prevention measures: better monitoring. Total time to resolve was an hour but now we have runbooks and monitoring to catch this early.

Additionally, we found that automation should augment human decision-making, not replace it entirely.

For context, we're using Grafana, Loki, and Tempo.

I'd recommend checking out conference talks on YouTube for more details.

For context, we're using Istio, Linkerd, and Envoy.


 
Posted : 02/09/2025 7:28 pm
(@jennifer.young148)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We encountered something similar during our last sprint. The problem: security vulnerabilities. Our initial approach was simple scripts but that didn't work because too error-prone. What actually worked: real-time dashboards for stakeholder visibility. The key insight was cross-team collaboration is essential for success. Now we're able to scale automatically.

I'd recommend checking out relevant blog posts for more details.

One thing I wish I knew earlier: the human side of change management is often harder than the technical implementation. Would have saved us a lot of time.


 
Posted : 03/09/2025 2:28 am
(@andrew.roberts887)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Same issue on our end! Symptoms: high latency. Root cause analysis revealed connection pool exhaustion. Fix: corrected routing rules. Prevention measures: chaos engineering. Total time to resolve was 30 minutes but now we have runbooks and monitoring to catch this early.

For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.

One thing I wish I knew earlier: security must be built in from the start, not bolted on later. Would have saved us a lot of time.

The end result was 99.9% availability, up from 99.5%.


 
Posted : 05/09/2025 8:40 pm
(@christopher.mitchell35)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Wanted to contribute some real-world operational insights we've developed: Monitoring - Prometheus with Grafana dashboards. Alerting - Opsgenie with escalation policies. Documentation - Notion for team wikis. Training - monthly lunch and learns. These have helped us maintain fast deployments while still moving fast on new features.

I'd recommend checking out relevant blog posts for more details.

One thing I wish I knew earlier: observability is not optional - you can't improve what you can't measure. Would have saved us a lot of time.


 
Posted : 07/09/2025 3:20 pm
(@benjamin.taylor696)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our experience was remarkably similar. The problem: scaling issues. Our initial approach was manual intervention but that didn't work because lacked visibility. What actually worked: cost allocation tagging for accurate showback. The key insight was the human side of change management is often harder than the technical implementation. Now we're able to deploy with confidence.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

The end result was 60% improvement in developer productivity.


 
Posted : 16/09/2025 2:46 pm
(@jennifer.bailey132)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Some practical ops guidance that might helps we've developed: Monitoring - CloudWatch with custom metrics. Alerting - PagerDuty with intelligent routing. Documentation - Confluence with templates. Training - monthly lunch and learns. These have helped us maintain high reliability while still moving fast on new features.

The end result was 40% cost savings on infrastructure.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One thing I wish I knew earlier: documentation debt is as dangerous as technical debt. Would have saved us a lot of time.


 
Posted : 21/09/2025 10:50 am
(@christina.gutierrez3)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

So relatable! Our experience was that we learned: Phase 1 (6 weeks) involved stakeholder alignment. Phase 2 (2 months) focused on process documentation. Phase 3 (1 month) was all about knowledge sharing. Total investment was $200K but the payback period was only 9 months. Key success factors: automation, documentation, feedback loops. If I could do it again, I would start with better documentation.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 28/09/2025 2:57 pm
(@kimberly.james491)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Makes sense! For us, the approach varied using Kubernetes, Helm, ArgoCD, and Prometheus. The main reason was starting small and iterating is more effective than big-bang transformations. However, I can see how your method would be better for fast-moving startups. Have you considered compliance scanning in the CI pipeline?

The end result was 3x increase in deployment frequency.

I'd recommend checking out relevant blog posts for more details.

One more thing worth mentioning: we had to iterate several times before finding the right balance.


 
Posted : 08/10/2025 5:50 am
(@rebecca.brown460)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

I'd like to share our complete experience with this. We started about 24 months ago with a small pilot. Initial challenges included team training. The breakthrough came when we improved observability. Key metrics improved: 70% reduction in incident MTTR. The team's feedback has been overwhelmingly positive, though we still have room for improvement in monitoring depth. Lessons learned: automate everything. Next steps for us: add more automation.

Additionally, we found that observability is not optional - you can't improve what you can't measure.


 
Posted : 11/10/2025 8:02 am
(@christopher.mitchell35)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Love this! In our organization and can confirm the benefits. One thing we added was cost allocation tagging for accurate showback. The key insight for us was understanding that documentation debt is as dangerous as technical debt. We also found that we discovered several hidden dependencies during the migration. Happy to share more details if anyone is interested.

One thing I wish I knew earlier: starting small and iterating is more effective than big-bang transformations. Would have saved us a lot of time.


 
Posted : 11/10/2025 9:45 am
(@benjamin.taylor696)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our implementation in our organization and can confirm the benefits. One thing we added was automated rollback based on error rate thresholds. The key insight for us was understanding that documentation debt is as dangerous as technical debt. We also found that we had to iterate several times before finding the right balance. Happy to share more details if anyone is interested.

For context, we're using Vault, AWS KMS, and SOPS.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 15/10/2025 7:11 am
Page 1 / 2
Share:
Scroll to Top