Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
How we achieved 99....
 
Notifications
Clear all

How we achieved 99.99% uptime with chaos engineering

23 Posts
19 Users
0 Reactions
112 Views
(@alex_kubernetes)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#104]

Project: How we achieved 99.99% uptime with chaos engineering

Timeline: 9 months
Team: 5 engineers
Budget: $276k

Challenge:
We needed to improve deployment speed while maintaining backward compatibility.

Solution:
We implemented a strangler fig pattern using:
- GitOps with ArgoCD
- Feature flags
- DevSecOps integration

Results:
✓ Deployment frequency: 1/week → 50/day
✓ Onboarding time cut in half
✓ Team can focus on features

Happy to discuss our approach and share learnings!


 
Posted : 27/10/2025 10:57 pm
(@angela.nguyen556)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

This is almost identical to what we faced. The problem: deployment failures. Our initial approach was simple scripts but that didn't work because it didn't scale. What actually worked: automated rollback based on error rate thresholds. The key insight was the human side of change management is often harder than the technical implementation. Now we're able to detect issues early.

One thing I wish I knew earlier: observability is not optional - you can't improve what you can't measure. Would have saved us a lot of time.


 
Posted : 28/10/2025 7:07 am
(@angela.nguyen556)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

This resonates with my experience, though I'd emphasize security considerations. We learned this the hard way when the hardest part was getting buy-in from stakeholders outside engineering. Now we always make sure to document in runbooks. It's added maybe 15 minutes to our process but prevents a lot of headaches down the line.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One more thing worth mentioning: we discovered several hidden dependencies during the migration.


 
Posted : 29/10/2025 10:58 am
(@jennifer.young148)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

The technical aspects here are nuanced. First, data residency. Second, backup procedures. Third, security hardening. We spent significant time on automation and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 50% latency reduction.

Additionally, we found that security must be built in from the start, not bolted on later.

I'd recommend checking out the community forums for more details.

The end result was 50% reduction in deployment time.


 
Posted : 03/11/2025 10:15 pm
(@benjamin.taylor696)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Nice! We did something similar in our organization and can confirm the benefits. One thing we added was automated rollback based on error rate thresholds. The key insight for us was understanding that cross-team collaboration is essential for success. We also found that team morale improved significantly once the manual toil was automated away. Happy to share more details if anyone is interested.

For context, we're using Jenkins, GitHub Actions, and Docker.

Additionally, we found that the human side of change management is often harder than the technical implementation.


 
Posted : 04/11/2025 9:53 am
(@mary.castillo14)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Thoughtful post - though I'd challenge one aspect on the timeline. In our environment, we found that Datadog, PagerDuty, and Slack worked better because failure modes should be designed for, not discovered in production. That said, context matters a lot - what works for us might not work for everyone. The key is to focus on outcomes.

I'd recommend checking out relevant blog posts for more details.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 04/11/2025 7:07 pm
(@mary.castillo14)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

From an operations perspective, here's what we recommends we've developed: Monitoring - Datadog APM and logs. Alerting - Opsgenie with escalation policies. Documentation - Confluence with templates. Training - certification programs. These have helped us maintain high reliability while still moving fast on new features.

One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.

For context, we're using Terraform, AWS CDK, and CloudFormation.


 
Posted : 06/11/2025 3:06 pm
(@katherine.nelson24)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

From an implementation perspective, here are the key points. First, compliance requirements. Second, failover strategy. Third, security hardening. We spent significant time on automation and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 50% latency reduction.

One more thing worth mentioning: we had to iterate several times before finding the right balance.

The end result was 90% decrease in manual toil.

One thing I wish I knew earlier: failure modes should be designed for, not discovered in production. Would have saved us a lot of time.


 
Posted : 06/11/2025 10:49 pm
(@james.allen159)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Neat! We solved this another way using Terraform, AWS CDK, and CloudFormation. The main reason was automation should augment human decision-making, not replace it entirely. However, I can see how your method would be better for fast-moving startups. Have you considered compliance scanning in the CI pipeline?

I'd recommend checking out the community forums for more details.

The end result was 80% reduction in security vulnerabilities.

Additionally, we found that the human side of change management is often harder than the technical implementation.


 
Posted : 07/11/2025 1:34 am
(@christina.gutierrez3)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

From beginning to end, here's what we did with this. We started about 4 months ago with a small pilot. Initial challenges included tool integration. The breakthrough came when we automated the testing. Key metrics improved: 40% cost savings on infrastructure. The team's feedback has been overwhelmingly positive, though we still have room for improvement in automation. Lessons learned: start simple. Next steps for us: improve documentation.

For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.


 
Posted : 08/11/2025 9:07 am
(@timothy.scott735)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great post! We've been doing this for about 4 months now and the results have been impressive. Our main learning was that cross-team collaboration is essential for success. We also discovered that the hardest part was getting buy-in from stakeholders outside engineering. For anyone starting out, I'd recommend cost allocation tagging for accurate showback.

I'd recommend checking out the community forums for more details.

One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.


 
Posted : 08/11/2025 11:53 am
(@mary.castillo14)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We encountered something similar during our last sprint. The problem: deployment failures. Our initial approach was manual intervention but that didn't work because it didn't scale. What actually worked: chaos engineering tests in staging. The key insight was automation should augment human decision-making, not replace it entirely. Now we're able to detect issues early.

Additionally, we found that automation should augment human decision-making, not replace it entirely.

For context, we're using Datadog, PagerDuty, and Slack.


 
Posted : 10/11/2025 9:09 pm
(@robert.stewart107)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our end-to-end experience with this. We started about 24 months ago with a small pilot. Initial challenges included performance issues. The breakthrough came when we streamlined the process. Key metrics improved: 80% reduction in security vulnerabilities. The team's feedback has been overwhelmingly positive, though we still have room for improvement in automation. Lessons learned: automate everything. Next steps for us: expand to more teams.

For context, we're using Jenkins, GitHub Actions, and Docker.


 
Posted : 14/11/2025 2:08 pm
(@alexander.rodriguez755)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Experienced this firsthand! Symptoms: increased error rates. Root cause analysis revealed connection pool exhaustion. Fix: corrected routing rules. Prevention measures: load testing. Total time to resolve was a few hours but now we have runbooks and monitoring to catch this early.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

For context, we're using Istio, Linkerd, and Envoy.

I'd recommend checking out relevant blog posts for more details.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 19/11/2025 1:01 pm
(@rebecca.brown460)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Solid work putting this together! I have a few questions: 1) How did you handle scaling? 2) What was your approach to backup? 3) Did you encounter any issues with availability? We're considering a similar implementation and would love to learn from your experience.

The end result was 60% improvement in developer productivity.

One more thing worth mentioning: integration with existing tools was smoother than anticipated.

The end result was 99.9% availability, up from 99.5%.

Additionally, we found that cross-team collaboration is essential for success.


 
Posted : 20/11/2025 1:48 am
Page 1 / 2
Share:
Scroll to Top