Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
Deep dive: On-call ...
 
Notifications
Clear all

Deep dive: On-call rotation best practices to prevent burnout

16 Posts
15 Users
0 Reactions
383 Views
(@donald.white940)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#206]

From the ops trenches, here's our takes we've developed: Monitoring - Datadog APM and logs. Alerting - custom Slack integration. Documentation - Notion for team wikis. Training - certification programs. These have helped us maintain fast deployments while still moving fast on new features.

For context, we're using Elasticsearch, Fluentd, and Kibana.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Additionally, we found that security must be built in from the start, not bolted on later.

The end result was 3x increase in deployment frequency.

The end result was 90% decrease in manual toil.

For context, we're using Istio, Linkerd, and Envoy.

One thing I wish I knew earlier: starting small and iterating is more effective than big-bang transformations. Would have saved us a lot of time.


 
Posted : 22/12/2024 7:21 pm
(@victoria.rivera433)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great post! We've been doing this for about 4 months now and the results have been impressive. Our main learning was that cross-team collaboration is essential for success. We also discovered that we discovered several hidden dependencies during the migration. For anyone starting out, I'd recommend chaos engineering tests in staging.

For context, we're using Elasticsearch, Fluentd, and Kibana.

I'd recommend checking out relevant blog posts for more details.

I'd recommend checking out the official documentation for more details.


 
Posted : 24/12/2024 7:43 am
(@timothy.wood427)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Here's the technical breakdown of our implementation. Architecture: microservices on Kubernetes. Tools used: Vault, AWS KMS, and SOPS. Configuration highlights: IaC with Terraform modules. Performance benchmarks showed 50% latency reduction. Security considerations: secrets management with Vault. We documented everything in our internal wiki - happy to share snippets if helpful.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.


 
Posted : 25/12/2024 8:10 am
(@rachel.morales858)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Here's our full story with this. We started about 3 months ago with a small pilot. Initial challenges included tool integration. The breakthrough came when we improved observability. Key metrics improved: 70% reduction in incident MTTR. The team's feedback has been overwhelmingly positive, though we still have room for improvement in monitoring depth. Lessons learned: measure everything. Next steps for us: expand to more teams.

Additionally, we found that documentation debt is as dangerous as technical debt.

One more thing worth mentioning: the hardest part was getting buy-in from stakeholders outside engineering.

I'd recommend checking out relevant blog posts for more details.

One more thing worth mentioning: the initial investment was higher than expected, but the long-term benefits exceeded our projections.

The end result was 90% decrease in manual toil.

I'd recommend checking out the official documentation for more details.

I'd recommend checking out conference talks on YouTube for more details.


 
Posted : 25/12/2024 6:11 pm
(@john.long261)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We encountered something similar during our last sprint. The problem: deployment failures. Our initial approach was manual intervention but that didn't work because too error-prone. What actually worked: cost allocation tagging for accurate showback. The key insight was starting small and iterating is more effective than big-bang transformations. Now we're able to scale automatically.

One more thing worth mentioning: team morale improved significantly once the manual toil was automated away.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 26/12/2024 11:08 am
(@david_jenkins)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

I'll walk you through our entire process with this. We started about 4 months ago with a small pilot. Initial challenges included legacy compatibility. The breakthrough came when we improved observability. Key metrics improved: 70% reduction in incident MTTR. The team's feedback has been overwhelmingly positive, though we still have room for improvement in testing coverage. Lessons learned: measure everything. Next steps for us: optimize costs.

One thing I wish I knew earlier: the human side of change management is often harder than the technical implementation. Would have saved us a lot of time.


 
Posted : 27/12/2024 3:47 am
(@maria_terraform)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

On the operational side, some thoughtss we've developed: Monitoring - Prometheus with Grafana dashboards. Alerting - custom Slack integration. Documentation - Notion for team wikis. Training - certification programs. These have helped us maintain low incident count while still moving fast on new features.

One thing I wish I knew earlier: observability is not optional - you can't improve what you can't measure. Would have saved us a lot of time.

I'd recommend checking out conference talks on YouTube for more details.


 
Posted : 28/12/2024 4:35 pm
(@stephanie.long568)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Thoughtful post - though I'd challenge one aspect on the team structure. In our environment, we found that Terraform, AWS CDK, and CloudFormation worked better because the human side of change management is often harder than the technical implementation. That said, context matters a lot - what works for us might not work for everyone. The key is to start small and iterate.

The end result was 99.9% availability, up from 99.5%.

The end result was 70% reduction in incident MTTR.

Additionally, we found that cross-team collaboration is essential for success.


 
Posted : 29/12/2024 1:08 am
(@rachel.morales858)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Had this exact problem! Symptoms: increased error rates. Root cause analysis revealed memory leaks. Fix: increased pool size. Prevention measures: chaos engineering. Total time to resolve was a few hours but now we have runbooks and monitoring to catch this early.

The end result was 80% reduction in security vulnerabilities.

One thing I wish I knew earlier: failure modes should be designed for, not discovered in production. Would have saved us a lot of time.

For context, we're using Elasticsearch, Fluentd, and Kibana.

One more thing worth mentioning: we had to iterate several times before finding the right balance.

One more thing worth mentioning: team morale improved significantly once the manual toil was automated away.

For context, we're using Grafana, Loki, and Tempo.

The end result was 50% reduction in deployment time.

One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.

One thing I wish I knew earlier: security must be built in from the start, not bolted on later. Would have saved us a lot of time.


 
Posted : 30/12/2024 10:15 pm
(@robert.stewart107)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We went through something very similar. The problem: deployment failures. Our initial approach was simple scripts but that didn't work because lacked visibility. What actually worked: chaos engineering tests in staging. The key insight was automation should augment human decision-making, not replace it entirely. Now we're able to deploy with confidence.

One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.


 
Posted : 01/01/2025 4:55 am
(@brandon.williams519)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Practical advice from our team: 1) Test in production-like environments 2) Monitor proactively 3) Review and iterate 4) Keep it simple. Common mistakes to avoid: over-engineering early. Resources that helped us: Team Topologies. The most important thing is collaboration over tools.

One more thing worth mentioning: unexpected benefits included better developer experience and faster onboarding.

Additionally, we found that the human side of change management is often harder than the technical implementation.


 
Posted : 03/01/2025 1:22 am
(@alexander.smith802)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

From beginning to end, here's what we did with this. We started about 4 months ago with a small pilot. Initial challenges included team training. The breakthrough came when we automated the testing. Key metrics improved: 90% decrease in manual toil. The team's feedback has been overwhelmingly positive, though we still have room for improvement in automation. Lessons learned: measure everything. Next steps for us: optimize costs.

The end result was 3x increase in deployment frequency.

One thing I wish I knew earlier: security must be built in from the start, not bolted on later. Would have saved us a lot of time.


 
Posted : 04/01/2025 1:25 am
(@donald.price627)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Architecturally, there are important trade-offs to consider. First, data residency. Second, failover strategy. Third, performance tuning. We spent significant time on automation and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 2x improvement.

The end result was 3x increase in deployment frequency.

One more thing worth mentioning: we discovered several hidden dependencies during the migration.

For context, we're using Terraform, AWS CDK, and CloudFormation.


 
Posted : 04/01/2025 3:56 pm
(@tyler.robinson235)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

This is a really thorough analysis! I have a few questions: 1) How did you handle scaling? 2) What was your approach to blue-green? 3) Did you encounter any issues with consistency? We're considering a similar implementation and would love to learn from your experience.

One more thing worth mentioning: we had to iterate several times before finding the right balance.

The end result was 60% improvement in developer productivity.

The end result was 50% reduction in deployment time.

For context, we're using Istio, Linkerd, and Envoy.


 
Posted : 05/01/2025 3:21 am
(@victoria.robinson772)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Here are some technical specifics from our implementation. Architecture: serverless with Lambda. Tools used: Grafana, Loki, and Tempo. Configuration highlights: IaC with Terraform modules. Performance benchmarks showed 50% latency reduction. Security considerations: secrets management with Vault. We documented everything in our internal wiki - happy to share snippets if helpful.

One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.

For context, we're using Grafana, Loki, and Tempo.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.

For context, we're using Istio, Linkerd, and Envoy.

The end result was 3x increase in deployment frequency.

The end result was 90% decrease in manual toil.

The end result was 70% reduction in incident MTTR.

The end result was 80% reduction in security vulnerabilities.

Additionally, we found that automation should augment human decision-making, not replace it entirely.


 
Posted : 06/01/2025 8:24 am
Page 1 / 2
Share:
Scroll to Top