Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
AI-driven incident ...
 
Notifications
Clear all

AI-driven incident response - our experience with PagerDuty Copilot

16 Posts
13 Users
0 Reactions
118 Views
(@sara)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#56]

We've been experimenting with ai-driven incident response - our experience with pagerduty copilot for the past 2 months and the results are impressive.

Our setup:
- Cloud: Azure
- Team size: 18 engineers
- Deployment frequency: 80/day

Key findings:
1. Cost anomalies caught automatically
2. Team productivity up significantly
3. Integrates well with existing tools

Happy to answer questions about our implementation!


 
Posted : 27/08/2025 4:52 pm
(@benjamin.campbell266)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Here are some operational tips that worked for uss we've developed: Monitoring - CloudWatch with custom metrics. Alerting - PagerDuty with intelligent routing. Documentation - Confluence with templates. Training - monthly lunch and learns. These have helped us maintain fast deployments while still moving fast on new features.

I'd recommend checking out relevant blog posts for more details.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 31/08/2025 5:03 pm
(@donald.stewart436)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Playing devil's advocate here on the team structure. In our environment, we found that Istio, Linkerd, and Envoy worked better because observability is not optional - you can't improve what you can't measure. That said, context matters a lot - what works for us might not work for everyone. The key is to experiment and measure.

For context, we're using Terraform, AWS CDK, and CloudFormation.

One thing I wish I knew earlier: security must be built in from the start, not bolted on later. Would have saved us a lot of time.


 
Posted : 02/09/2025 12:10 am
(@alex_kubernetes)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Here's what operations has taught uss we've developed: Monitoring - Prometheus with Grafana dashboards. Alerting - PagerDuty with intelligent routing. Documentation - Notion for team wikis. Training - certification programs. These have helped us maintain high reliability while still moving fast on new features.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Additionally, we found that cross-team collaboration is essential for success.


 
Posted : 12/09/2025 12:38 pm
(@matthew.ramos738)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

So relatable! Our experience was that we learned: Phase 1 (1 month) involved assessment and planning. Phase 2 (1 month) focused on team training. Phase 3 (ongoing) was all about knowledge sharing. Total investment was $100K but the payback period was only 3 months. Key success factors: good tooling, training, patience. If I could do it again, I would invest more in training.

I'd recommend checking out conference talks on YouTube for more details.

Additionally, we found that failure modes should be designed for, not discovered in production.


 
Posted : 21/09/2025 8:46 am
(@evelyn.lewis664)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great job documenting all of this! I have a few questions: 1) How did you handle scaling? 2) What was your approach to canary? 3) Did you encounter any issues with compliance? We're considering a similar implementation and would love to learn from your experience.

For context, we're using Datadog, PagerDuty, and Slack.

For context, we're using Terraform, AWS CDK, and CloudFormation.

Additionally, we found that observability is not optional - you can't improve what you can't measure.

One more thing worth mentioning: we had to iterate several times before finding the right balance.


 
Posted : 21/09/2025 9:12 am
(@maria.james115)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We chose a different path here using Kubernetes, Helm, ArgoCD, and Prometheus. The main reason was observability is not optional - you can't improve what you can't measure. However, I can see how your method would be better for larger teams. Have you considered feature flags for gradual rollouts?

One more thing worth mentioning: integration with existing tools was smoother than anticipated.

The end result was 70% reduction in incident MTTR.

The end result was 99.9% availability, up from 99.5%.


 
Posted : 22/09/2025 11:50 am
(@emily.gutierrez57)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

From the ops trenches, here's our takes we've developed: Monitoring - CloudWatch with custom metrics. Alerting - custom Slack integration. Documentation - Confluence with templates. Training - pairing sessions. These have helped us maintain high reliability while still moving fast on new features.

One more thing worth mentioning: we underestimated the training time needed but it was worth the investment.

One thing I wish I knew earlier: documentation debt is as dangerous as technical debt. Would have saved us a lot of time.


 
Posted : 23/09/2025 4:14 am
(@michelle.gutierrez269)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We tackled this from a different angle using Istio, Linkerd, and Envoy. The main reason was the human side of change management is often harder than the technical implementation. However, I can see how your method would be better for legacy environments. Have you considered feature flags for gradual rollouts?

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

For context, we're using Elasticsearch, Fluentd, and Kibana.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 24/09/2025 1:06 am
(@linda.foster79)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

I respect this view, but want to offer another perspective on the tooling choice. In our environment, we found that Terraform, AWS CDK, and CloudFormation worked better because failure modes should be designed for, not discovered in production. That said, context matters a lot - what works for us might not work for everyone. The key is to start small and iterate.

I'd recommend checking out conference talks on YouTube for more details.

Additionally, we found that observability is not optional - you can't improve what you can't measure.


 
Posted : 29/09/2025 5:02 am
(@joan.hill519)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We went a different direction on this using Terraform, AWS CDK, and CloudFormation. The main reason was failure modes should be designed for, not discovered in production. However, I can see how your method would be better for larger teams. Have you considered compliance scanning in the CI pipeline?

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.

One thing I wish I knew earlier: cross-team collaboration is essential for success. Would have saved us a lot of time.


 
Posted : 01/10/2025 6:47 pm
(@christina.gutierrez3)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

On the technical front, several aspects deserve attention. First, compliance requirements. Second, backup procedures. Third, cost optimization. We spent significant time on automation and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 2x improvement.

One thing I wish I knew earlier: cross-team collaboration is essential for success. Would have saved us a lot of time.

I'd recommend checking out the community forums for more details.


 
Posted : 13/10/2025 3:41 am
(@benjamin.campbell266)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

This is a really thorough analysis! I have a few questions: 1) How did you handle testing? 2) What was your approach to blue-green? 3) Did you encounter any issues with compliance? We're considering a similar implementation and would love to learn from your experience.

Additionally, we found that observability is not optional - you can't improve what you can't measure.

The end result was 40% cost savings on infrastructure.

One more thing worth mentioning: we had to iterate several times before finding the right balance.


 
Posted : 15/10/2025 12:40 pm
(@christina.gutierrez3)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Solid analysis! From our perspective, team dynamics. We learned this the hard way when the hardest part was getting buy-in from stakeholders outside engineering. Now we always make sure to monitor proactively. It's added maybe 30 minutes to our process but prevents a lot of headaches down the line.

The end result was 3x increase in deployment frequency.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

For context, we're using Istio, Linkerd, and Envoy.


 
Posted : 16/10/2025 2:51 pm
(@matthew.ramos738)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Practical advice from our team: 1) Document as you go 2) Monitor proactively 3) Practice incident response 4) Build for failure. Common mistakes to avoid: ignoring security. Resources that helped us: Phoenix Project. The most important thing is consistency over perfection.

For context, we're using Vault, AWS KMS, and SOPS.

For context, we're using Jenkins, GitHub Actions, and Docker.

One more thing worth mentioning: the hardest part was getting buy-in from stakeholders outside engineering.


 
Posted : 20/10/2025 6:38 pm
Page 1 / 2
Share:
Scroll to Top