Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
AI-driven incident ...
 
Notifications
Clear all

AI-driven incident response - our experience with PagerDuty Copilot

19 Posts
17 Users
0 Reactions
480 Views
(@jason.brooks11)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#48]

We've been experimenting with ai-driven incident response - our experience with pagerduty copilot for the past 2 months and the results are impressive.

Our setup:
- Cloud: GCP
- Team size: 47 engineers
- Deployment frequency: 27/day

Key findings:
1. Cost anomalies caught automatically
2. False positives still an issue
3. Impressive accuracy rate

Happy to answer questions about our implementation!


 
Posted : 11/10/2025 11:26 am
(@evelyn.williams270)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Some tips from our journey: 1) Test in production-like environments 2) Monitor proactively 3) Practice incident response 4) Measure what matters. Common mistakes to avoid: over-engineering early. Resources that helped us: Phoenix Project. The most important thing is collaboration over tools.

For context, we're using Grafana, Loki, and Tempo.

I'd recommend checking out conference talks on YouTube for more details.

One more thing worth mentioning: the hardest part was getting buy-in from stakeholders outside engineering.


 
Posted : 16/10/2025 9:02 pm
(@david.morales35)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Valid approach! Though we did it differently using Jenkins, GitHub Actions, and Docker. The main reason was cross-team collaboration is essential for success. However, I can see how your method would be better for legacy environments. Have you considered integration with our incident management system?

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 25/10/2025 11:07 pm
(@joyce.hughes421)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

I'd like to share our complete experience with this. We started about 13 months ago with a small pilot. Initial challenges included team training. The breakthrough came when we streamlined the process. Key metrics improved: 99.9% availability, up from 99.5%. The team's feedback has been overwhelmingly positive, though we still have room for improvement in automation. Lessons learned: automate everything. Next steps for us: optimize costs.

For context, we're using Istio, Linkerd, and Envoy.

One more thing worth mentioning: the initial investment was higher than expected, but the long-term benefits exceeded our projections.


 
Posted : 27/10/2025 9:05 pm
(@victoria.rivera433)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great approach! In our organization and can confirm the benefits. One thing we added was cost allocation tagging for accurate showback. The key insight for us was understanding that starting small and iterating is more effective than big-bang transformations. We also found that team morale improved significantly once the manual toil was automated away. Happy to share more details if anyone is interested.

One more thing worth mentioning: we had to iterate several times before finding the right balance.


 
Posted : 31/10/2025 12:17 pm
(@jose.williams694)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Thoughtful post - though I'd challenge one aspect on the timeline. In our environment, we found that Terraform, AWS CDK, and CloudFormation worked better because starting small and iterating is more effective than big-bang transformations. That said, context matters a lot - what works for us might not work for everyone. The key is to invest in training.

I'd recommend checking out the community forums for more details.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.


 
Posted : 04/11/2025 9:29 am
(@evelyn.lewis664)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

The technical specifics of our implementation. Architecture: serverless with Lambda. Tools used: Terraform, AWS CDK, and CloudFormation. Configuration highlights: CI/CD with GitHub Actions workflows. Performance benchmarks showed 50% latency reduction. Security considerations: container scanning in CI. We documented everything in our internal wiki - happy to share snippets if helpful.

One more thing worth mentioning: we underestimated the training time needed but it was worth the investment.


 
Posted : 07/11/2025 8:58 am
(@michelle.gutierrez269)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

On the technical front, several aspects deserve attention. First, data residency. Second, failover strategy. Third, cost optimization. We spent significant time on testing and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 50% latency reduction.

One more thing worth mentioning: we underestimated the training time needed but it was worth the investment.

The end result was 99.9% availability, up from 99.5%.

The end result was 90% decrease in manual toil.


 
Posted : 09/11/2025 12:12 pm
(@jose.jackson593)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our experience was remarkably similar! We learned: Phase 1 (2 weeks) involved stakeholder alignment. Phase 2 (3 months) focused on pilot implementation. Phase 3 (2 weeks) was all about full rollout. Total investment was $100K but the payback period was only 6 months. Key success factors: good tooling, training, patience. If I could do it again, I would invest more in training.

One thing I wish I knew earlier: observability is not optional - you can't improve what you can't measure. Would have saved us a lot of time.


 
Posted : 13/11/2025 11:07 am
(@stephanie.long568)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great job documenting all of this! I have a few questions: 1) How did you handle security? 2) What was your approach to rollback? 3) Did you encounter any issues with latency? We're considering a similar implementation and would love to learn from your experience.

The end result was 50% reduction in deployment time.

I'd recommend checking out the official documentation for more details.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 13/11/2025 3:43 pm
(@timothy.wood427)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Just dealt with this! Symptoms: high latency. Root cause analysis revealed memory leaks. Fix: fixed the leak. Prevention measures: better monitoring. Total time to resolve was 15 minutes but now we have runbooks and monitoring to catch this early.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One thing I wish I knew earlier: failure modes should be designed for, not discovered in production. Would have saved us a lot of time.

The end result was 60% improvement in developer productivity.


 
Posted : 18/11/2025 6:34 am
(@tyler.foster787)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our parallel implementation in our organization and can confirm the benefits. One thing we added was compliance scanning in the CI pipeline. The key insight for us was understanding that the human side of change management is often harder than the technical implementation. We also found that the initial investment was higher than expected, but the long-term benefits exceeded our projections. Happy to share more details if anyone is interested.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 19/11/2025 11:45 pm
(@dennis.king704)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Playing devil's advocate here on the timeline. In our environment, we found that Vault, AWS KMS, and SOPS worked better because automation should augment human decision-making, not replace it entirely. That said, context matters a lot - what works for us might not work for everyone. The key is to focus on outcomes.

One thing I wish I knew earlier: security must be built in from the start, not bolted on later. Would have saved us a lot of time.

For context, we're using Istio, Linkerd, and Envoy.


 
Posted : 20/11/2025 12:49 pm
(@nancy.howard864)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Here's our full story with this. We started about 14 months ago with a small pilot. Initial challenges included performance issues. The breakthrough came when we streamlined the process. Key metrics improved: 40% cost savings on infrastructure. The team's feedback has been overwhelmingly positive, though we still have room for improvement in testing coverage. Lessons learned: automate everything. Next steps for us: expand to more teams.

The end result was 80% reduction in security vulnerabilities.


 
Posted : 26/11/2025 7:07 am
(@william.harris811)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our data supports this. We found that the most important factor was failure modes should be designed for, not discovered in production. We initially struggled with security concerns but found that feature flags for gradual rollouts worked well. The ROI has been significant - we've seen 50% improvement.

Additionally, we found that the human side of change management is often harder than the technical implementation.

Additionally, we found that failure modes should be designed for, not discovered in production.


 
Posted : 26/11/2025 9:04 am
Page 1 / 2
Share:
Scroll to Top