Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
Deep dive: Implemen...
 
Notifications
Clear all

Deep dive: Implementing AIOps for intelligent incident management

7 Posts
7 Users
0 Reactions
251 Views
(@christopher.mitchell35)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#231]

Our parallel implementation in our organization and can confirm the benefits. One thing we added was feature flags for gradual rollouts. The key insight for us was understanding that security must be built in from the start, not bolted on later. We also found that we had to iterate several times before finding the right balance. Happy to share more details if anyone is interested.

For context, we're using Elasticsearch, Fluentd, and Kibana.

For context, we're using Vault, AWS KMS, and SOPS.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One more thing worth mentioning: unexpected benefits included better developer experience and faster onboarding.

I'd recommend checking out conference talks on YouTube for more details.


 
Posted : 18/04/2025 9:21 pm
(@alexander.rodriguez755)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

The technical specifics of our implementation. Architecture: serverless with Lambda. Tools used: Elasticsearch, Fluentd, and Kibana. Configuration highlights: GitOps with ArgoCD apps. Performance benchmarks showed 3x throughput improvement. Security considerations: container scanning in CI. We documented everything in our internal wiki - happy to share snippets if helpful.

The end result was 70% reduction in incident MTTR.

One more thing worth mentioning: we discovered several hidden dependencies during the migration.


 
Posted : 19/04/2025 3:46 pm
(@rachel.price769)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

This is almost identical to what we faced. The problem: security vulnerabilities. Our initial approach was manual intervention but that didn't work because too error-prone. What actually worked: automated rollback based on error rate thresholds. The key insight was failure modes should be designed for, not discovered in production. Now we're able to detect issues early.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.

For context, we're using Datadog, PagerDuty, and Slack.

The end result was 60% improvement in developer productivity.

One more thing worth mentioning: we discovered several hidden dependencies during the migration.

For context, we're using Grafana, Loki, and Tempo.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Additionally, we found that cross-team collaboration is essential for success.

For context, we're using Jenkins, GitHub Actions, and Docker.

Additionally, we found that the human side of change management is often harder than the technical implementation.


 
Posted : 21/04/2025 12:01 pm
(@matthew.ross327)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great post! We've been doing this for about 14 months now and the results have been impressive. Our main learning was that failure modes should be designed for, not discovered in production. We also discovered that integration with existing tools was smoother than anticipated. For anyone starting out, I'd recommend feature flags for gradual rollouts.

The end result was 80% reduction in security vulnerabilities.

One more thing worth mentioning: we underestimated the training time needed but it was worth the investment.


 
Posted : 21/04/2025 9:11 pm
(@gregory.davis565)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Looking at the engineering side, there are some things to keep in mind. First, network topology. Second, monitoring coverage. Third, cost optimization. We spent significant time on automation and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 2x improvement.

For context, we're using Vault, AWS KMS, and SOPS.

One thing I wish I knew earlier: starting small and iterating is more effective than big-bang transformations. Would have saved us a lot of time.

I'd recommend checking out the community forums for more details.

I'd recommend checking out relevant blog posts for more details.

I'd recommend checking out the community forums for more details.

I'd recommend checking out the official documentation for more details.

For context, we're using Terraform, AWS CDK, and CloudFormation.

One more thing worth mentioning: team morale improved significantly once the manual toil was automated away.

For context, we're using Jenkins, GitHub Actions, and Docker.


 
Posted : 22/04/2025 8:16 pm
(@sharon.garcia321)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

What a comprehensive overview! I have a few questions: 1) How did you handle authentication? 2) What was your approach to backup? 3) Did you encounter any issues with costs? We're considering a similar implementation and would love to learn from your experience.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

For context, we're using Vault, AWS KMS, and SOPS.

One thing I wish I knew earlier: observability is not optional - you can't improve what you can't measure. Would have saved us a lot of time.


 
Posted : 23/04/2025 7:32 pm
(@evelyn.lewis664)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

From an implementation perspective, here are the key points. First, network topology. Second, monitoring coverage. Third, security hardening. We spent significant time on testing and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 2x improvement.

One more thing worth mentioning: we discovered several hidden dependencies during the migration.

One thing I wish I knew earlier: failure modes should be designed for, not discovered in production. Would have saved us a lot of time.


 
Posted : 23/04/2025 9:53 pm
Share:
Scroll to Top