Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
Follow-up: MLOps: B...
 
Notifications
Clear all

Follow-up: MLOps: Building ML pipelines with Kubeflow and MLflow

25 Posts
23 Users
0 Reactions
275 Views
(@maria.jimenez673)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#170]

Our experience was remarkably similar. The problem: deployment failures. Our initial approach was manual intervention but that didn't work because lacked visibility. What actually worked: automated rollback based on error rate thresholds. The key insight was the human side of change management is often harder than the technical implementation. Now we're able to detect issues early.

For context, we're using Elasticsearch, Fluentd, and Kibana.

For context, we're using Elasticsearch, Fluentd, and Kibana.

For context, we're using Grafana, Loki, and Tempo.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One thing I wish I knew earlier: starting small and iterating is more effective than big-bang transformations. Would have saved us a lot of time.


 
Posted : 05/10/2025 6:21 am
(@gregory.brooks453)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Wanted to contribute some real-world operational insights we've developed: Monitoring - CloudWatch with custom metrics. Alerting - custom Slack integration. Documentation - GitBook for public docs. Training - certification programs. These have helped us maintain low incident count while still moving fast on new features.

The end result was 90% decrease in manual toil.

I'd recommend checking out relevant blog posts for more details.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 07/10/2025 12:46 am
(@deborah.howard208)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Love how thorough this explanation is! I have a few questions: 1) How did you handle authentication? 2) What was your approach to backup? 3) Did you encounter any issues with consistency? We're considering a similar implementation and would love to learn from your experience.

For context, we're using Datadog, PagerDuty, and Slack.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.

For context, we're using Grafana, Loki, and Tempo.

For context, we're using Terraform, AWS CDK, and CloudFormation.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Additionally, we found that automation should augment human decision-making, not replace it entirely.

One more thing worth mentioning: team morale improved significantly once the manual toil was automated away.

Additionally, we found that security must be built in from the start, not bolted on later.


 
Posted : 08/10/2025 1:50 am
(@ruth.white53)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Nice! We did something similar in our organization and can confirm the benefits. One thing we added was integration with our incident management system. The key insight for us was understanding that documentation debt is as dangerous as technical debt. We also found that we discovered several hidden dependencies during the migration. Happy to share more details if anyone is interested.

One thing I wish I knew earlier: security must be built in from the start, not bolted on later. Would have saved us a lot of time.


 
Posted : 09/10/2025 7:45 am
(@maria_terraform)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Experienced this firsthand! Symptoms: increased error rates. Root cause analysis revealed connection pool exhaustion. Fix: corrected routing rules. Prevention measures: better monitoring. Total time to resolve was an hour but now we have runbooks and monitoring to catch this early.

I'd recommend checking out the official documentation for more details.

The end result was 99.9% availability, up from 99.5%.

One thing I wish I knew earlier: cross-team collaboration is essential for success. Would have saved us a lot of time.


 
Posted : 10/10/2025 12:44 pm
(@john.long261)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great post! We've been doing this for about 24 months now and the results have been impressive. Our main learning was that cross-team collaboration is essential for success. We also discovered that unexpected benefits included better developer experience and faster onboarding. For anyone starting out, I'd recommend automated rollback based on error rate thresholds.

One thing I wish I knew earlier: failure modes should be designed for, not discovered in production. Would have saved us a lot of time.

One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.

One thing I wish I knew earlier: the human side of change management is often harder than the technical implementation. Would have saved us a lot of time.

I'd recommend checking out the community forums for more details.

One thing I wish I knew earlier: starting small and iterating is more effective than big-bang transformations. Would have saved us a lot of time.


 
Posted : 12/10/2025 1:32 am
(@opsx-tom)
Posts: 76
Member Admin
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Let me tell you how we approached this. We started about 24 months ago with a small pilot. Initial challenges included performance issues. The breakthrough came when we improved observability. Key metrics improved: 90% decrease in manual toil. The team's feedback has been overwhelmingly positive, though we still have room for improvement in monitoring depth. Lessons learned: start simple. Next steps for us: optimize costs.

I'd recommend checking out relevant blog posts for more details.

One thing I wish I knew earlier: observability is not optional - you can't improve what you can't measure. Would have saved us a lot of time.


 
Posted : 13/10/2025 10:20 am
(@joyce.hughes421)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We faced this too! Symptoms: increased error rates. Root cause analysis revealed memory leaks. Fix: fixed the leak. Prevention measures: better monitoring. Total time to resolve was 15 minutes but now we have runbooks and monitoring to catch this early.

The end result was 40% cost savings on infrastructure.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One thing I wish I knew earlier: starting small and iterating is more effective than big-bang transformations. Would have saved us a lot of time.


 
Posted : 15/10/2025 7:17 am
(@gregory.davis565)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Diving into the technical details, we should consider. First, network topology. Second, backup procedures. Third, cost optimization. We spent significant time on automation and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 10x throughput increase.

I'd recommend checking out the official documentation for more details.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 16/10/2025 5:06 am
(@jason.brooks11)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our end-to-end experience with this. We started about 9 months ago with a small pilot. Initial challenges included performance issues. The breakthrough came when we automated the testing. Key metrics improved: 70% reduction in incident MTTR. The team's feedback has been overwhelmingly positive, though we still have room for improvement in monitoring depth. Lessons learned: start simple. Next steps for us: improve documentation.

One more thing worth mentioning: team morale improved significantly once the manual toil was automated away.


 
Posted : 16/10/2025 2:52 pm
(@gregory.davis565)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Practical advice from our team: 1) Automate everything possible 2) Monitor proactively 3) Share knowledge across teams 4) Measure what matters. Common mistakes to avoid: over-engineering early. Resources that helped us: Google SRE book. The most important thing is outcomes over outputs.

I'd recommend checking out relevant blog posts for more details.

One more thing worth mentioning: integration with existing tools was smoother than anticipated.

I'd recommend checking out the community forums for more details.


 
Posted : 18/10/2025 6:21 am
(@tyler.foster787)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We hit this same wall a few months back. The problem: deployment failures. Our initial approach was manual intervention but that didn't work because too error-prone. What actually worked: chaos engineering tests in staging. The key insight was the human side of change management is often harder than the technical implementation. Now we're able to deploy with confidence.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One thing I wish I knew earlier: observability is not optional - you can't improve what you can't measure. Would have saved us a lot of time.

Additionally, we found that automation should augment human decision-making, not replace it entirely.

Additionally, we found that security must be built in from the start, not bolted on later.

For context, we're using Terraform, AWS CDK, and CloudFormation.

For context, we're using Vault, AWS KMS, and SOPS.


 
Posted : 19/10/2025 6:11 am
(@mark.murphy761)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

I'd like to share our complete experience with this. We started about 10 months ago with a small pilot. Initial challenges included team training. The breakthrough came when we automated the testing. Key metrics improved: 80% reduction in security vulnerabilities. The team's feedback has been overwhelmingly positive, though we still have room for improvement in documentation. Lessons learned: start simple. Next steps for us: add more automation.

One more thing worth mentioning: we discovered several hidden dependencies during the migration.

For context, we're using Datadog, PagerDuty, and Slack.

For context, we're using Vault, AWS KMS, and SOPS.

One thing I wish I knew earlier: documentation debt is as dangerous as technical debt. Would have saved us a lot of time.

The end result was 3x increase in deployment frequency.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Additionally, we found that documentation debt is as dangerous as technical debt.


 
Posted : 21/10/2025 4:51 am
(@donald.white940)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Here's our full story with this. We started about 22 months ago with a small pilot. Initial challenges included legacy compatibility. The breakthrough came when we improved observability. Key metrics improved: 3x increase in deployment frequency. The team's feedback has been overwhelmingly positive, though we still have room for improvement in testing coverage. Lessons learned: measure everything. Next steps for us: optimize costs.

Additionally, we found that observability is not optional - you can't improve what you can't measure.


 
Posted : 22/10/2025 11:18 am
(@emily.gutierrez57)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

So relatable! Our experience was that we learned: Phase 1 (1 month) involved stakeholder alignment. Phase 2 (1 month) focused on team training. Phase 3 (1 month) was all about optimization. Total investment was $50K but the payback period was only 6 months. Key success factors: good tooling, training, patience. If I could do it again, I would start with better documentation.

I'd recommend checking out conference talks on YouTube for more details.

Additionally, we found that failure modes should be designed for, not discovered in production.


 
Posted : 23/10/2025 10:49 am
Page 1 / 2
Share:
Scroll to Top