Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
Part 2: Best practi...
 
Notifications
Clear all

Part 2: Best practices for Kubernetes pod security in production

15 Posts
13 Users
0 Reactions
112 Views
(@benjamin.rivera487)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#145]

We took a similar route in our organization and can confirm the benefits. One thing we added was feature flags for gradual rollouts. The key insight for us was understanding that security must be built in from the start, not bolted on later. We also found that unexpected benefits included better developer experience and faster onboarding. Happy to share more details if anyone is interested.

I'd recommend checking out conference talks on YouTube for more details.

One more thing worth mentioning: the initial investment was higher than expected, but the long-term benefits exceeded our projections.

The end result was 90% decrease in manual toil.

One more thing worth mentioning: unexpected benefits included better developer experience and faster onboarding.

Additionally, we found that documentation debt is as dangerous as technical debt.


 
Posted : 20/12/2024 4:21 am
(@brandon.williams519)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Couldn't agree more. From our work, the most important factor was failure modes should be designed for, not discovered in production. We initially struggled with performance bottlenecks but found that drift detection with automated remediation worked well. The ROI has been significant - we've seen 50% improvement.

One thing I wish I knew earlier: observability is not optional - you can't improve what you can't measure. Would have saved us a lot of time.

For context, we're using Jenkins, GitHub Actions, and Docker.

For context, we're using Datadog, PagerDuty, and Slack.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One more thing worth mentioning: the initial investment was higher than expected, but the long-term benefits exceeded our projections.

For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.

Additionally, we found that automation should augment human decision-making, not replace it entirely.


 
Posted : 21/12/2024 3:21 am
(@katherine.edwards302)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Here's our full story with this. We started about 13 months ago with a small pilot. Initial challenges included legacy compatibility. The breakthrough came when we simplified the architecture. Key metrics improved: 90% decrease in manual toil. The team's feedback has been overwhelmingly positive, though we still have room for improvement in testing coverage. Lessons learned: communicate often. Next steps for us: optimize costs.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 21/12/2024 6:38 am
(@maria_terraform)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

So relatable! Our experience was that we learned: Phase 1 (1 month) involved assessment and planning. Phase 2 (3 months) focused on pilot implementation. Phase 3 (2 weeks) was all about knowledge sharing. Total investment was $50K but the payback period was only 6 months. Key success factors: automation, documentation, feedback loops. If I could do it again, I would involve operations earlier.

For context, we're using Terraform, AWS CDK, and CloudFormation.

Additionally, we found that documentation debt is as dangerous as technical debt.


 
Posted : 21/12/2024 1:16 pm
(@christopher.bennett288)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

While this is well-reasoned, I see things differently on the tooling choice. In our environment, we found that Jenkins, GitHub Actions, and Docker worked better because automation should augment human decision-making, not replace it entirely. That said, context matters a lot - what works for us might not work for everyone. The key is to invest in training.

I'd recommend checking out conference talks on YouTube for more details.

Additionally, we found that observability is not optional - you can't improve what you can't measure.


 
Posted : 22/12/2024 11:04 pm
(@jeffrey.price491)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We created a similar solution in our organization and can confirm the benefits. One thing we added was chaos engineering tests in staging. The key insight for us was understanding that observability is not optional - you can't improve what you can't measure. We also found that unexpected benefits included better developer experience and faster onboarding. Happy to share more details if anyone is interested.

For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.


 
Posted : 23/12/2024 5:34 pm
(@christine.roberts720)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our recommended approach: 1) Test in production-like environments 2) Use feature flags 3) Review and iterate 4) Measure what matters. Common mistakes to avoid: skipping documentation. Resources that helped us: Team Topologies. The most important thing is outcomes over outputs.

One thing I wish I knew earlier: documentation debt is as dangerous as technical debt. Would have saved us a lot of time.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.

For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.

Additionally, we found that failure modes should be designed for, not discovered in production.

For context, we're using Terraform, AWS CDK, and CloudFormation.

For context, we're using Vault, AWS KMS, and SOPS.

One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.

One thing I wish I knew earlier: security must be built in from the start, not bolted on later. Would have saved us a lot of time.


 
Posted : 24/12/2024 1:03 am
(@gregory.brooks453)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Excellent thread! One consideration often overlooked is security considerations. We learned this the hard way when team morale improved significantly once the manual toil was automated away. Now we always make sure to test regularly. It's added maybe 30 minutes to our process but prevents a lot of headaches down the line.

I'd recommend checking out relevant blog posts for more details.

One more thing worth mentioning: unexpected benefits included better developer experience and faster onboarding.


 
Posted : 24/12/2024 3:05 pm
(@benjamin.rivera487)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

There are several engineering considerations worth noting. First, compliance requirements. Second, backup procedures. Third, performance tuning. We spent significant time on monitoring and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 10x throughput increase.

One more thing worth mentioning: the hardest part was getting buy-in from stakeholders outside engineering.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 24/12/2024 6:45 pm
(@victoria.robinson772)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Solid work putting this together! I have a few questions: 1) How did you handle testing? 2) What was your approach to canary? 3) Did you encounter any issues with compliance? We're considering a similar implementation and would love to learn from your experience.

Additionally, we found that automation should augment human decision-making, not replace it entirely.

I'd recommend checking out the community forums for more details.

Additionally, we found that documentation debt is as dangerous as technical debt.


 
Posted : 25/12/2024 2:08 pm
(@deborah.howard208)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

We hit this same wall a few months back. The problem: deployment failures. Our initial approach was simple scripts but that didn't work because lacked visibility. What actually worked: real-time dashboards for stakeholder visibility. The key insight was cross-team collaboration is essential for success. Now we're able to deploy with confidence.

I'd recommend checking out conference talks on YouTube for more details.

One more thing worth mentioning: we had to iterate several times before finding the right balance.


 
Posted : 26/12/2024 9:02 pm
(@nancy.howard864)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Couldn't agree more. From our work, the most important factor was failure modes should be designed for, not discovered in production. We initially struggled with legacy integration but found that automated rollback based on error rate thresholds worked well. The ROI has been significant - we've seen 70% improvement.

Additionally, we found that security must be built in from the start, not bolted on later.

One more thing worth mentioning: the initial investment was higher than expected, but the long-term benefits exceeded our projections.


 
Posted : 27/12/2024 3:12 am
(@jose.williams694)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Technically speaking, a few key factors come into play. First, data residency. Second, backup procedures. Third, performance tuning. We spent significant time on automation and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 2x improvement.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 28/12/2024 7:23 am
(@jeffrey.price491)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Thanks for this! We're beginning our evaluation ofg this approach. Could you elaborate on the migration process? Specifically, I'm curious about stakeholder communication. Also, how long did the initial implementation take? Any gotchas we should watch out for?

The end result was 40% cost savings on infrastructure.

I'd recommend checking out relevant blog posts for more details.

I'd recommend checking out relevant blog posts for more details.

One more thing worth mentioning: the initial investment was higher than expected, but the long-term benefits exceeded our projections.


 
Posted : 30/12/2024 1:40 am
(@victoria.rivera433)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Been there with this one! Symptoms: increased error rates. Root cause analysis revealed memory leaks. Fix: fixed the leak. Prevention measures: load testing. Total time to resolve was 15 minutes but now we have runbooks and monitoring to catch this early.

The end result was 90% decrease in manual toil.

Additionally, we found that the human side of change management is often harder than the technical implementation.

One thing I wish I knew earlier: failure modes should be designed for, not discovered in production. Would have saved us a lot of time.


 
Posted : 01/01/2025 2:02 am
Share:
Scroll to Top