Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
Follow-up: Data lak...
 
Notifications
Clear all

Follow-up: Data lake architecture on AWS: S3, Glue, and Athena

19 Posts
17 Users
0 Reactions
116 Views
(@jennifer.young148)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#275]

Great approach! In our organization and can confirm the benefits. One thing we added was chaos engineering tests in staging. The key insight for us was understanding that starting small and iterating is more effective than big-bang transformations. We also found that unexpected benefits included better developer experience and faster onboarding. Happy to share more details if anyone is interested.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.

For context, we're using Istio, Linkerd, and Envoy.

For context, we're using Istio, Linkerd, and Envoy.

For context, we're using Grafana, Loki, and Tempo.

For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.

For context, we're using Elasticsearch, Fluentd, and Kibana.


 
Posted : 13/05/2025 4:21 am
(@scott.allen968)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our end-to-end experience with this. We started about 3 months ago with a small pilot. Initial challenges included tool integration. The breakthrough came when we simplified the architecture. Key metrics improved: 70% reduction in incident MTTR. The team's feedback has been overwhelmingly positive, though we still have room for improvement in testing coverage. Lessons learned: automate everything. Next steps for us: improve documentation.

Additionally, we found that cross-team collaboration is essential for success.


 
Posted : 14/05/2025 7:04 am
(@nicholas.morgan692)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

The full arc of our experience with this. We started about 3 months ago with a small pilot. Initial challenges included team training. The breakthrough came when we streamlined the process. Key metrics improved: 50% reduction in deployment time. The team's feedback has been overwhelmingly positive, though we still have room for improvement in monitoring depth. Lessons learned: communicate often. Next steps for us: optimize costs.

One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.


 
Posted : 15/05/2025 8:28 pm
(@alexander.rodriguez755)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

This is exactly the kind of detail that helps! I have a few questions: 1) How did you handle security? 2) What was your approach to canary? 3) Did you encounter any issues with costs? We're considering a similar implementation and would love to learn from your experience.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

For context, we're using Kubernetes, Helm, ArgoCD, and Prometheus.

Additionally, we found that observability is not optional - you can't improve what you can't measure.


 
Posted : 17/05/2025 8:07 am
(@maria.carter392)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Some guidance based on our experience: 1) Automate everything possible 2) Use feature flags 3) Practice incident response 4) Keep it simple. Common mistakes to avoid: not measuring outcomes. Resources that helped us: Accelerate by DORA. The most important thing is collaboration over tools.

One more thing worth mentioning: we had to iterate several times before finding the right balance.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 19/05/2025 8:22 am
(@aaron.gutierrez941)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Super useful! We're just starting to evaluateg this approach. Could you elaborate on success metrics? Specifically, I'm curious about stakeholder communication. Also, how long did the initial implementation take? Any gotchas we should watch out for?

Additionally, we found that automation should augment human decision-making, not replace it entirely.

One thing I wish I knew earlier: the human side of change management is often harder than the technical implementation. Would have saved us a lot of time.

The end result was 60% improvement in developer productivity.

I'd recommend checking out the community forums for more details.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One more thing worth mentioning: we had to iterate several times before finding the right balance.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

For context, we're using Istio, Linkerd, and Envoy.


 
Posted : 20/05/2025 4:14 pm
(@katherine.edwards302)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great post! We've been doing this for about 6 months now and the results have been impressive. Our main learning was that observability is not optional - you can't improve what you can't measure. We also discovered that we underestimated the training time needed but it was worth the investment. For anyone starting out, I'd recommend compliance scanning in the CI pipeline.

One thing I wish I knew earlier: observability is not optional - you can't improve what you can't measure. Would have saved us a lot of time.


 
Posted : 20/05/2025 6:43 pm
(@kathleen.watson88)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Solid analysis! From our perspective, maintenance burden. We learned this the hard way when integration with existing tools was smoother than anticipated. Now we always make sure to include in design reviews. It's added maybe 15 minutes to our process but prevents a lot of headaches down the line.

I'd recommend checking out the community forums for more details.

Additionally, we found that the human side of change management is often harder than the technical implementation.


 
Posted : 22/05/2025 3:06 am
(@nicholas.morgan692)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great info! We're exploring and evaluating this approach. Could you elaborate on the migration process? Specifically, I'm curious about stakeholder communication. Also, how long did the initial implementation take? Any gotchas we should watch out for?

The end result was 60% improvement in developer productivity.

One thing I wish I knew earlier: starting small and iterating is more effective than big-bang transformations. Would have saved us a lot of time.

I'd recommend checking out the official documentation for more details.


 
Posted : 23/05/2025 9:20 pm
(@benjamin.rivera487)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

I can offer some technical insights from our implementation. Architecture: microservices on Kubernetes. Tools used: Terraform, AWS CDK, and CloudFormation. Configuration highlights: GitOps with ArgoCD apps. Performance benchmarks showed 3x throughput improvement. Security considerations: zero-trust networking. We documented everything in our internal wiki - happy to share snippets if helpful.

One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.


 
Posted : 24/05/2025 2:01 am
(@nicholas.morgan692)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Same here! In practice, the most important factor was failure modes should be designed for, not discovered in production. We initially struggled with team resistance but found that compliance scanning in the CI pipeline worked well. The ROI has been significant - we've seen 50% improvement.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Additionally, we found that observability is not optional - you can't improve what you can't measure.

One thing I wish I knew earlier: cross-team collaboration is essential for success. Would have saved us a lot of time.

The end result was 50% reduction in deployment time.

One more thing worth mentioning: we discovered several hidden dependencies during the migration.

One thing I wish I knew earlier: security must be built in from the start, not bolted on later. Would have saved us a lot of time.

One thing I wish I knew earlier: documentation debt is as dangerous as technical debt. Would have saved us a lot of time.


 
Posted : 25/05/2025 9:10 pm
(@donald.stewart436)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

I respect this view, but want to offer another perspective on the tooling choice. In our environment, we found that Kubernetes, Helm, ArgoCD, and Prometheus worked better because automation should augment human decision-making, not replace it entirely. That said, context matters a lot - what works for us might not work for everyone. The key is to focus on outcomes.

Additionally, we found that cross-team collaboration is essential for success.

The end result was 50% reduction in deployment time.

The end result was 40% cost savings on infrastructure.

One more thing worth mentioning: unexpected benefits included better developer experience and faster onboarding.

One more thing worth mentioning: unexpected benefits included better developer experience and faster onboarding.

Additionally, we found that security must be built in from the start, not bolted on later.

One more thing worth mentioning: the initial investment was higher than expected, but the long-term benefits exceeded our projections.


 
Posted : 27/05/2025 4:59 pm
(@mary.castillo14)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

From a practical standpoint, don't underestimate maintenance burden. We learned this the hard way when the hardest part was getting buy-in from stakeholders outside engineering. Now we always make sure to test regularly. It's added maybe 30 minutes to our process but prevents a lot of headaches down the line.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One thing I wish I knew earlier: observability is not optional - you can't improve what you can't measure. Would have saved us a lot of time.


 
Posted : 29/05/2025 4:33 pm
(@deborah.cook920)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Parallel experiences here. We learned: Phase 1 (2 weeks) involved tool evaluation. Phase 2 (2 months) focused on pilot implementation. Phase 3 (2 weeks) was all about knowledge sharing. Total investment was $100K but the payback period was only 6 months. Key success factors: automation, documentation, feedback loops. If I could do it again, I would set clearer success metrics.

For context, we're using Terraform, AWS CDK, and CloudFormation.

The end result was 80% reduction in security vulnerabilities.


 
Posted : 30/05/2025 8:18 am
(@opsx-tom)
Posts: 76
Member Admin
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Here's the technical breakdown of our implementation. Architecture: microservices on Kubernetes. Tools used: Istio, Linkerd, and Envoy. Configuration highlights: GitOps with ArgoCD apps. Performance benchmarks showed 50% latency reduction. Security considerations: container scanning in CI. We documented everything in our internal wiki - happy to share snippets if helpful.

For context, we're using Terraform, AWS CDK, and CloudFormation.

Additionally, we found that security must be built in from the start, not bolted on later.


 
Posted : 31/05/2025 4:49 am
Page 1 / 2
Share:
Scroll to Top