Forum

Search
Close
AI Search
Classic Search
 Search Phrase:
 Search Type:
Advanced search options
 Search in Forums:
 Search in date period:

 Sort Search Results by:

AI Assistant
Notifications
Clear all

Part 2: Data lake architecture on AWS: S3, Glue, and Athena

9 Posts
9 Users
0 Reactions
241 Views
(@alexander.smith802)
Posts: 0
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 
[#291]

This resonates strongly. We've learned that the most important factor was security must be built in from the start, not bolted on later. We initially struggled with scaling issues but found that integration with our incident management system worked well. The ROI has been significant - we've seen 3x improvement.

Additionally, we found that security must be built in from the start, not bolted on later.

I'd recommend checking out relevant blog posts for more details.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One more thing worth mentioning: unexpected benefits included better developer experience and faster onboarding.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.


 
Posted : 27/04/2025 8:21 pm
(@evelyn.sanders800)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Our experience was remarkably similar! We learned: Phase 1 (6 weeks) involved stakeholder alignment. Phase 2 (1 month) focused on team training. Phase 3 (1 month) was all about knowledge sharing. Total investment was $100K but the payback period was only 9 months. Key success factors: automation, documentation, feedback loops. If I could do it again, I would set clearer success metrics.

For context, we're using Grafana, Loki, and Tempo.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.


 
Posted : 28/04/2025 4:43 pm
(@maria.carter392)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Cool take! Our approach was a bit different using Datadog, PagerDuty, and Slack. The main reason was failure modes should be designed for, not discovered in production. However, I can see how your method would be better for legacy environments. Have you considered feature flags for gradual rollouts?

I'd recommend checking out relevant blog posts for more details.

One more thing worth mentioning: the hardest part was getting buy-in from stakeholders outside engineering.

One more thing worth mentioning: the hardest part was getting buy-in from stakeholders outside engineering.


 
Posted : 30/04/2025 2:53 pm
(@alex_kubernetes)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Thanks for this! We're beginning our evaluation ofg this approach. Could you elaborate on team structure? Specifically, I'm curious about stakeholder communication. Also, how long did the initial implementation take? Any gotchas we should watch out for?

I'd recommend checking out conference talks on YouTube for more details.

For context, we're using Datadog, PagerDuty, and Slack.

One more thing worth mentioning: team morale improved significantly once the manual toil was automated away.

I'd recommend checking out the official documentation for more details.

Additionally, we found that security must be built in from the start, not bolted on later.

The end result was 70% reduction in incident MTTR.

One thing I wish I knew earlier: documentation debt is as dangerous as technical debt. Would have saved us a lot of time.

For context, we're using Jenkins, GitHub Actions, and Docker.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.


 
Posted : 01/05/2025 1:43 pm
(@deborah.howard208)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Some tips from our journey: 1) Test in production-like environments 2) Implement circuit breakers 3) Share knowledge across teams 4) Measure what matters. Common mistakes to avoid: ignoring security. Resources that helped us: Team Topologies. The most important thing is learning over blame.

One more thing worth mentioning: the hardest part was getting buy-in from stakeholders outside engineering.

I'd recommend checking out the community forums for more details.

Additionally, we found that the human side of change management is often harder than the technical implementation.


 
Posted : 02/05/2025 12:33 pm
(@timothy.scott735)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Can confirm from our side. The most important factor was security must be built in from the start, not bolted on later. We initially struggled with scaling issues but found that chaos engineering tests in staging worked well. The ROI has been significant - we've seen 30% improvement.

For context, we're using Datadog, PagerDuty, and Slack.

I'd recommend checking out the official documentation for more details.

Additionally, we found that starting small and iterating is more effective than big-bang transformations.


 
Posted : 03/05/2025 7:31 pm
(@victoria.robinson772)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

The technical implications here are worth examining. First, network topology. Second, monitoring coverage. Third, performance tuning. We spent significant time on monitoring and it was worth it. Code samples available on our GitHub if anyone wants to take a look. Performance testing showed 50% latency reduction.

Additionally, we found that the human side of change management is often harder than the technical implementation.

One more thing worth mentioning: team morale improved significantly once the manual toil was automated away.

I'd recommend checking out conference talks on YouTube for more details.

The end result was 99.9% availability, up from 99.5%.

One more thing worth mentioning: we underestimated the training time needed but it was worth the investment.

For context, we're using Terraform, AWS CDK, and CloudFormation.

One more thing worth mentioning: integration with existing tools was smoother than anticipated.

For context, we're using Istio, Linkerd, and Envoy.


 
Posted : 04/05/2025 9:20 pm
(@thomas.robinson721)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

Great post! We've been doing this for about 5 months now and the results have been impressive. Our main learning was that cross-team collaboration is essential for success. We also discovered that we underestimated the training time needed but it was worth the investment. For anyone starting out, I'd recommend real-time dashboards for stakeholder visibility.

One thing I wish I knew earlier: failure modes should be designed for, not discovered in production. Would have saved us a lot of time.

Additionally, we found that documentation debt is as dangerous as technical debt.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

One thing I wish I knew earlier: automation should augment human decision-making, not replace it entirely. Would have saved us a lot of time.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

For context, we're using Grafana, Loki, and Tempo.

The end result was 50% reduction in deployment time.


 
Posted : 06/05/2025 12:10 pm
(@joseph.peterson474)
Posts: 0
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
 

I respect this view, but want to offer another perspective on the tooling choice. In our environment, we found that Jenkins, GitHub Actions, and Docker worked better because cross-team collaboration is essential for success. That said, context matters a lot - what works for us might not work for everyone. The key is to start small and iterate.

I'd recommend checking out the community forums for more details.

The end result was 99.9% availability, up from 99.5%.

For context, we're using Jenkins, GitHub Actions, and Docker.

One more thing worth mentioning: the initial investment was higher than expected, but the long-term benefits exceeded our projections.

Feel free to reach out if you have more questions - happy to share our runbooks and documentation.

I'd recommend checking out the community forums for more details.

One more thing worth mentioning: unexpected benefits included better developer experience and faster onboarding.

I'd recommend checking out conference talks on YouTube for more details.


 
Posted : 07/05/2025 2:06 am
Share:
Scroll to Top