Working with Large Numbers of Non-Trivial ETL Pipelines

Please log in to watch this conference skillscast.

Data pipelines need to be flexible, modular and easily monitored. They are not just set-and-forget. The team that monitors a pipeline might not have developed it and may not be experts on the dataset. End users must have confidence in the output.

This talk is a practical walkthrough of a suggested pipeline architecture on AWS using Step functions, Spot instance, AWS Batch, Glue, Lambda and Data Dog.

I'll be covering techniques using AWS and DataDog, but many of the approaches are applicable in an Apache Airflow/Kibana environment.

YOU MAY ALSO LIKE:

• Inside Every Calculus Is A Little Algebra Waiting To Get Out (SkillsCast recorded in December 2020)
• Inside Every Calculus Is A Little Algebra Waiting To Get Out (SkillsCast recorded in December 2020)