Building robust data pipelines in Scala
Please log in to watch this conference skillscast.
In this talk, Alex will draw on his experiences at Snowplow to explore how to build rock-sold data pipelines in Scala, highlighting a range of techniques including:
- Translating the Unix stdin/out/err pattern to stream processing
- ""Railway oriented"" programming using the Scalaz Validation
- Validating data structures with JSON Schema
- Visualizing event stream processing errors in ElasticSearch
Alex's talk will draw on his experiences working with event streams in Scala over the last two and a half years at Snowplow, and by Alex's recent work penning Unified Log Processing, a Manning book.
YOU MAY ALSO LIKE:
Thank you to our sponsors and partners
Platinum
Gold
Silver
Bronze
Building robust data pipelines in Scala
Alex Dean
I'm the co-founder and tech lead at Snowplow Analytics, the open source web and event analytics platform (https://github.com/snowplow/snowplow). Snowplow is almost exclusively written in Scala, using a range of technologies including Scalaz, Scalding and Spray. I spend a lot of time working with distributed systems (historically Hadoop, increasingly Kinesis, Kafka et al) to deliver really scalable event stream processing. I'm also the author of Unified Log Processing from Manning Publications (http://manning.com/dean/).