Please log in to watch this conference skillscast.
Over the past couple of years, Scala has become a go-to language for building data processing applications, as evidenced by the emerging ecosystem of frameworks and tools including LinkedIn's Kafka, Twitter's Scalding and our own Snowplow project (https://github.com/snowplow/snowplow).
In this talk, Alex will draw on his experiences at Snowplow to explore how to build rock-sold data pipelines in Scala, highlighting a range of techniques including:
- Translating the Unix stdin/out/err pattern to stream processing
- ""Railway oriented"" programming using the Scalaz Validation
- Validating data structures with JSON Schema
- Visualizing event stream processing errors in ElasticSearch
YOU MAY ALSO LIKE:
- Building data processing applications in Scala: the Snowplow experience (SkillsCast recorded in March 2014)
- Functional Concurrency in .NET with C# and F# (in London on 9th - 10th September 2019)
- Fast Track to F# with Tomas Petricek & Phil Trelford (in London on 9th - 10th September 2019)
- Haskell eXchange 2019 (in London on 10th - 11th October 2019)
- Clojure eXchange 2019 (in London on 2nd - 3rd December 2019)
- Hands-on: Fractal art with Fable and WebGL (in London on 20th June 2019)
- Keynote by Dick Wall on Why API Design Matters, and Why Yours Sucks! (and mine sucks too!) (in London on 24th June 2019)
- The Elements of Tagless Final Style (SkillsCast recorded in May 2019)
- Scala 2.13 and Beyond! (SkillsCast recorded in April 2019)
Building robust data pipelines in Scala
I'm the co-founder and tech lead at Snowplow Analytics, the open source web and event analytics platform (https://github.com/snowplow/snowplow). Snowplow is almost exclusively written in Scala, using a range of technologies including Scalaz, Scalding and Spray. I spend a lot of time working with distributed systems (historically Hadoop, increasingly Kinesis, Kafka et al) to deliver really scalable event stream processing. I'm also the author of Unified Log Processing from Manning Publications (http://manning.com/dean/).