Please log in to watch this conference skillscast.

Clojure has always been good at manipulating data. With the release of spec and Onyx (“a masterless, cloud scale, fault tolerant, high performance distributed computation system”) good became best. In this talk Simon will walk you through a streaming data layer architecture build around Kafka and Onyx that is self-describing, declarative, scalable and convenient to work with for the end user. The focus will be on the power and elegance of describing data and computation with data; and the inferences and automations that can be built on top of that.

The three main lessons you will learn are:

1) the goto data layer infrastructure build around Kafka, Onyx and aggressive usage of materialized views, with emphasis on how to build a system that requires relatively little effort upfront but can grow with one's needs.

2) The problem of managing data and a case for declarative and queryable data descriptions. How these can be used as the basis for automatic materialized view inference where specialized views and data crossings are inferred from raw incoming data or other views based on a combination of heuristics, statistical analysis (seasonality, outlier removal, ...) and predefined ontologies. Doing so is a practical way to maintain a large number of views, increasing their availability and abstracting the complexity into declarative rules, rather than having an ETL pipeline with dozens or even hundreds of hand crafted tasks.

3) how and why Clojure is a natural choice for tasks that involve a lot of data manipulation, touching both on functional programming and lisp-specifics such as code-is-data.

YOU MAY ALSO LIKE:

• The Subtle Art of Recommendation (Algorithms) (SkillsCast recorded in July 2019)
• Automating Elaborate-Transform-Load for Busy Data Scientists (SkillsCast recorded in October 2019)
• Using Kubeflow Pipelines for building machine learning pipelines (SkillsCast recorded in September 2019)