A data layer in Clojure - Intermediate
Please log in to watch this conference skillscast.
Clojure has always been good at manipulating data. With the release of spec and Onyx (“a masterless, cloud scale, fault tolerant, high performance distributed computation system”) good became best. In this talk Simon will walk you through a streaming data layer architecture build around Kafka and Onyx that is self-describing, declarative, scalable and convenient to work with for the end user. The focus will be on the power and elegance of describing data and computation with data; and the inferences and automations that can be built on top of that.
The three main lessons you will learn are:
1) the goto data layer infrastructure build around Kafka, Onyx and aggressive usage of materialized views, with emphasis on how to build a system that requires relatively little effort upfront but can grow with one's needs.
2) The problem of managing data and a case for declarative and queryable data descriptions. How these can be used as the basis for automatic materialized view inference where specialized views and data crossings are inferred from raw incoming data or other views based on a combination of heuristics, statistical analysis (seasonality, outlier removal, ...) and predefined ontologies. Doing so is a practical way to maintain a large number of views, increasing their availability and abstracting the complexity into declarative rules, rather than having an ETL pipeline with dozens or even hundreds of hand crafted tasks.
3) how and why Clojure is a natural choice for tasks that involve a lot of data manipulation, touching both on functional programming and lisp-specifics such as code-is-data.
YOU MAY ALSO LIKE:
- The Subtle Art of Recommendation (Algorithms) (SkillsCast recorded in July 2019)
- Real-time Systems with Spark Streaming and Kafka (in London on 30th September - 1st October 2019)
- Leonardo De Marchi's Deep Learning Fundamentals (in London on 22nd - 23rd October 2019)
- Keynote by Noelia Jiménez Martínez and Phil Harvey on Data: A Guide To Humans (in London on 16th September 2019)
- A look inside the Data Scientist Playbook (in London on 19th September 2019)
- How real-time data processing is used for application in customer experience? (SkillsCast recorded in July 2019)
- A Look at the technology of LUNO and Adjust (SkillsCast recorded in July 2019)
Thanks to our sponsors
A data layer in Clojure - Intermediate
Simon Belak
Simon built his first computer out of Lego bricks and learned to program soon after. Emergence, networks, modes of thought, limits of language and expression are what makes him smile (and stay up at night). The combination of lisp and machine learning put him on the path of always striving to make himself redundant if not outright obsolete.
Please log in to watch this conference skillscast.
Clojure has always been good at manipulating data. With the release of spec and Onyx (“a masterless, cloud scale, fault tolerant, high performance distributed computation system”) good became best. In this talk Simon will walk you through a streaming data layer architecture build around Kafka and Onyx that is self-describing, declarative, scalable and convenient to work with for the end user. The focus will be on the power and elegance of describing data and computation with data; and the inferences and automations that can be built on top of that.
The three main lessons you will learn are:
1) the goto data layer infrastructure build around Kafka, Onyx and aggressive usage of materialized views, with emphasis on how to build a system that requires relatively little effort upfront but can grow with one's needs.
2) The problem of managing data and a case for declarative and queryable data descriptions. How these can be used as the basis for automatic materialized view inference where specialized views and data crossings are inferred from raw incoming data or other views based on a combination of heuristics, statistical analysis (seasonality, outlier removal, ...) and predefined ontologies. Doing so is a practical way to maintain a large number of views, increasing their availability and abstracting the complexity into declarative rules, rather than having an ETL pipeline with dozens or even hundreds of hand crafted tasks.
3) how and why Clojure is a natural choice for tasks that involve a lot of data manipulation, touching both on functional programming and lisp-specifics such as code-is-data.
YOU MAY ALSO LIKE:
- The Subtle Art of Recommendation (Algorithms) (SkillsCast recorded in July 2019)
- Real-time Systems with Spark Streaming and Kafka (in London on 30th September - 1st October 2019)
- Leonardo De Marchi's Deep Learning Fundamentals (in London on 22nd - 23rd October 2019)
- Keynote by Noelia Jiménez Martínez and Phil Harvey on Data: A Guide To Humans (in London on 16th September 2019)
- A look inside the Data Scientist Playbook (in London on 19th September 2019)
- How real-time data processing is used for application in customer experience? (SkillsCast recorded in July 2019)
- A Look at the technology of LUNO and Adjust (SkillsCast recorded in July 2019)
Thanks to our sponsors
A data layer in Clojure - Intermediate
Simon Belak
Simon built his first computer out of Lego bricks and learned to program soon after. Emergence, networks, modes of thought, limits of language and expression are what makes him smile (and stay up at night). The combination of lisp and machine learning put him on the path of always striving to make himself redundant if not outright obsolete.