Maximize your Spark: Applying the Lambda Architecture with Spark/Spark Streaming
The Lambda Architecture (LA), being fault-tolerant against hardware failures and human errors, enables developers to build large-scale, distributed data processing systems in a flexible and extensible manner. In this talk Think Reactive co-founder Deenar Toraskar will walk you through building a stream analytics engine using Spark and the Lambda architecture.
The talk will cover building all three layers using Spark, each coming with its own set of requirements: i) the BATCH layer, managing the master dataset (an immutable, append-only set of raw data) and pre-computing batch views, ii) the SERVING layer, indexing batch views so that they can be queried in a low-latency, ad-hoc way, and iii) the SPEED layer, dealing with recent data only, and compensating for the high latency of the batch layer. The talk would be accompanied by a real world example with code and a live demo.
A tutorial style talk, this event will be valuable for developers, architects, or project leads who already know about Spark and are now looking for more insight into how it can be leveraged to implement real-world applications.
You might also be interested in the following course :
Typesafe's Apache Spark: An Introductory Workshop For Developers - September 10-11, 2015
YOU MAY ALSO LIKE:
- Introduction to Apache Spark (SkillsCast recorded in December 2015)
- ScalaCon 2022: Opening Keynote Networking Party (in London on 4th October 2022)
- ScalaCon 2022 (Online Conference on 4th - 8th October 2022)
- LJC: I Started Testing In Production... Then I Went On Holiday (Online Meetup on 15th August 2022)
- Real-time Stream Processing in Spring Made Easy (in London on 25th August 2022)
- A History of Enterprise Monads (SkillsCast recorded in May 2021)
- Connecting the dots - building and structuring a functional application in Scala (SkillsCast recorded in May 2021)
Thank you to our sponsors and partners
Platinum
Maximize your Spark: Applying the Lambda Architecture with Spark/Spark Streaming
Deenar Toraskar
Deenar Toraskar is the co-founder of Think Reactive, which provides Spark based responsive, resilient, elastic and ready-to-go data analytics solutions. The solution is built using state-of-the-art technology, end-to-end; from ETL and data pipelines (both batch and streaming), persistence adapters, to analysis and algorithms. All components are packaged using Docker and can run on bare metal or any cloud.