Keynote: The Magic Behind Spark
Please log in to watch this conference skillscast.
Apache Spark is one of the most popular general purpose distributed systems, and has driven a lot of growth in the Scala community. This talk will look at the magic which makes Spark work, peeling back the curtain to revel the several hundred gnomes that secretly power most distributed systems.
In essence parts of this talk could be considered “why spark is built the way it is, why its not perfect, and how to work around our mistakes." It’s not all doom and gloom though, we will explore the new APIs and the exciting new things we can do with them with a brief detour into how to work around some of the trade-offs in the new APIs – but mostly focused on the new exciting shiny things we can play with.
A basic background with Apache Spark will probably make the talk more exciting, or depressing depending on your point of view, but for those new to Apache Spark just enough to understand whats going will be covered at the start. The presenter would of course encourage you to buy and read her books on the topic (“Learning Spark” & “High Performance Spark”), because which presenter doesn’t do that.
Even if distributed systems aren't your jam, there will be pictures of cats, gnomes, and maybe even a panda to keep things exciting. Also learning how systems like Spark have been designed and evolved can be useful to avoid our mistakes (or make you feel better about your own mistakes).
YOU MAY ALSO LIKE:
- Validating Big Data Jobs - Stopping Failures before Production (w/ Spark, BEAM, & friends!) (SkillsCast recorded in December 2018)
- Lightbend Akka for Scala - Professional (in London on 11th - 12th November 2019)
- Advanced Scala with Dick Wall (in London on 9th - 11th December 2019)
- Scala eXchange London 2019 (in London on 12th - 13th December 2019)
- Scalax2gether Community Day 2019 (in London on 14th December 2019)
- Code Kata: Yilin Wei - Optics with Monocle (in London on 22nd October 2019)
- Keynote by Ruben Bridgewater on Performance Optimisations and the Possible Savings Doing So (in London on 30th October 2019)
- Going Multicloud with Serverless (SkillsCast recorded in October 2019)
- Abstract Data Types In The Region Of Abysmal Pain, And How To Navigate Them (SkillsCast recorded in September 2019)
Thanks to our sponsors
Keynote: The Magic Behind Spark
Holden Karau
Holden is a transgender Canadian open source developer advocate @ Google with a focus on Apache Spark, BEAM, and related "big data" tools. She is the co-author of Learning Spark, High Performance Spark, and another Spark book that's a bit more out of date. She is a committer and PMC on Apache Spark and committer on SystemML & Mahout projects. She was tricked into the world of big data while trying to improve search and recommendation systems and has long since forgotten her original goal.