Keynote: The Magic Behind Spark

Please log in to watch this conference skillscast.

In essence parts of this talk could be considered “why spark is built the way it is, why its not perfect, and how to work around our mistakes." It’s not all doom and gloom though, we will explore the new APIs and the exciting new things we can do with them with a brief detour into how to work around some of the trade-offs in the new APIs – but mostly focused on the new exciting shiny things we can play with.

A basic background with Apache Spark will probably make the talk more exciting, or depressing depending on your point of view, but for those new to Apache Spark just enough to understand whats going will be covered at the start. The presenter would of course encourage you to buy and read her books on the topic (“Learning Spark” & “High Performance Spark”), because which presenter doesn’t do that.

Even if distributed systems aren't your jam, there will be pictures of cats, gnomes, and maybe even a panda to keep things exciting. Also learning how systems like Spark have been designed and evolved can be useful to avoid our mistakes (or make you feel better about your own mistakes).

YOU MAY ALSO LIKE:

• Distributed pandas – long promised, finally sort of (SkillsCast recorded in June 2022)
• Taming the Context Beast (SkillsCast recorded in October 2022)
• The Middle Way for Static Typing in Spark DataFrames (SkillsCast recorded in October 2022)