Ulrqjojvcdg1uxhldfdl
SkillsCast

Tesser: Another Level of Indirection

4th December 2014 in London at Skills Matter

There are 18 other SkillsCasts available from Clojure eXchange 2014

Please log in to watch this conference skillscast.

499384866 640

Clojure's sequence library and the threading macro make lazy sequence operations like map, filter, and reduce composable, and their immutable semantics allow efficient re-use of intermediate results. Core.reducers combine multiple map, filter, takes, et al into a single *fold*, taking advantage of stream fusion--and in the upcoming Clojure 1.7, transducers abstract away the underlying collection entirely.

I've been working on concurrent folds, where we sacrifice some order in exchange for parallelism. Tesser generalizes reducers to a two-dimensional fold: concurrent reductions over independent chunks of a sequence, and a second reduction over those values. Higher-order fold combinators allow us to build up faceted data structures which compute many properties of a dataset in a single pass. The same fold can be run efficiently on multicore systems or transparently distributed--e.g. over Hadoop.

YOU MAY ALSO LIKE:

Thanks to our sponsors

Tesser: Another Level of Indirection

Kyle Kingsbury

Kyle Kingsbury, a.k.a "Aphyr", is a computer safety researcher working as an independent consultant. He is the author of the Riemann monitoring system, the Clojure from the Ground Up introduction to programming, and the Jepsen series on distributed systems correctness. He grills databases in the American Midwest.

SkillsCast

Please log in to watch this conference skillscast.

499384866 640

Clojure's sequence library and the threading macro make lazy sequence operations like map, filter, and reduce composable, and their immutable semantics allow efficient re-use of intermediate results. Core.reducers combine multiple map, filter, takes, et al into a single *fold*, taking advantage of stream fusion--and in the upcoming Clojure 1.7, transducers abstract away the underlying collection entirely.

I've been working on concurrent folds, where we sacrifice some order in exchange for parallelism. Tesser generalizes reducers to a two-dimensional fold: concurrent reductions over independent chunks of a sequence, and a second reduction over those values. Higher-order fold combinators allow us to build up faceted data structures which compute many properties of a dataset in a single pass. The same fold can be run efficiently on multicore systems or transparently distributed--e.g. over Hadoop.

YOU MAY ALSO LIKE:

Thanks to our sponsors

About the Speaker

Tesser: Another Level of Indirection

Kyle Kingsbury

Kyle Kingsbury, a.k.a "Aphyr", is a computer safety researcher working as an independent consultant. He is the author of the Riemann monitoring system, the Clojure from the Ground Up introduction to programming, and the Jepsen series on distributed systems correctness. He grills databases in the American Midwest.