Count-Min Sketch in Real Data Applications
Please log in to watch this conference skillscast.
In this talk Laura will briefly explain several probabilistic data structures used for approximate query answering, such as a Bloom Filter, HyperLogLog and Count-Min Sketch, as well as present canonical examples of use at Twitter for each of the structures mentioned. You will learn how these structures are implemented in Twitter’s open-source abstract algebra library Algebird and how they are used by Summingbird - an open-source domain-specific language (implemented in Scala) that integrates online and batch MapReduce computations in a single framework.
Queries such as whether a user has seen a tweet, the number of unique users who have favourited a tweet, or the number of times a tweet was seen over a period of time, are all hard to answer exactly due to having huge datasets and tight time constraints. However, the exact number is often not necessary as long as the errors are bounded. Certain algebraic data structures enable approximate answers to the queries mentioned, owing to distributed computations performed over them that are guaranteed to be correct because of their algebraic properties.
Take a look at Algebird and Summingbird.
YOU MAY ALSO LIKE:
- Real-time Systems with Spark Streaming and Kafka (in London on 30th September - 1st October 2019)
- Damjan Vujnovic's Advanced JavaScript Workshop (in London on 4th - 6th November 2019)
- Scala eXchange London 2019 (in London on 12th - 13th December 2019)
- Scalax2gether Community Day 2019 (in London on 14th December 2019)
- How Open Source is Funded & Privacy Engineering for the World of Kafka (in London on 24th September 2019)
- Keynote by Zainab Ali on Abstract Data Types In The Region Of Abysmal Pain, And How To Navigate Them (in London on 30th September 2019)
- The Last Frontier and Beyond (SkillsCast recorded in August 2019)
- Why API Design Matters, and Why Yours Sucks! (and mine sucks too!) (SkillsCast recorded in June 2019)
Thanks to our sponsors
Count-Min Sketch in Real Data Applications
Laura Bledaite
Laura is a data scientist at Twitter with a taste for functional programming. She mostly writes Scala, though she really admires O'Caml as well. Laura knows a bit about recommender systems, statistics and math. She did her BSc in Financial and Actuarial Mathematics in Vilnius University, where with several friends she wrote a thesis about The Role of Tail Index in Analysis of Currency Returns.
Please log in to watch this conference skillscast.
In this talk Laura will briefly explain several probabilistic data structures used for approximate query answering, such as a Bloom Filter, HyperLogLog and Count-Min Sketch, as well as present canonical examples of use at Twitter for each of the structures mentioned. You will learn how these structures are implemented in Twitter’s open-source abstract algebra library Algebird and how they are used by Summingbird - an open-source domain-specific language (implemented in Scala) that integrates online and batch MapReduce computations in a single framework.
Queries such as whether a user has seen a tweet, the number of unique users who have favourited a tweet, or the number of times a tweet was seen over a period of time, are all hard to answer exactly due to having huge datasets and tight time constraints. However, the exact number is often not necessary as long as the errors are bounded. Certain algebraic data structures enable approximate answers to the queries mentioned, owing to distributed computations performed over them that are guaranteed to be correct because of their algebraic properties.
Take a look at Algebird and Summingbird.
YOU MAY ALSO LIKE:
- Real-time Systems with Spark Streaming and Kafka (in London on 30th September - 1st October 2019)
- Damjan Vujnovic's Advanced JavaScript Workshop (in London on 4th - 6th November 2019)
- Scala eXchange London 2019 (in London on 12th - 13th December 2019)
- Scalax2gether Community Day 2019 (in London on 14th December 2019)
- How Open Source is Funded & Privacy Engineering for the World of Kafka (in London on 24th September 2019)
- Keynote by Zainab Ali on Abstract Data Types In The Region Of Abysmal Pain, And How To Navigate Them (in London on 30th September 2019)
- The Last Frontier and Beyond (SkillsCast recorded in August 2019)
- Why API Design Matters, and Why Yours Sucks! (and mine sucks too!) (SkillsCast recorded in June 2019)
Thanks to our sponsors
Count-Min Sketch in Real Data Applications
Laura Bledaite
Laura is a data scientist at Twitter with a taste for functional programming. She mostly writes Scala, though she really admires O'Caml as well. Laura knows a bit about recommender systems, statistics and math. She did her BSc in Financial and Actuarial Mathematics in Vilnius University, where with several friends she wrote a thesis about The Role of Tail Index in Analysis of Currency Returns.