Reconciling Eventually-Consistent Data with CRDTs

2nd December 2013 in London at Kings Place

There are 47 other SkillsCasts available from Scala eXchange 2013

Please log in to watch this conference skillscast.

456896519 640

Would you like to learn how to reconcile eventually consistent data? Join Noel's talk and learn how to do this through conflict-free (sometimes commutative or convergent) replicated data types (CRDTs). In this talk Noel will describe the foundations of CRDTs, give some examples of known data types in Scala, and discuss issues that arise in practice. You will learn how to create straightforward CRDT counters, as well as how to create CRDTs for complex data types such as sets.

Eventually consistent data is common in modern web applications. We all know about eventually consistent data stores, popularised by Dynamo and Cassandra, but there are other sources of eventually consistent data. Mobile applications can go offline, but that doesn't stop people from using them, requiring synchronising data when they come back online. For robustness and to reduce network latency, applications may be hosted in multiple data centres connected by relatively high latency connections. It's infeasible to maintain strong consistency between these connections, again requiring synchronisation at regular intervals. How do we reconcile eventually consistent data?

One way is to use data types that we are guaranteed to merge without conflict. Then we easily reconcile all the different replicas in our system with a simple merge operation that we know cannot fail. These data types are known as conflict-free (sometimes commutative or convergent) replicated data types (CRDTs).

It seems straightforward to create, say, CRDT counters, but can we create CRDTs for complex data types such as sets? It turns out we can. There are wide variety of CRDTs available.


Thanks to our sponsors

Reconciling Eventually-Consistent Data with CRDTs

Noel Welsh

Noel has over fifteen years experience in software architecture and development, and over a decade in machine learning and data mining. Examples of the projects he's been involved with include one of the first commercial products to apply machine learning to the Internet (eventually acquired by Omniture), a BAFTA award winning website, and a custom CMS used daily by thousands of students. Noel is an active writer, presenter, and open source contributor. Noel has a PhD in machine learning from the University of Birmingham.