Reconciling Eventually-Consistent Data with CRDTs

2nd December 2013 in London at Kings Place

There are 47 other SkillsCasts available from Scala eXchange 2013

Please log in to watch this conference skillscast.

456896519 640

Would you like to learn how to reconcile eventually consistent data? Join Noel's talk and learn how to do this through conflict-free (sometimes commutative or convergent) replicated data types (CRDTs). In this talk Noel will describe the foundations of CRDTs, give some examples of known data types in Scala, and discuss issues that arise in practice. You will learn how to create straightforward CRDT counters, as well as how to create CRDTs for complex data types such as sets.

Eventually consistent data is common in modern web applications. We all know about eventually consistent data stores, popularised by Dynamo and Cassandra, but there are other sources of eventually consistent data. Mobile applications can go offline, but that doesn't stop people from using them, requiring synchronising data when they come back online. For robustness and to reduce network latency, applications may be hosted in multiple data centres connected by relatively high latency connections. It's infeasible to maintain strong consistency between these connections, again requiring synchronisation at regular intervals. How do we reconcile eventually consistent data?

One way is to use data types that we are guaranteed to merge without conflict. Then we easily reconcile all the different replicas in our system with a simple merge operation that we know cannot fail. These data types are known as conflict-free (sometimes commutative or convergent) replicated data types (CRDTs).

It seems straightforward to create, say, CRDT counters, but can we create CRDTs for complex data types such as sets? It turns out we can. There are wide variety of CRDTs available.


Thanks to our sponsors

Reconciling Eventually-Consistent Data with CRDTs

Noel Welsh

Noel is a partner at Underscore, a global Scala consultancy, where he uses his expertise to develop simple and scalable systems for clients. His main professional interests are programming languages and machine learning.