Super Charging Cyanite

4th December 2014 in London at Skills Matter

There are 19 other SkillsCasts available from Clojure eXchange 2014

Please log in to watch this conference skillscast.

499147833 640

Join me for a rapid journey through making metrics move rapidly. The medium in question for these metrics being Cyanite, an open source Clojure project that replaces Graphites default storage engine, Carbon, with a combination of Cassandra and ElasticSearch. We’ll look at the overall data flow, before looking at how it got dialled up to 11 using Clojure’s core.async, custom channels, changing database clients and modifying libraries to suit our needs. Expect code, graphs, words and, in the end, lots of metrics!

The session will walk through the performance enhancements i’ve added to the Cyanite project over the last few weeks. The initial conversation can be found on github:

The final pull request and sorting out can be seen here:

That main focus of the talk will be putting Clojure’s core.async library to work. The following points will form the basis of the talk:

  • Showing the impact of different go channel creation styles.
  • Keep them to a minimum.
  • The custom timer buffer i developed, with a look at the initial version that contained a subtle memory leak.
  • Highlight the ease with which you can add new implementations.
  • Visualising the multi core utilisation of using go blocks in this fashion.
  • The importance of making all external calls asynchronous.

The last point will lead into the work done to the database clients:

  • Modifying ClojureWerks Elastisch client use http-kit instead of the standard http-client.
  • Performance findings from using the Cassandra client Alia with the Hayt library and the simple solution to those problems.
  • Underlying all these points will be the methodology of performance tuning and measuring instead of guessing about performance.

The talk will aim to provide some concrete tips for this kind of work and to show, rather than tell, what the impact of using these non-blocking techniques can be.


Thanks to our sponsors

Super Charging Cyanite

Tom Coupland

Tom Coupland is a developer at MixRadio Bristol. Having spent years working with transaction processing monoliths, producing one of his own in the process, he’s now enjoying the world of SOA and the freedoms it can bring. After subjective eons working with Java, he set out to find more productive and elegant tools.