Simplifying systems with Elixir
Please log in to watch this conference skillscast.
Elixir is often described as a language which offers great support for massive concurrency. First-hand reports cite the ease of handling millions of connected users, sub-millisecond response times, and superb fault-tolerance. These are all great benefits, but we’re left wondering whether Elixir is useful only for large scale systems, or can it bring some benefits in the simpler cases too?
This talk aims to demonstrate that Elixir is also a great choice for building smaller systems. Through a very simple but still a real-life example, I’ll explain how using Elixir can help simplify the system architecture, and lead to a more homogeneous solution. The talk targets backend developers who are new to Elixir. After the talk, the audience will have a clearer idea about what makes Elixir attractive, and why should they consider using it to build their next backend system.
Q&A
Some Links:
https://github.com/sasa1977/erlangelist/#running-the-site-locally
The library for Let's Encrypt certification in Elixir: https://github.com/sasa1977/site_encrypt
Question: Do you have a process for migrating a system like you described to something with high availability? Or in your minimalist approach where do you start when the system needs to be more robust to failure in the underlying infrastructure?
Answer: There are many options, and I prefer to start as simple and as lightweight as possible. For example, for some basic fault-tolerance WRT to machine failure, it's enough to have some failover mechanism, which can be as simple as using heroku, or otherwise something like k8s running a singleton version of the system.
Obviously this is not going to suffice in all scenarios, but it's also a simple enough start which is going to be enough in some scenarios. Further increase of availability will require various techniques depending on the particular case.
Question: Have you run elixir in a clustered setup in production? I think a lot of people (us included) run phoenix behind a regular load balancer and a shared nothing architecture, which means you sadly cannot really use the awesome process model very much since non-clustered processes cannot talk to each other which is a bit of a shame.
Answer: The distributed BEAM is easy to setup, though in my view dealing with a distributed state has many challenges, so I like to avoid it as much as I can. That said, the abstractions in beam ecosystem are progressing nicely, and I'm particularly excited about the ra library (https://github.com/rabbitmq/ra). I feel that this could be a good building block for obtaining more of k8s-like service directly in beam.
In addition, I've started some experiments on a native Elixir load balancer (basic TCP passthrough). In general, I'd like it to be easy to build a small-to-medium distributed stateful fault-tolerant system with SQL persistence using nothing but Elixir (or any other beam language).
Question: i’m curious to learn some of the challenges you’ve faced dealing with a distributed setup.
Answer: Consistently replicating state is IMO the most difficult challenge, and the ecosystem used to be somewhat lacking in that regard. At some point I fell back to using a database as the source of truth as much as possible, because it's simpler to reason about. That said, the story definitely seems to be improving, with abstractions such as Phoenix Tracker, and the aforementioned ra library.
Question: I know Akka and Microsoft Orleans have had considerable success running distributed actors in complex systems, maybe there’s a few things Elixir/Erlang can learn from them in return (wink)
Answer: Yeah, I believe there's definitely potential to take some ideas from these techs. There is erleans (https://github.com/erleans/erleans) which aims to bring some ideas from Orleans to beam.
Question: I personally really like your "Just in Time"/DIY approach! But many times on the job I had trouble defending a homemade solution against a traditional one. Sometimes I was taken as "Afraid of the technology" or something like that. Do you have this similar problem? How do you overcome the complexity addiction of our industry?
Answer: Yeah, I've had similar experiences, and sometimes I find it difficult to argue. I feel that as an industry we became too dependent on out-of-the-box available solutions. I found that the most effective way to battle this attitude is to build a small incomplete demo, and try to explain the gains we can get if we roll our own solution.
To be clear, I'm certainly not opposed to using 3rd party products or libs, but I like to be a bit more critical and ask myself is it solving a real difficult problem for me, or can I get the similar kind of properties if I write a bit of code, and can this hand-made code actually simplify things for me. Sometimes the answer is yes, and other times it's no.
Question: Have you ever worn any serious pain because you've gone with an erlang native solution and needing to back pedal to something mainstream and non-native? I mean, that's always going to happen now and then, but any significant war story?
Answer: To be honest, can't think of a single case. In general, if a simple solution takes me where I want to be, and performs fine for some amount of time (e.g. a few months or more), then I think it was worth it even if I have to fallback to something else. To me it's just incremental/evolving system architecture. Start simple, move to more complicated when there's real need.
Question: Simplicity paying off is something that I can buy for sure as long as the system is observable enough to understand even though it's unique and unfamiliar (compared to using COTS devops things). Being able to prototype and move fast especially at the start of a product makes or breaks things. This feels like it grows out very nicely if you ever outgrow the mvp. That graph of the supervision tree in your blog is way more understandable than any devops thing I've ever seen. So it seems like you win a lot at the observability.
Answer: Yeah, I find it very intuitive to reason about supervision tree, and most importantly, it's one thing you learn and apply it at all levels of granularity.
Question: Are you using boundary (https://github.com/sasa1977/boundary) in production / commercial team setting?
Answer: I've added it experimentally to a small project of my client. Had no problems so far, but the project is still quite small to provide a reliable assessment. Boundary is more about in-app deps, though it has some support for cross-app deps too, like e.g. you can prevent calling Ecto from web modules, or calling Phoenix from contexts. You can even allow Mix to be used only at compile-time. The point is that boundary will emit a compile-time warning, whereas with releases using mix at runtime will break at runtime (so only after you deploy it).
Question: Did you have any real case number about how much load you had to migrate to a 3rd party solution? Of course it depends a lot on the details of the project, but just to have an idea of when it is time to change. Which are the metrics that lead you to make the change? Which 3rd party solution was that?
Answer: Not really. To be clear, in the past 10 years of using beam languages, I never had a need to migrate away from a hand-made solution. Admittedly, I didn't work on systems the size of Netflix, Twitter & such, it was definitely smaller scale in comparison, but still I believe that this is a practical testament that we can go really far without using this heavy 3rd party machinery.
YOU MAY ALSO LIKE:
Simplifying systems with Elixir
Saša Jurić
MentorIndependent