Waterpark: Distributed Actors vs the Pandemic

Please log in to watch this conference skillscast.

At HCA Healthcare we own and operate 185 hospitals, and around 2,000 surgery centers, freestanding emergency rooms, and clinics. Our scale breaks many vendor products. In 2018 we began building “Waterpark”, which would become the company’s next-gen integration platform (one-part streaming system, one-part distributed database). We chose the ErlangVM and Elixir for fault-tolerance and productivity. When COVID-19 arrived, an interesting tech story became something more serious; more essential.

In this talk we will discuss * how the actor model maps so well onto healthcare * the virtues of wheel reinvention * how a big, Fortune 100 company can be more fun and innovative than a startup * how we transitioned from proof-of-concept mode to continuously available (no planned or unplanned downtime) in three weeks * how we use Erlang's hot code loading (for real) in a production cluster spread across four data centers * how the wisdom of "Papers We Love" is indistinguishable from magic, * how my hero and friend Joe Armstrong (from beyond the grave) helped make American nursing home residents safer during COVID-19

Specifically, we will discuss process pairs, long-lived digital twins, no-masters, bloom filters, rendezvous hashing, our recipes for continuous availability, location transparency, open sourcing our HL7 libraries, and using the bit-syntax for clinical data.