Islands in the Stream - What country music can teach us about event driven systems

Please log in to watch this conference skillscast.

Event driven systems are all the rage. It's with good reason we're witnessing a transformation with businesses adopting event driven systems. Building systems around an event-driven architecture is powerful pattern for creating awesome data intensive applications.  But before we sail away to another world, let's avoid the common pitfalls of designing & running event driven systems.

Islands in the Stream - what Kenny Rogers can teach us about event driven systems from the wisdom of a country music classic

Q&A

Question: One question when we use the event driven process, how can we guarantee the response time and maintain the state. for example with synchronous way we call an api, we know it success or not in a sec. in the asynchronous way we don't know when we can get the result, do you have any idea or suggestion how to solve this?

Answer: Great question. Building EDA systems with some known SLA constraint I reckon is easier than traditional microservices arch as you have isolation for the consumers. Something like a bunch of Kafka consumers can be load balanced, and scaled pretty independently. When an actual SLA outcome is required you can build a back-pressure system - but it gets a bit complex.

Question: I remember going to town when I discovered the Observer pattern around 2002 and trying to create and event driven engine model. Super hard at the time, but I suspect a whole lot easier with todays tools.

Answer: Thanks. Yeah - a lot of great frameworks exist now that manage the hard “distributed” problems so you can concentrate on just solving the unique (business) problems. I’m a huge fan of k-streams and flink - but these problems are much easier to solve these days for sure!

Question: . A question about when things go wrong — if a producer produces erroneous events for a while, what’s the pattern for telling consumers of that event topic to ignore those events?

Answer: There’s a pattern called a “poison pill” when you really need to discard a message logically from immutable storage. This can also take the form of a new-day offset if you just want to scrub history. In Kafka this takes the form of a consumer offset.

Download Slides as PDF

YOU MAY ALSO LIKE:

• Which Plane Woke Snowy the Cat? (SkillsCast recorded in May 2019)
• Apache Pulsar and the Streaming Ecosystem (SkillsCast recorded in May 2021)
• Data Mesh; A principled introduction (SkillsCast recorded in May 2021)