Streaming Data Platforms and Event-Driven Design

Streaming data platforms allow for the processing of continual data records at large volume, high velocity, and with low latency. Event-driven design applies a programming style where the flow of the program is determined by events, which are changes in the state of the system. Events are captured over time (a time-stream) and then strongly typed into an event schema with the associated data.

Three foundational questions are important. First, what are the core principles behind streaming platforms? Second, how is event-driven design applied to build event-driven systems? Third, how do Cap Theorem trade-offs shape the design of streaming systems? Answers draw from experience building and deploying systems for capturing and analysing data from transactions and demographic flows at Twilio and booking data at the Farelogix airline distribution platform.