The Rise of Real Time

Editor's Notes

1. Really excited to be here.
2. People at this conference are part of something big. Won’t be quite Woodstock—it’s one day of Kafka and stream processing not 3 days of love, and I’m not sure what our CoC says about LSD. But nonetheless there is a big change happening in the world—there is something happening here—and I think we’re really lucky to be here at the beginning of it.
3. The heart of it is that what business are is changing. A business used to be something that was built out of paper and people.
4. Now a business is something that is equally made out of data and software (with lot’s of great people doing the hard bits!) This is really different. So what does a fully digital business look like?
5. What does a fully digital business look like? We’re still in the process of really learning how to build one. One of the nice things about my job is I get to talk to people who have built this kind of fully digital company, lot’s of them here in silicon valley, but also I get to talk to people who are in the process of business that were originally built out of people and paper and which are transitioning.
6. The early versions of this process mirrored what humans did. You get data systems replace the file cabinets as a storage mechanism, and computer programs that replace some of the human processes. But early on it is piecemeal and grounded in the constraints and thought patterns of what came before—more a horseless carriage than—than a modern car.
7. It will be no surprise to anyone in this room that I think a big part of what this natively digital business looks like is about streams. Data on paper is inherently static, but that limitation is long gone and in a fully digital world an event that occurs in one part of an organization needs to be available everywhere immediately. Piles => Streams Processing and reacting to these streams is at the heart of what businesses do.
8. You can take a concrete example of a type of business we all know—Retail. Sales and shipments aren’t a static thing, they happen all the time, and reacting to these streams is at the heart of what a retail company does. This is by no means unique to retail: maybe 30-50% of what companies do falls into this domain of reacting to streams of events.
9. This is very much why we built Apache Kafka. We had a very particular vision for what a company would look like if you reimagined it’s use of data around streams of events.
10. The vision that inspired Kafka was this idea of a streaming platform. A central place where every part of the organization could publish the stream of data it had, and that would let every other thing
11. 0.9
12. Core: Data pipeline Venture bet: Stream processing
13. Kafka has been around the longest and only at 0.9! Storm - Windowing - Small State Spark Streaming 2.0 - Flink - RocksDB-based state - Strong guarantees - True streaming
14. The vision that inspired Kafka was this idea of a streaming platform. A central place where every part of the organization could publish the stream of data it had, and that would let every other thing