Live from QCon London: Streaming in a World of Legacy Applications

There are common themes when people describe their reasons for rearchitecting legacy business applications, at a technical level: Speed & Scalability. At a business level: The need to gain new insights flowing from an increasing stream of data. These legacy applications commonly centre around some central datastore such as a relational database.  Moving away from this architecture requires massive migration effort. The costs and risks associated with such an effort can sometimes be prohibitive for business owners, you can’t just rip out your relational database.

A lower risk, gradual transition to a target architecture often wins the day.  It turns out that Streaming, Caching & CDC technologies are vital tools for this journey. CDC (Change Data Capture) can turn your legacy data stores into streaming sources.  Modern caching technologies can host data in a way that provides speed and scalability, and finally streaming acts as the glue that can drive new uses cases as well as bridging the old.

By the end of this talk, you’ll understand how to employ these technologies (with concrete examples and demos) over a legacy architecture and also be able to reason about the trade-offs involved.