The Time is Now — Literally — for Real-Time Stream Processing

Avtar Raikmo | Apr 5, 2023

Reacting to data is one thing; reacting to data as quickly as possible — as soon as it’s generated — is quite another. Today, it’s the difference between slaying the competition or being slayed by the competition. 

I saw the power of streaming when I was with Meta. In fact, I joined Hazelcast because I was blown away by how it enables companies that aren’t Meta-size or that don’t have Meta-dollars to take full advantage of real-time stream processing. Just a few years ago, companies didn’t understand what real-time stream processing could do for them, especially if they were in an industry that didn’t historically use real-time data. But now they see the benefits streaming has across an organization, regardless of size or industry, and in consumers’ everyday lives.

Just look at mobile phones today. They have crash detection. They can read from a number of different sensors at the same time and make an assessment that they believe you have been in an accident. Based on anomalous behavior detected by the sensors, your phone dials emergency services. Simultaneously, the phone directs them to your precise location using real-time information (e.g. GPS coordinates) combined with historical context and reference information.

It’s really about combining and comparing what’s known to be “normal” with what’s happening now and using the resulting insight to react. More specifically, the phone reacts by taking action, not in a few minutes or even a few seconds, but instantaneously. That’s what consumers have come to expect and what companies must give them. 

We’ve seen the significant benefits of real-time stream processing and decision-making in the financial arena. For example, institutions have generated and saved many millions of dollars through the ability to offer loans and prevent fraud, respectively.

But finance is just the tip of a very big iceberg. 

From a supply chain perspective, combining transactional, operational and historical data into a single platform benefits every organization.

The pandemic was a really big (OK, gigantic) example of how changes to the supply chain affect organizations and their customers. Still, even the smallest anomaly can affect whether organizations can get what they need so that customers can get what they need (and want). In fact, companies that react immediately and automatically to even the smallest changes in the supply chain not only rise above the rest, but do so in a lean and mean manner. They need fewer people focusing on logistics, which puts them in a totally different competitive category than organizations that require hand-holding when links in the supply chain begin to weaken and — often, without immediate intervention — eventually break.

You can really see the flexibility and potential of real-time stream processing when you start to apply the technology to specific industries. The finance industry has been on the real-time streaming journey for a long time — it was really predisposed to streaming. Now, we’re seeing how the technology can filter into more and more spaces, including (but certainly not limited to):

Energy: Real-time streaming technology can help optimize clean energy generation, such as wind and solar. If you know there is an environmental event somewhere, real-time stream processing can change the profiles of how energy is captured, stored and/or delivered to make it as cost-effective as possible and ensure the energy is delivered where it is needed in the moment — or even ahead of time. 

Healthcare: Healthcare is already changing due to real-time data and wearable technologies. Real-time stream processing can help healthcare providers and consumers be more proactive about health because they can predict that something will happen. On an individual-patient basis, this could mean preventing an hours-long wait in the ER or even a potentially fatal event. On a macro level, healthcare organizations – from hospitals to universities to non-profits – can optimize services to ensure they have the right providers and supplies for a dynamically changing landscape, where emergent healthcare needs can be promptly addressed.

Government: If you think about how governments divert cash into things that matter today, it’s a very manual, very slow process. The use of real-time streaming data could help drive that decision-making. Governments are really good at collecting information, but not so good at reacting to it. It will be a slower uptake, but I predict real-time stream processing capabilities will filter through to the public sector in the next decade.

No matter the industry or application, it’s all about personalizing and enriching reference data with streaming data – and vice versa.

But while all organizations can benefit from real-time stream processing, few today have the resources to enable true real-time decision-making. Not only must relevant data sources be connected, but it also takes a significant amount of compute power and storage space to support the intelligent combining of transactional, operational and historical data. Whether on-premises or in the cloud, Hazelcast provides the technology and support organizations need to deploy fast, highly scalable applications for running large calculations, simulations, and other data- and compute-intensive workloads.

We’re still quite early on the journey. Still, mobile phones, smartwatches and social media are making organizations and consumers more aware of real-time data and decision-making benefits. Soon, they’ll expect nothing less if it hasn’t happened already.

Relevant Resources

View All Resources
About the Author
Avtar Raikmo

Avtar Raikmo

Avtar Raikmo started his career as a developer of Java, Python, C# and C++ solutions before becoming a senior leader at Goldman Sachs, Morningstar and, subsequently Meta (Facebook). Today, he is still just as passionate about technology and motivated by large-scale data challenges, as well as raising two children. Based in the UK, his current position is the Head of Engineering for Hazelcast Platform.