Companies need a data-processing solution that increases the speed of business agility, not one that is complicated by too many technology requirements. This requires a system that delivers continuous/real-time data-processing capabilities for the new business reality.
Stream processing is a hot topic right now, especially for any organization looking to provide insights faster. But what does it mean for users of Java applications, microservices, and in-memory computing?
In this webinar, we will cover the evolution of stream processing and in-memory related to big data technologies and why it is the logical next step for in-memory processing projects.
Setting up servers and configuring software can get in the way of the problems you are trying to solve. With Hazelcast Cloud we take all of those pain points away.
Watch this webinar to learn how you can instantly fire up and then work with Hazelcast Cloud from anywhere in the world. With our auto-generated client stubs for Java, Go, Node.js, Python and .NET, we can have you connected and coding in less than a minute!
Headquartered in Houston, TX, SigmaStream develops systems for energy companies, specifically in the oil and gas industries, to collect, persist, transport and analyze a broad range of operational data. SigmaStream specializes in the capture and analysis of high-frequency data streams generated in the drilling process.
Drilling for Data to Deliver Highly Complex, Streaming KPIs
A drilling rig is one of the most recognizable symbols of the oil and gas industries. The operating costs of a drilling rig are very high and any downtime throughout the drilling process can have a significant effect on the rig operator’s bottom line. The rigs are equipped with a large number of sensors to detect small vibrations during the drilling process. SigmaStream can gather, monitor and analyze real-time high-frequency data, and uses this to feed analytic systems that enable operators to immediately act on the streaming data to prevent costly, catastrophic failures in the drilling process.
The data generated from these sensors is for machine consumption; at any time during rig operations, up to 60 to 70 channels of high-frequency data enters the system at various frequencies. With no system in place to process, store and analyze the data, many critical events went unnoticed or even recorded.
Facing a large amount of “data exhaust,” which is discarded data, the founders of SigmaStream recognized an opportunity to develop and provide an event-based technology solution that addressed a pervasive and expensive problem for drill operators.
After contracting with its first customer, SigmaStream quickly realized that acquiring the data and making it transportable was insufficient. To address this challenge and offer a complete solution unavailable from any other company or source, SigmaStream integrated its operational platform with Hazelcast Jet. Analytics are now part of the process, which morphed into a data-gathering and a complete closed-loop control automation platform. SigmaStream integrates Jet with a visualization tool called Hummingbird, which is capable of analyzing and displaying high-frequency data.
One of its clients was using IBM Streams, which was not a financially feasible solution for SigmaStream, so they searched for an alternative streaming engine and discovered Hazelcast. They evaluated Hazelcast Jet and were impressed by its relative simplicity and ease of use, as well as with the light footprint of the product.
“My team and I were attracted to Hazelcast because it is lightweight and easy to integrate,” said Koduru. “It was also more affordable compared to other products. We installed it, tried it and very quickly decided Hazelcast Jet was the solution SigmaStream and our clients needed.”
SigmaStream has been a Hazelcast Jet customer since 2017. The highly time-sensitive data stream entering SigmaStream’s analytics engine and the associated insights are stored in multiple NoSQL databases, and the analytic engines are homebuilt R and Python.
With Hazelcast Jet, SigmaStream can record or sample high-frequency data or events that occur, and then apply its
proprietary algorithms to make very fine-tuned adjustments to the drilling process. An example of this is the realtime adjustment of the RPM of the drilling string and bit; this type of drilling must be managed at a very exacting level to prevent equipment failure and costly delays to the drilling process.
Typically, operational stakeholders, backed by their technical teams, contact SigmaStream to help extract more
information from their existing data. Previously, there wasn’t a pre-existing solution from an operational KPI perspective. The combination of SigmaStream and Hazelcast Jet technologies allows for the streaming of highly complex KPIs.Additionally, SigmaStream provides its clients with dashboarding for real-time data visualization and interpretation.
The operational and business benefits are in being able to gather, monitor, analyze and apply delicate tweaks to drilling parameters in near real-time, allowing clients to manage physical resources better through high-frequency
feedback on a per-well basis. They are often able to reduce the drilling time by as much as 20 percent, from a typical 15 days to 12 days.
The most significant benefit for SigmaStream clients is the creation of streaming KPIs using Hazelcast Jet with its operation stack. By doing so, SigmaStream is reducing the amount of time to drill a well from spud to the true vertical depth or TVD, potentially saving its clients millions of dollars.
According to Koduru, the company is planning to expand its customer base to other industries that would benefit from the capabilities of SigmaStream and Hazelcast Jet to gather, monitor and analyze high-frequency streaming data.
“We are focused on using artificial intelligence and machine learning with Hazelcast Jet. This has the potential to be a big opportunity for SigmaStream because of the many rapidly spinning parts in these industries generating
considerable amounts of high-frequency data that we can bring insight to” said Koduru.
While this particular use case revolves around streaming high-frequency data for oil and gas drilling, the capability delivered by a high-speed in-memory stream processing engine is applicable across a broad array of industries and applications.
Whether you're interested in learning the basics of in-memory systems, or you're looking for advanced, real-world production examples and best practices, we've got you covered.