The Hazelcast Platform: Taking off to the next level

This blog continues our series of Hazelcast Platform 5.X product releases, which are focused on enabling enterprises to build real-time applications that instantly take action on data-in-motion by leveraging:

• Stream processing capabilities combined with a real-time fast data store
• Streamlined data integration experience for batch and streaming data sources
• SQL-based creation, management, and monitoring of streaming data pipelines
• Handle larger data sets by expanding storage beyond memory to disks

Activate the countdown clock and prepare to launch Hazelcast Platform 5.3…

T-6 hours and counting…

Akin to loading the rocket’s external tanks with liquid hydrogen and liquid oxygen propellants, bounded (batch) data and unbounded (streaming) data can be loaded into the Hazelcast Platform.  These out-of-the-box connectors provide an easy and efficient way to access data sources (wherever they are, including files, messaging systems, databases, data structures, etc…) and support multiple data formats (text, CSV, JSON, and Avro).

Now with Hazelcast Platform 5.3, we continue adding to the industry’s most robust unified real-time data platform. The JDBC connector provides a zero code and configuration-driven capability to connect to any JDBC-compliant external data stores. Also, new data stores can be added dynamically at runtime.

SQL over JDBC can map the table in external data stores to relevant data structures in the Hazelcast Platform and perform CRUD operations. This connector currently supports MySQL, PostgreSQL, and MongoDB. The release also improves the change data capture (CDC) connector based on Debezium, which can turn databases into streaming data sources supporting MongoDB, Oracle, MySQL, PostgreSQL, DB2, amongst others.

Turning our attention to the platform’s data ingestion capabilities, we now support Kafka Connect as a source directly into our platform. With a Kafka Connect source connector, you can reliably import data from an external system supported by Kafka Connect – such as a database, key-value store, search index, or file system – directly into a Hazelcast data pipeline for processing. Over 100 popular platforms, including Neo4j and Couchbase, are supported via this infrastructure improvement (This feature is in beta*).

T-4 hours and counting…

Developer productivity gets a boost with our upgraded support for Hibernate and Spring Boot! Unlock new levels of efficiency and productivity as we Seamlessly integrate Hibernate into your projects and harness its power to simplify database operations.

Upgrades get easier: Advanced tooling and unparalleled support for lightning-fast, risk-free migration between Hazelcast Platform open source (OSS) and enterprise versions, fueled by our data migration assistance (currently in beta). Turbocharge your transition from OSS to Enterprise with our cutting-edge, out-of-the-box (OTB) data migration tools with a frictionless migration experience. During the beta phase, our support team is at the ready to support you, ensuring upgrades with minimal downtime and world-class consultation on the best architectural approaches to ensure your business’s service level agreements (SLAs) are honored. Post beta, we will have self-service modules, where you can self-migrate data for your upgrades or blue-green deployments.

Energize your operations and embrace a seamless transformation, powered by our robust tooling and support. Get ready to accelerate your migration and unleash boundless potential!

T-2 hours and counting…

Main engines start… the Hazelcast Platform, powered by a combination of the proven low-latency data store and a real-time stream processing engine, provides a unified platform for real-time applications. The platform does the heavy lifting of ingesting, processing, storing data, and making it available in one distributed, easy-to-scale, highly available cluster so developers can focus on the business logic.

Not all launches go according to plan. In the event of a failure, we have you covered with new self-healing capabilities! The automated cluster state management for persistence on Kubernetes was enhanced to support the cluster-wide shutdown, rolling restart, and partial member recovery from failures. We have enhanced the experience of using Hazelcast with persistence under Kubernetes by removing requirements for manual interventions and having no data loss. Furthermore, for customers running Hazelcast in embedded mode, we’ve added a Kubernetes client only for the DNS lookup mode to interact with the application(s) within the K8s cluster.

The latest release delivers on the Hazelcast Platform’s capability to store much larger volumes of data while maintaining low-latency access. Tiered Storage, part of our fast data store and an enterprise feature, provides spill-to-disk functionality, eliminating cluster memory size constraints. Storing data across tiers, memory, and disks allows more significant volumes of data to be stored while the intelligent migration of this data across memory regions and disks allows us to provide low-latency access.

So, how do I access data across the layers (memory and disk)? Users can use SQL to query these larger datasets spread across memory and disk transparently. Tiered Storage can help drive cost-effective data growth as disk-based storage is significantly cheaper than RAM. You can put more business-critical data into the platform’s fast data store without incurring a huge expense for all the added hardware. And it does this while delivering low-latency storage in a reliable, consistent manner. We expect this capability to enable our customers to scale much further and cost-effectively as we introduce additional tiers such as Amazon S3 Standard / Intelligent etc.

Tiered storage is in beta* in 5.3 release.

T-60 minutes and counting…

In today’s dynamic digital landscape, organizations are generating vast amounts of data in real-time. Stream processing data pipelines have become a crucial component of modern data infrastructure to derive meaningful insights and actionable intelligence from this data. SQL, a widely adopted language for database management, is now making its way into stream processing to provide a familiar and powerful tool for data engineers and analysts.

Hazelcast Platform improves and enhances the  capabilities of its SQL engine residing atop the underlying stream processing engine. SQL-based data pipelines can be constructed to ingest and transform data. Furthermore, ANSI-compliant SQL can filter, merge, enrich, and aggregate data. We brought SQL capabilities to create stream processing data pipelines to:.

• Leverage familiar syntax: With stream processing, SQL extends its capabilities to handle real-time data streams, offering a familiar syntax to create data connections to various sources. Whether connecting to popular data sources, such as Kafka, MongoDB, or JDBC-compliant databases, SQL provides a unified approach to accessing and transforming the data for analysis. For example, use familiar SQL syntax to create DATA CONNECTIONS to common data sources and enrich the streaming data to perform intelligent stream processing.
• Build real-time stream processing pipelines: Our stream processing engine involves ingesting and transforming data in real time, for organizations to respond instantly to changing conditions and make timely decisions. SQL simplifies building such pipelines with tools to ingest data from streaming platforms like Kafka, databases, and other sources. With SQL, data engineers can apply transformations on the fly, ensuring the data is prepared for actionable intelligence.
• Perform complex data transformations: SQL’s versatility extends to stream processing, allowing for complex data transformations and analysis. One of the key capabilities is the ability to aggregate data over fixed time windows. This feature enables organizations to calculate average, sum, count, or any custom metric over a specific time frame. For example, in fraud management, SQL can detect suspicious patterns by aggregating transaction data over fixed intervals and applying advanced algorithms for anomaly detection. Streaming SQL with temporal filters for both tumbling and sliding windows is also supported.
• Fast execution: We want SQL to be a first-class interface to the Hazelcast Platform the basic map get/put operations using SQL have to be equally fast as their native counterparts. We have made the SQL execution partition aware: You can send the SQL commands only to the members having the relevant data, which reduces the network hops and improves the query performances.
• Monitoring and managing streaming SQL jobs: Efficient management and monitoring of streaming SQL jobs are essential for maintaining the reliability and performance of data pipelines. SQL Browser within Hazelcast Management Center provides a user-friendly interface with multi-tab support, enabling data engineers and analysts to oversee and manage multiple streaming SQL jobs simultaneously. It allows for real-time monitoring of job statuses, performance metrics, and the ability to identify and troubleshoot issues promptly.
• Rapid deployment with use case demos: Getting started with stream processing can be challenging, especially for those new to the field. To facilitate the adoption of SQL-based stream processing, many platforms offer pre-built templates and demos for common use cases like fraud management, anomaly detection, or real-time analytics. These templates serve as starting points, providing ready-to-use configurations, and can be customized to fit specific requirements. These demos are designed so organizations can accelerate their time-to-insight and quickly adapt to their data processing needs.

By incorporating SQL into their stream processing workflows, organizations can achieve actionable intelligence faster, monitor and manage their jobs efficiently, and use pre-built demos to jumpstart their projects. As the demand for real-time insights grows, SQL’s role in stream processing will only become more critical in the future.

T-0: ignition and engines are igniting!

The data has now been ingested, processed, and stored. It is now ready for consumption and lift-off! But, how fast can we get there?

The thread per core (TPC) based engine – currently in beta – is a turbocharged option designed for rocket-like performance and ultra-low latency operations.  In TPC-mode, the engine propels your applications to new heights, demonstrating exceptional speed, low latency, and throughput with existing data structures. Enterprises can now ignite their performance by experimenting with this high-octane engine, easily toggling it on and off as needed. users can soar through tasks with unparalleled efficiency. Buckle up and prepare for a thrilling ride as the TPC engine powers your applications to the stars.

Get on board and feel the power of the TPC engine, where speed and efficiency propel you to uncharted heights.

Blast off!

You can take the Hazelcast Platform spaceship on an interplanetary test drive  to explore the new features!

Other Cosmic News

You can read more about what’s new and what’s been fixed in our release notes.

What’s more, in GitHub, we’ve closed 557 issues and merged 1152 PRs with Platform 5.3. Special thanks to the contributors!

If you’d like to become more involved in our community or ask some questions about Hazelcast, please join us on our Slack channel, and also please check out our new Developers homepage.

Ignite your innovations with Hazelcast Innovation Labs! Explore our revolutionary products and get early sneak peeks at Innovation Labs. Accelerate your innovation and soar ahead of the competition by joining forces. Unleash the power of cutting-edge technologies and reach new heights of success. Take a seat, join us on this exhilarating journey, and propel your innovations to infinity and beyond! See you in an innovative space!

*For our beta features, we would love to get some early feedback from you and improve the user experience. Please reach out if you want to be an early adopter.

Again, don’t forget to try this new version and let us know what you think. Our engineers love hearing your feedback, and we’re always looking for ways to improve. You can automatically receive a 30-day Enterprise license key by completing the form at https://hazelcast.com/trial-request/