Hazelcast Jet Connectors Release Announcement

Since the first release of Hazelcast Jet Extension Modules in July, we’ve been focused on enhancing the set of extension modules that we have to make the integration easy between Hazelcast Jet and third-party data providers. 

Today, we are happy to announce the release of the following modules:

MongoDB Connector

MongoDB is a general-purpose, document-based, distributed database built for modern application developers and for the cloud era. With this connector, your Jet pipelines can ingest the query results from MongoDB or watch the changes to documents in a collection and emit those changes to downstream processors.

MongoDB is a document store. Documents contain denormalized data, structured for fast consumption without needing to fetch related records using foreign keys. All relevant information is embedded in the document. 

However, data is not always produced in this rich format. Fortunately, Jet can be used as a “preprocessor” to prepare and enrich the data prior to storing it to MongoDB.

MongoDB also allows an event-driven way of data consumption by exposing its changelog. All changes to the documents in MongoDB are available as Change Events. Jet can be used as an elegant high-level API to consume these events, offloading the processing from the MongoDB cluster.

Check out the MongoDB Connector page for more details on how to install and use it as a source or sink.

Redis Connector

Redis is an open source, in-memory data structure store, used as a database, cache, and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, HyperLogLogs, geospatial indexes with radius queries, and streams. With the Hazelcast Jet Redis Connector, your Jet pipelines can make use of hashes, sorted sets, and streams.

Redis, being a key-value store, isn’t optimized for analytical workloads. You can use Jet as a processing engine on top of Redis. Queries expressed as Jet pipelines can transform, aggregate, and join data from Redis structures (hashes, sorted sets), returning results either to a caller or writing it to another system of record (Redis, Hazelcast, MongoDB, etc.).

Redis comes with the Streams general-purpose data structure. Streams is designed for streaming messaging (pub-sub with non-destructive reading), and it is used as a light-weight alternative to Kafka. Redis, however, lacks the API to process the continuous stream of data—e.g., to reason about time-series data, correctly assign it to time windows, or to perform stateful computations on top of it correctly. One can plug in Jet as a stream processor. Moreover, Jet reduces the number of moving parts of the data pipeline to a single library /one JVM process compared to other processing engines. Keep in mind that Redis streams are lightweight, so stick to this architectural decision by using a lightweight processing tool, too.

Check out the Redis Connectors page for more details on how to install and use it as a source or sink.

Wrapping up

We are working hard on delivering new extension modules for Hazelcast Jet. Please let us know if you want to see a particular extension module, and send us any feedback that you have via Github. Stay tuned for more updates!