Hazelcast Jet Connectors Release Announcement

September 18, 2019
Hazelcast Jet Connectors Release Announcement

Since the first release of Hazelcast Jet Extension Modules in July, we’ve been focused on enhancing the set of extension modules that we have to make the integration easy between Hazelcast Jet and third-party data providers. 

Today, we are happy to announce the release of the following modules:

MongoDB Connector

MongoDB is a general-purpose, document-based, distributed database built for modern application developers and for the cloud era. With this connector, your Jet pipelines can ingest the query results from MongoDB or watch the changes to documents in a collection and emit those changes to downstream processors.

MongoDB is a document store. Documents contain denormalized data, structured for fast consumption without needing to fetch related records using foreign keys. All relevant information is embedded in the document. 

However, data is not always produced in this rich format. Fortunately, Jet can be used as a “preprocessor” to prepare and enrich the data prior to storing it to MongoDB.

MongoDB also allows an event-driven way of data consumption by exposing its changelog. All changes to the documents in MongoDB are available as Change Events. Jet can be used as an elegant high-level API to consume these events, offloading the processing from the MongoDB cluster.

Check out the MongoDB Connector page for more details on how to install and use it as a source or sink.

Redis Connector

Redis is an open source, in-memory data structure store, used as a database, cache, and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, HyperLogLogs, geospatial indexes with radius queries, and streams. With the Hazelcast Jet Redis Connector, your Jet pipelines can make use of hashes, sorted sets, and streams.

Redis, being a key-value store, isn’t optimized for analytical workloads. You can use Jet as a processing engine on top of Redis. Queries expressed as Jet pipelines can transform, aggregate, and join data from Redis structures (hashes, sorted sets), returning results either to a caller or writing it to another system of record (Redis, Hazelcast, MongoDB, etc.).

Redis comes with the Streams general-purpose data structure. Streams is designed for streaming messaging (pub-sub with non-destructive reading), and it is used as a light-weight alternative to Kafka. Redis, however, lacks the API to process the continuous stream of data—e.g., to reason about time-series data, correctly assign it to time windows, or to perform stateful computations on top of it correctly. One can plug in Jet as a stream processor. Moreover, Jet reduces the number of moving parts of the data pipeline to a single library /one JVM process compared to other processing engines. Keep in mind that Redis streams are lightweight, so stick to this architectural decision by using a lightweight processing tool, too.

Check out the Redis Connectors page for more details on how to install and use it as a source or sink.

Wrapping up

We are working hard on delivering new extension modules for Hazelcast Jet. Please let us know if you want to see a particular extension module, and send us any feedback that you have via Github. Stay tuned for more updates!

About the Author

About the Author

Emin Demirci

Emin Demirci

Team Lead, Jet Integration, Hazelcast

Emin leads the integration efforts of Hazelcast Jet with the outer world. Prior to that, he had to get his hands dirty with all the moving parts of the product family of Hazelcast, from implementing new features to creating fancy UIs. Before joining to Hazelcast, Emin did an internship at PGS Software and Hazelcast. He holds a BSc. degree in Computer Engineering from Karadeniz Technical University. He is a petrolhead, enjoys playing football (also proud to be a fan of Galatasaray) and cycling.

Follow me on

Latest Blogs

Announcing the First Release of Hazelcast Jet Extension Modules

Announcing the First Release of Hazelcast Jet Extension Modules

How to Deploy Hazelcast Jet – Part II on Docker with Docker Compose

Deploying Hazelcast Jet

View all blogs by the author

Subscribe to the blog

Follow us on: