11

More specifically, what usecases does Hazelcast Jet solve that Flink does not solve (equally well) and vice versa?

Atle
  • 428
  • 3
  • 11

2 Answers2

18

NOTE: I belong to Hazelcast Jet's core engineering team.

I'd say the main advantage of Hazelcast Jet isn't in offering a brand-new computing model, but in bringing the same level of convenience that Hazelcast is known for to the realm of DAG-based distributed computing.

If you currently have a Java application running in a cluster, adding Jet will be a snap: add the Maven dependency and write one line of code to start a Jet instance on the local member. The instances will self-discover to form their own cluster, and you can now submit your job to it.

If you want a dedicated distributed computing cluster, you can download the distribution ZIP to the cluster machines. Jet has native support for the most popular cloud environments, allowing the nodes you start to self-discover. You can then connect to the cluster using a Jet client.

Needless to say, Jet makes it very convenient to use a Hazelcast IMap or IList as a data source. Jet cluster can host Hazelcast structures directly; then you benefit from data locality and get the data with no network traffic. On the other hand, the choice of data source is completely unconstrained and there is public API dedicated to implementing fast, arbitrarily partitioned, custom data sources.

Jet solves the concerns of infinite streams processing like aggregating over time-based windows, dealing with reordered events and resilience to changes in the cluster topology (e.g., failure of individual Jet nodes) while maintaining the Exactly-Once processing guarantee.

Jet's main programming paradigm is the Pipeline API which is quite similar to java.util.stream API but adapted to the specifics of distributed computing (lambda serialization and other concerns).

Pipeline API builds upon a lower-level DAG-based model that is also exposed as public API.

Marko Topolnik
  • 195,646
  • 29
  • 319
  • 436
  • 1
    So you can say that the main motivation for releasing Jet is to offer a tool that is easier to use than Flink, and that its planned featureset overlaps with the current market? Or would you say that Jet is a "lightweight" solution for applications that do not need the same features as existing solutions? – Atle Feb 17 '17 at 07:28
  • As Jet is about to reach the 4.0 release, I have revisited this question. I'd say that Jet is carving its own path through the supported feature set. There are many source/sink connectors to interact with diverse external systems; there is solid support for streaming joins and windowed aggregation; and one use case that is currently in focus is the deployment of cross-language ML models to a distributed streaming pipeline. – Marko Topolnik Jan 23 '20 at 07:33
1

in my opinion, flink seems to offer some very useful streaming features, wich are not yet offered by hatecast jet.

  • Different flexible window operators, wich also can handle out-of-order and late items.
  • fault tolerance on the cluster and delivery guarantees

Beside this it also seems to be more stable and well known at the moment. For example, you can use it as a runtime for Apache Beam and then migrate easily between Google Data Flow on the cloud and your own deployment. So I would currently use flink.

Best

Philipp Grulich
  • 123
  • 1
  • 2