Kafka - how to transform Rest service message to ingest into a kafka topic?

Question

I'm new at kafka community and i'm facing a challenging question. I have two applications that talk each other via Rest webservice, with the body being a json message.

How can I use kafka to be the middleware between these two application, with the minor or zero impact on the applications?

This is my As-Is scenario:

My To-Be scenario:

I've thought in use STM or some interceptor to transform the header and body.

Is it possible to do that way? I know that I have to transform the body from:

      {
        "first name" : "Donald" ,
        "last name" : "Trump"
      },
      {
        "first name" : "Vladimir" ,
        "last name" : "Putin"
      }

to:

{
  "records":
  [
    {
      "key": "K001",
      "value":
      {
        "first name" : "Donald" ,
        "last name" : "Trump"
      }
    },
    {
      "key": "K002",
      "value":
      {
        "first name" : "Vladimir" ,
        "last name" : "Putin"
        }
    }
  ]
}

Essentially my transformation would change the beginning and the end of the message, and not a field of each record.

Anyone have done something like that, or has a indication of which way should I go?

Can you give some more background as to what's driving this change? Why the introduction of Kafka? Why the assumption you'll still use REST? With that it'll be easier to help :) — Robin Moffatt, Jan 25 '19 at 14:27
Hi Robin, I work as solution architect, and we are changing our reference architecture regarding integration, we are proposing to evolve from peer to peer integration to mediated integration. These two applications are integrated via webservice today, and the body notation is json, we want to use kafka as our middleware and we don't want to bring impact to these applications. These are the main reason to translate the message before ingest it to a topic. — Higor, Jan 25 '19 at 16:12
What I'm still not clear on is why, given that there are Kafka client libraries for numerous languages, you still want to use REST for this? — Robin Moffatt, Jan 25 '19 at 16:40
Robin, These two applications are COTS that were customized in order to speach each other, now we want to introduce kafka on this scenario with zero impact on these applications — Higor, Jan 25 '19 at 16:57

score 0 · Answer 1 · answered Jan 25 '19 at 17:04

0

You probably want a Kafka Streams application to do this. It would subscribe to the source topic populated via REST from Application A, transform the message as required and write it to a new topic, from which Application B could consume it.

Kafka Streams is part of Apache Kafka, and is a Java library with which you can build stream processing applications. There is also KSQL which is an abstraction on top of Kafka Streams in a SQL-like language, but I'm not sure it would support the type of transformation that you're looking at here.

Since you're not using Kafka Connect, Single Message Transform (SMT) are not applicable here.

answered Jan 25 '19 at 17:04

Robin Moffatt

30,382
3
65
92

Robin, Let me see if i understood right, on your proposal, i should create an kafka streams to read the information from a topic and write it to another topic, is it? I can't write the message into a topic, because the Application A (producer), calls a webservice, that has a header: "Content-Type: application/json", instead a header: "Content-Type: application/vnd.kafka.json.v2+json" and the message with a collection of values, without the beginning of the message: `{ "records": [` and without the set: `"key": "K00x", "value": { data }`, before each set of data. – Higor Jan 28 '19 at 12:35

score 0 · Accepted Answer · answered Jan 25 '19 at 22:48

0

Outside of the Kafka ecosystem, you can use NiFi or Streamsets for these "message transforms" you are looking for, and both can do HTTP calls as well as starting HTTP servers themselves for direct API consumption.

answered Jan 25 '19 at 22:48

OneCricketeer

179,855
19
132
245

Thank you @cricket_007, any of these applications can be the solution for my problem, and i can even use them to be my connector, allowing to connect to a database and restfull webservice. – Higor Jan 31 '19 at 11:53
That's right, those tools are more flexible than just what Kafka & Kafka Connect provides – OneCricketeer Feb 01 '19 at 19:58

Kafka - how to transform Rest service message to ingest into a kafka topic?

2 Answers2