2

Our test automation needs to interact with kafka and we are looking at how we can achieve this with karate.

We have a java class which reads from kafka and puts records in an internal list. We then ask for these records from karate, filter out all messages from background traffic, and return the first message that matches our filter.

So our consumer looks like this (simplified):

// consume.js
function(bootstrapServers, topic, filter, timeout, interval) {
  var KafkaLib = Java.type('kafka.KafkaLib')
  var records = KafkaLib.getRecords(bootstrapServers, topic)

  for (record_id in records) {
    // TODO here we want to convert record to a json (and later xml for xml records) so that
    // we can access them as 'native' karate data types and use notation like: cat.cat.scores.score[1]
    var record = records[record_id]
    if (filter(record)) {
      karate.log("Record matched: " + record)
      return record
    }
  }

  throw "No records found matching the filter: " + filter
}

Records can be json, xml, or plain text, but looking in the json case now. In this case given that in kafka there is a message like this: {"correlationId":"b3e6bbc7-e5a6-4b2a-a8f9-a0ddf435de67","text":"Hello world"} This is loaded as a string in the record variable above.

We want to convert this to json so that a filter like this would work:

* def uuid = java.util.UUID.randomUUID() + ''
# This is what we are publishing to kafka
* def payload = ({ correlationId: uuid, text: "Hello world" })
* def filter = function(m) { return m.correlationId == uuid }

Is there a way to convert a string to a native karate variable in javascript? Might have missed it looking at https://intuit.github.io/karate/#the-karate-object. By the way var jsonRecord = karate.toJson(record) did not work and jsonRecord.uuid was undefined.

Edit: I have made an example of what I am trying to achieve here: https://github.com/KostasKgr/karate-issues/blob/java_json_interop/src/test/java/examples/consumption/consumption.feature

Many thanks

Peter Thomas
  • 54,465
  • 21
  • 84
  • 248
Konstantinos Kollias
  • 77
  • 1
  • 8

4 Answers4

2

Sometime ago I had put together a something that could be used to test Kafka from within Karate. Pls see if https://github.com/Sdaas/karate-kafka helps. Happy to enhance / improve if it helps you.

sdaas
  • 174
  • 1
  • 5
  • Hello @sdaas, thanks for the feedback! I went through the readme, very interesting! Are you still actively testing with kafka? I have made something similar, but only bootstrap servers are configurable. However on the consumer side ive been trying to implement the following, still evolving. If you are in the karate slack drop me a message! https://www.confluent.io/blog/testing-event-driven-systems/ – Konstantinos Kollias Jul 24 '20 at 06:24
  • Sorry @KonstantinosKollias, was away for a while and didn't see this. i have again restarted development of karate-kafka ( same link above ).And yes, I am on the karate slack - will drop you a note. – sdaas Sep 10 '20 at 04:12
1

Can you try,

* json payload = { correlationId: uuid, text: "Hello world" }

ref : Type Conversion

for type conversion within javascript ideally karate.toMap(object) or karate.toJson(object) should.

rather than wrapping up everything into one JS function, I would suggest keeping the record invoking part outside the JS and let karate cast it.

* json records = Java.type('kafka.KafkaLib').getRecords(bootstrapServers, topic)
* consume(records, filter, timeout, interval)
Babu Sekaran
  • 4,129
  • 1
  • 9
  • 20
  • Hello @Babu, thank you for your response, I want to achieve this while in a javascript function. I uploaded a simplified example to showcase my issue: https://github.com/KostasKgr/karate-issues/blob/java_json_interop/src/test/java/examples/consumption/consumption.feature – Konstantinos Kollias Jun 29 '20 at 14:17
  • is it possible to change your `KafkaLib.getRecords` ti return `List` instead of `List` ? – Babu Sekaran Jun 29 '20 at 15:25
  • Hello Babu, we dont know what the records will be, e.g. in some cases they will be json and in others xml. Our java consumer code wont know the specifics, so justs reads as a string to be able to work in general. When engineers test a specific case they will then know more, and they can use that knowledge to write the filter and their match and assertions. Also trying to make the syntax as simple as possible so that more people in our org can contribute – Konstantinos Kollias Jun 29 '20 at 15:38
  • 1
    Maybe we could try to add more methods in the java class, e.g. getRecordsAsJson, getRecordsAsXml, and pass the desired type we want to the reusable consume.feature(which assumes the messages in a topic are homogenous, which sounds fair). However if we could reuse the logic that def X = ... Uses to derive the type, would seem preferable. Ill need to trial the above and see how it feels as a DSL, and if it works – Konstantinos Kollias Jun 29 '20 at 15:49
  • 2
    @KonstantinosKollias dev of Karate here - I'm open to adding some JS helpers. Karate's logic is very simple - if it starts with a `<` it is XML and if `{` or `[` it is JSON. you are right, the existing logic is biased towards the non-JS (DSL / Gherkin) side. I've opened an enhancement req - feel free to weigh in here: https://github.com/intuit/karate/issues/1202 – Peter Thomas Jun 29 '20 at 18:10
1

As mentioned in the comments of another answer, there is now an enhancement ticket on karate to achieve what was discussed in this thread, see https://github.com/intuit/karate/issues/1202

Until that is in place, I managed to get most of what I wanted concerning JSON by parsing string to json in Java and returning that to karate.

Map<String,Object> result = new ObjectMapper().readValue(record, HashMap.class);

Not sure if the same can be worked around for xml

You can see the workaround in action here: https://github.com/KostasKgr/karate-issues/blob/java_json_interop_v2/src/test/java/examples/consumption/consumption.feature

Konstantinos Kollias
  • 77
  • 1
  • 8
0

Because of Karate's support for Java inter-op you can easily write some "glue" code to connect your existing Kafka systems to Karate test-suites, see the first link below.

Here are a few references:

Peter Thomas
  • 54,465
  • 21
  • 84
  • 248