Take output from a user defined service and export it to Kafka

Question

I am working on an application that needs to take the output of a tool(output in the terminal - constantly running output) that is running as a service on a machine and then find a way to use that output as the input for a Kafka Producer. I was unable to find too many details about this approach so any help would be appreciated. The only option that I could think of was to create a file of that output at a certain interval of time and then use that file as the input for my own Kafka Producer app but I am losing speed as the creation of the files will have a delay.

Answer 1

You can pipe the output of any command to kafka-console-producer or kcat

my-command

#!/usr/bin/env python
print("hello world")

$ chmod +x my-command
$ ./my-command | kafka-console-producer --topic foobar --bootstrap-server localhost:9092
$ kafka-console-consumer --topic foobar --from-beginning --bootstrap-server localhost:9092
hello world

Keep in mind, this will be line-delimited strings, so if you want something more robust, or are dealing with binary data, you'll need to attach a stream to the process's output stream file-descriptor (rather than a separate output file) via code. For example see this post, then translate the tail command to something else.