5

I am learning Apache Kafka from their quickstart tutorial: http://kafka.apache.org/documentation.html#quickstart. Upto now, I have done the setup as follows. A producer node, where a web server is running at port 8888. A Kafka server(broker), Consumer and Zookeeper instance on another node. And I have tested the default console/file enabled producer and consumer with 3 partitions. The setup is perfect, and I am able to see the messages I sent in the order they created (with in each partition).

Now, I want to send the logs generated from the web server to Kafka Broker. These messages will be processed by consumer later. Currently I am using syslog-ng to capture server logs to a text file. I have come up with 3 rough ideas on how to implement producer to use kafka for log aggregation

Producer Implementations

First Kind: Listen to tcp port of syslog-ng. Fetch each message and send to kafka server. Here we have two middle processes: Producer and syslog-ng
Second Kind: Using syslog-ng as Producer. Should find a way to send messages to Kafka server instead of writing to a file. Syslog-ng, the producer is the middle process.
Third Kind: Configuring the webserver itself as producer.

Am I correct in my thinking. In the last case we don't have any middle process. But I doubt its implementation will effect server performance. Can anyone let me know the best way of using Apache Kafka(if the above 3 are not good) and guide me through appropriate configuration of server?..

P.S.: I am using node.js for my web server

Thanks,
Sarath

Sarath Chandra Jiguru
  • 135
  • 1
  • 1
  • 10

1 Answers1

2

Since you specify that you wish to send the logs generated to kafka broker, it indeed looks as if executing a process to listen and resend messages mainly creates another point of failure with no additional value (unless you need a specific syslog-ng capability).

Syslog-ng can send messages to external applications using: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guides/en/syslog-ng-ose-v3.4-guide-admin/html/configuring-destinations-program.html. I don't know if there are other ways to do that.

For the third option, I am not sure if kafka can easily be integrated into Node.js as it requires a c++ producer and when I last looked for one, I was not able to find. However, an easy alternative could be to have kafka read the log file created by the server and send those logs (using the console producer provided with kafka). This is usually a good way, as it completely remove dependencies between kafka and the web server (embedding the producer in would require error handling, configuration, etc). It requires the use of tail --follow and it works for us very well. If you wish more details on that, I can include them as well. Still you would need to supervise kafka execution to make sure messages are not lost (and provide a recovery option to offline send messages that failed). But, the good thing about this method is that there are no dependency between the tools.

Hope it helps...

Eran

erankl
  • 316
  • 1
  • 6
  • 1
    Thanks @erankl. And apologies for very delayed response. Actually stopped working on this as I have other work. I have decided to use the first option. The reason is that from syslog-ng I can send all messages to a pipe, that could be read by the producer. producer will send this to Kafka and ultimately processed by storm for some realtile analysis. Also syslog-ng is used to save them to a file, which will be used for some offline batch processing. Will write a blog once successfully completed it.. – Sarath Chandra Jiguru Oct 23 '13 at 12:55
  • Hello Sarath. Did you write a blog article about your setup? – gue Apr 27 '14 at 11:59
  • We have similar set up. Instead `syslog-ng` we have been tailing the `error_log` file and feeding to producer. But as the `error_log` file gets rotated, we missed few logs. So I was wondering how reliable is consuming from `syslog-ng` We tested the flow and did not miss the log. But in that case we have too many indirections `stdout > tee >> error_log and logger` then `logger >> socket >> syslog-ng >> nc >> producer`. So I would like to know about the reliability of `syslog-ng` set up – Albatross Jan 20 '17 at 18:45