1

I have installed Kafka "kafka_2.12-0.11.0.0".

Started services of both zookeper and Kafka-server.

Then i created a java project (kafka lib included) on IntelliJ.

I also included an SQLite jar needed for the JDBC connect.

I initially thought of using confluent.kafka-connect, but since mine is a windows based machine CLIplatform doesn't work.

So i wrote a stand alone java program to connect to my sqlite DB fetched data and sent this as a message using a custom Kafka-Producer program creating a topic "connect-test".

I received the same data using a custom Kafka-Consumer program. The purpose of this test, is to see if i can do the same with an existing windows based, enterprise application to stream data into a warehouse for Data Analytics platforms. I understand that i have done it in stand-alone mode, but am i in the right direction?

Can you please suggest if there is a better way for me to integrate my enterprise application (java based Custom API + oracle DB +another layer on top of Oracle DB) with Kafka?

Would really appreciate your help.

Brad Larson
  • 170,088
  • 45
  • 397
  • 571
Shashank
  • 101
  • 2
  • 9
  • in addition to @robin-moffatt 's answer kafka is not platform bound apart from [this bug under windows](https://issues.apache.org/jira/browse/KAFKA-1194) you can write your implementation and/or use an existing [connector](https://www.confluent.io/product/connectors/) – Paizo Jun 11 '18 at 15:47
  • Apache Nifi or Streamsets would also work. – OneCricketeer Jun 12 '18 at 12:59
  • Thank you @Paizo for the hint, i will look into that as well. This is helpful. – Shashank Jun 12 '18 at 13:35
  • @cricket_007, I have not looked into them yet, i will deep dive. Thank you for the information. – Shashank Jun 12 '18 at 13:36

1 Answers1

0

Kafka Connect is part of Apache Kafka, and is the recommended way for doing integration with other systems including databases. This article explains the specific options you have for integrating with Oracle, and this talk shows it in action.

since mine is a windows based machine CLIplatform doesn't work.

Kafka Connect should work on Windows. I'm not sure what you mean by CLIplatform since it's Java-based it should run on Windows, and if not there are Docker images you can use.

Disclaimer: I work for Confluent and wrote the above articles linked to

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
  • Hi @Robin, Your articles are an inspiration for looking into Confluent. The "Command line interface" of confluent package is i believe still not readily usable to start all services on windows. – Shashank Jun 12 '18 at 01:36
  • To add on... Nevertheless, I am able to work by manually start zookeper & kafka-server but i am still wondering how do i start the "schema-registry-start" service on windows. – Shashank Jun 12 '18 at 01:50
  • [link](https://stackoverflow.com/questions/39129210/start-confluent-schema-registry-in-windows). I think i found something here which is relevant. – Shashank Jun 12 '18 at 02:48
  • @Shashank Schema Registry now has Windows scripts, according to github https://github.com/confluentinc/schema-registry/tree/master/bin/windows – OneCricketeer Jun 12 '18 at 13:01
  • Thank you @cricket_007 for the reference, i will try this out and comment if i progress. – Shashank Jun 12 '18 at 13:30