5

I am using Kafka with strimzi operator, I created a Kafka cluster and and also deployed Kafka connect using yml file. But after this I am totally blank what to do next . I read that Kafka connect is used to copy data from a source to Kafka cluster or from Kafka cluster to another destination. I want to use Kafka connect to copy the data from a file to Kafka cluster's any topic. Can any one please help me how can I do that I am sharing the yml file using which I created my Kafka connect cluster.

apiVersion: kafka.strimzi.io/v1beta1

kind: KafkaConnect

metadata:

  name: my-connect-cluster

#  annotations:

#  # use-connector-resources configures this KafkaConnect

#  # to use KafkaConnector resources to avoid

#  # needing to call the Connect REST API directly

#    strimzi.io/use-connector-resources: "true"

spec:

  version: 2.6.0

  replicas: 1

  bootstrapServers: my-cluster-kafka-bootstrap:9093

  tls:

    trustedCertificates:

      - secretName: my-cluster-cluster-ca-cert

        certificate: ca.crt

  config:

    group.id: connect-cluster

    offset.storage.topic: connect-cluster-offsets

    config.storage.topic: connect-cluster-configs

    status.storage.topic: connect-cluster-status

@kubeclt create -f kafka-connect.yml -n strimzi

After that pod for Kafka connect is in running status ,I don't know what to do next. Please help me.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
rohit290554
  • 79
  • 3
  • 10

2 Answers2

3

Kafka Connect exposes a REST API, so you need to expose that HTTP endpoint from the Connect pods

I read that Kafka connect is used to copy data from a source to Kafka cluster or from Kafka cluster to another destination.

That is one application, but sounds like you want MirrorMaker2 instead for that

If you don't want to use the REST API, then uncomment this line

#    strimzi.io/use-connector-resources: "true"

and use another YAML file to configure the Connect resources , as shown here for Debezium. See kind: "KafkaConnector"

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • 1
    Thanks for your help ...but I very new with this so it's quite tough to understand the documents ...can you please tell me once I create the Kafka connect and Kafka connector using the yml , suggested by you. What to do next..I mean how can I read data from one file and see it to Kafka topic through terminal ...my Kafka cluster is deployed on kubernetes cluster. – rohit290554 Nov 19 '20 at 21:57
  • 1
    Kafka Connect in k8s cannot be used to consume a file unless you exec into the connect container. But you can use any kafka client to produce data into the topic. Then, as mentioned, you would want MirrorMaker, not Connect for sending that data to another topic in another Kafka cluster from where you would write a consumer process... Stimzi isn't really needed for doing any of that – OneCricketeer Nov 19 '20 at 22:53
  • 1
    I'm confused ....My teacher said that use Strimzi operator and deploy kafka cluster using CRD on your local machine and kafka connect, kafka bridge and kafka mirror maker are the listed topics ....I created kafka cluster using CRD and after that I want to learn how to work with kafka connect ...I created kafka connect cluster and its pod is in running status...after that I don't know what to do ....can I use Kafka connect and Kafka mirror maker on my local machine ?....I also read that Kafka connector are also needed h but I dont know how to use it. – rohit290554 Nov 20 '20 at 06:24
  • 1
    I am confused here ...actually my teacher told me to deploy kafka cluster using strimzi and learnt about kafka connect, kafka bridge and kafka mirror maker ...and deploy them using CRD yml on your local machine...I created kafka cluster using CRD and produced and consumed data and understood its working ....but now i am reading about kafka connect ,I created kafka connect cluster using CRD but I totally blank after that what to do next just to understand its exact working ....can you please tell me what can I do to use it on my loacal machine. – rohit290554 Nov 20 '20 at 06:33
  • 1
    I already answered that... You need another KafkaConnector crd and also uncomment the line I mentioned. Did you read through the Debezium example? – OneCricketeer Nov 20 '20 at 15:11
2

Look at this simple example from scratch. Not really what you want to do, but pretty close. We are sending messages to a topic using the kafka-console-producer.sh and consuming them using a file sink connector.

The example also shows how to include additional connectors by creating your own custom Connect image, based on the Strimzi one. This step would be needed for more complex examples involving external systems.

fvaleri
  • 638
  • 1
  • 4
  • 10