0

I am using dataflow kafka to bigquery template. after launching the dataflow job, it stays in queue for some time then fails with below error:

java.lang.RuntimeException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

While lauching job, i have provided below parameters:

  1. kafka topic name
  2. bootstrap server ip and port (private ip)
  3. bigquery topic name
  4. SA email
  5. zone.
  6. Network , subnet (same network where kafka server running)

My kafka topic only contanis message: hello

kafka is installed in gcp instance which is in same zone and subnet as dataflow worker.

Joseph N
  • 540
  • 8
  • 28
  • The issue seems to be on the side of Kafka, a possible solution is mentioned in the following [Stackoverflow thread](https://stackoverflow.com/questions/54254686/timeoutexception-timeout-expired-while-fetching-topic-metadata-kafka) where is mentioned that the issue can happen when trying to connect to a broker expecting SSL connections and the client config did not specify and the solution is to set the security protocol to SSL: `security.protocol=SSL` – Messier_31 Dec 07 '20 at 21:40
  • Make sure your Kafka broker `listeners` properties are setup to bind externally – OneCricketeer Dec 07 '20 at 21:41
  • Hey Thanks @OneCricketeer , i was trying to access kafka with internal ip. it worked when i ched it to public ip. Actually i am running both kafka machines and workers in same subnet. so it should work with internal ip also... i am checking it now – Joseph N Dec 08 '20 at 07:28

1 Answers1

1

This issue is solved. i was trying to access kafka bootstrap server with internal ip. it worked when i changed it to public ip.

Joseph N
  • 540
  • 8
  • 28