0

I want to publish my JSON data to PublishKafka Processor in Apache Nifi

  • Processor : PublishKafka_2_0
  • Apache Nifi Version : 1.15.3
  • Kafka Version : kafka_2.13-3.1.0

Here is my configuration settings : enter image description here

enter image description here

My kafka server is live and I can produce the "mytopic" topic from the console. I get this error.What am i missing ?.What should ı do ? enter image description here

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Tugrul Gokce
  • 160
  • 8
  • Are both Nifi and Kafka on the same machine? Not in containers? What have you configured to use GSSAPI? – OneCricketeer Mar 14 '22 at 02:55
  • 1
    @OneCricketeer No its not same machine.Im not use containers.Machines in compute engine(GCP).I dont know what does it mean 'GSSAPI' by that.Where can ı found this configure. – Tugrul Gokce Mar 14 '22 at 18:09
  • 1
    1) If there are separate machines in GCP, why are you trying to use `localhost` to connect to Kafka? Did you try an IP or hostname instead? 2) In your image, you are using GSSAPI authentication, rather than no authentication – OneCricketeer Mar 14 '22 at 19:00
  • 1
    Try increasing `Max Request Size` property value, something like 10 MB or observe flowfile sizes in upstream. – Vikramsinh Shinde Mar 15 '22 at 07:11
  • @VikramsinhShinde I set it to Max Request Size the 45 MB.But ı got different error.Kafka: org.apache.kafka.common.errors.RecordTooLargeException: The message is 42733517 bytes when serialized which is larger than the total memory buffer you have configured with the buffer.memory configuration. I change the producer.properties settings.I set it to buffer.memory=47185920. Then restart kafka again.But nothing change. – Tugrul Gokce Mar 15 '22 at 10:58
  • 1
    1) Your request size has nothing to do with timeout errors. 2) Kafka is not meant to be used with large records or whole files. Can you break apart your data? If not, see https://stackoverflow.com/q/21020347/2308683 Where are you getting 45MB json events from? – OneCricketeer Mar 15 '22 at 14:46
  • 1
    @OneCricketeer sorry man.I'm new to Kafka.I'm trying to learn Kafka's logic.I was able to consume the data from Kafka when I sent it small.Thanks for everthing. – Tugrul Gokce Mar 16 '22 at 11:16
  • 1
    Feel free to answer your own question below with the solution that eventually worked for you – OneCricketeer Mar 16 '22 at 19:03

0 Answers0