0

When I install a Mosquitto Broker and publish messages to a topic and subscribe the the messages using an Apache Beam MQTTIO pipeline and print the message in the console, I am able to get all the messages.

Even after a gap of 5 minutes if I publish a message, I am able to see the message in the console of the Beam application

Now I changed nothing but the ServerUri in the ConnectionConfiguration of MQTTIO.Read() and gave a corresponding topic that is available in the that broker. Refer Documentation

This change made the application work differently, only the topics that have higgh frequency of messages are being printed in the console. When the message frequency if 1 message per minute or somewhere near to that range of frequency, only the first message is getting printed in the console.

I even tried out withMaxNumRecords and withMaxReadTime but still it just lists the first message.

Is there Some timeout that is happening in the broker configuration because of its idlenesss between messages?

The same behaviour was observed in runners - DirectRunner and FlinkRunner

Changing the options to --streaming=true for the pipeline using the arguments did not work as well

Viswadeep V
  • 13
  • 5
  • Do you have any kind of windowing and/or triggers set up in this pipeline? That's a good first step to check when there are issues like this in pipelines with unbounded pcollections (which is what MqttIO.Read produces). – Daniel Oliveira Sep 23 '21 at 02:35
  • Yes, was able to try with Fixed Windows / Triggers. Still not sure what is happening.. – Viswadeep V Oct 04 '21 at 09:45
  • It would help if you could include a description of your pipeline, including what kind of windowing and triggers are in use. A sample of your code would be even better if possible. – Daniel Oliveira Oct 12 '21 at 22:39

0 Answers0