-1

I am writing a web app to log MQTT Data in a database after processing then a dashboard will query the database and visualize the data.

I am using a python script, which runs all time with Paho MQTT Client to process and store MQTT data in a database.

I think there should be a better and scalable way to process MQTT payloads. Can anyone suggest a better way?

Monem Ahmed
  • 71
  • 8
  • You would need to define what you mean by "better and scalable". Unless you are processing [hundreds of messages per second](https://muetsch.io/basic-benchmarks-of-5-different-mqtt-brokers.html) scalability is unlikely to be too much of an issue. "Better" is subjective; for example using a separate broker is better because you can update your webapp without loosing any messages. – Brits Feb 21 '22 at 01:15
  • @Brits I am not talking about broker's scalability here. I am concerned about My python client, which logs mqtt data in database after processing. Better means since, I am no expert in IoT Architecture, I want to learn how experts do it. Thank You. – Monem Ahmed Feb 21 '22 at 03:03
  • Sorry - still not clear on your issue. If you are concerned about your python client then please raise a specific issue (and ideally share source). [MQTT](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html) is a "Client Server publish/subscribe messaging transport protocol" so you appear to be using it as intended (client connects to broker and subscribes to topic(s)) and products like [Telegraf](https://www.influxdata.com/integration/mqtt-monitoring/) use the same approach. Note that opinion based questions are [off-topic](https://stackoverflow.com/help/on-topic). – Brits Feb 21 '22 at 03:27
  • "The MQTT Consumer Telegraf Input Plugin reads from specified MQTT topics and adds messages to InfluxDB. " I am following same approach using a single python script. I was hoping to find out a better MQTT data processing mechanism. Since I do not think a python script, with Paho Client will be able to handle MQTT data when number of messages increases. Or it will take longer time. I want to implement some mechanism here, which will ensure real time data processing and will be scalable easily. Sorry for the ambiguity. I will raise another issue with source code. Thanks for your time. – Monem Ahmed Feb 21 '22 at 13:35
  • Should I delete this question? – Monem Ahmed Feb 21 '22 at 13:36
  • I'd leave it up - it's been closed so is not visible to most and deleting it would remove the points you game balun. It appears to me that you are trying to [prematurely optimise](https://stackoverflow.com/a/385529/11810946) your system; its likely that your python code will be fine (the database will probably become an issue before that because the python code does very little). – Brits Feb 21 '22 at 20:34

1 Answers1

0

I ever have the same doubt before, but I think you are doing the right way.

If we think further of MQTT nature, there is no traditional "server side" in MQTT context. Most probably, people treat "broker" as server side, but it is totally different, broker in MQTT core concept shall be content-agnostic and thus shall not handle messages directly.

Of course, you can always bundle an internal subscriber for data handling features with the broker and make it looks like one complete "server" solution, like some commercial MQTT brokers do. But conceptually, it is still handled by a subscriber and not much different from your proposed way.

balun
  • 1,191
  • 1
  • 7
  • 12
  • Thank you. I Understand that this data should be handled by a client. Can you tell me how can I test the performance of my python script here. Although I haven’t faced any problems yet. But I am not confident about it's scalability. – Monem Ahmed Feb 21 '22 at 03:06
  • so your actual question is how to test your performance of your python code? for that, you can use many load-testing frameworks, e.g. jmeter, gatling, locust, etc. Here is one example from thingsboard, `https://thingsboard.io/docs/reference/performance-tools/` , FYI. – balun Feb 21 '22 at 06:17
  • Thank you for your response. My actual question was about MQTT data processing, since you said I am doing things in right way, then I asked about performance testing. Thank you so much for information regarding this. – Monem Ahmed Feb 21 '22 at 13:25