0

Hi currently I am using python telepot to:

  1. Listen to incoming message
  2. Based on the message it will do a long operation to generate some image.

Problem: The message is being process 1 by 1.

So if 2 users send in the message at the same time, it will process and send the image 1 by 1.

Have started reading Multiprocessing, but it is not getting what I want. What I have achieve now.

  1. Switched to aiogram, listen to message
  2. Pass the message to Process(target=imageprocessing.incoming, args=(userMsg,))
  3. This kind of work, but every time a message comes in it will start a new process, it is kind of slow because it has to initialise some libraries etc before it process it.

Thus it is possible to:

  1. Start X amount of "sub-process" in the main script initially.
  2. Main script will listen for incoming message, queue it and pass it to one of the "sub-process"
  3. So that X amount of message simultaneously based on how many "sub-process" I have defined initially.

Have been self-learning python as a side hobby. Not sure if I have used the term correctly.

Try googling but can't find a solution for now, got a feeling I might be looking up the wrong keyword.

Anyone has any idea what should I look for?

Thank you very much.

Cytan
  • 53
  • 6
  • What you describe is a common architectural pattern. For example, many web servers work this way: the listener process accepts incoming connections and hands each over to a worker process, which has ideally been started up earlier. This isn't trivial to implement from scratch, though. The scaling of processes requires some thought: how many workers do you start up, and under what conditions do you increase or decrease their number? Perhaps you could find an existing library which implements the base logic. The other challenge is the IPC; how exactly does the server talk to a waiting worker? – tripleee May 14 '22 at 09:24
  • The "concurrent.futures" module and its "ProcessPoolExecutor" provide basic functionality to achieve this. – Michael Butscher May 14 '22 at 09:31
  • An option might be to use an existing web server and adapt it to support your chat protocol. For example, https://stackoverflow.com/questions/57104398/python-flask-how-to-run-subprocess-pass-a-command shows how to run a subprocess from Flask and https://stackoverflow.com/questions/35616639/python-multiprocessing-in-flask discusses how to set it up to scale. – tripleee May 14 '22 at 09:32

0 Answers0