0

I have a function like:

def func():
    While True:
        # here is some code that parsing some website

I have a telegram bot that interacts with the user. The user has the opportunity to choose via telegram bot which pages of the site he wants to parse. The user selects parameters for parsing through the bot and then all these parameters are stored in MySQL database.

So, now I have a number of rows in MySQL database. How can I run func() several times at the same moment?

Because if I have like 100 rows in my DB I need to run func() 100 times at the same time, not one by one.

Tomerikoo
  • 18,379
  • 16
  • 47
  • 61
noname
  • 9
  • 1
  • Look at the modules `threading`, `multiprocessing` or `asyncio`. All of these offer some form of concurrency (though in a very different way) – mousetail Feb 01 '21 at 11:53
  • Unfortunately Python isn't the best language for parallel execution due to internal design of the language (namely Global Interpreter Lock is the thing that prevents multiple commands from executing in parallel). A solution would be to use multiple processes using `multiprocessing` module, or some other tool, say MPI. – Alexey S. Larionov Feb 01 '21 at 11:53
  • 1
    Real answer is: you can't request 100 pages in the same moment, since your internet connection is serial. Even if it wasn't, you only have ~8 CPUs, so 100 is not possible. – Thomas Weller Feb 01 '21 at 11:54
  • @AlexeyLarionov Fetching rows is mostly a IO heavy operation so the GIL won't matter that much – mousetail Feb 01 '21 at 11:56
  • 1
    Does this answer your question? [How to use python to query database in parallel](https://stackoverflow.com/questions/37975904/how-to-use-python-to-query-database-in-parallel) – Tomerikoo Feb 01 '21 at 11:58
  • Or [MySQL select request in parallel (python)](https://stackoverflow.com/questions/35040848/mysql-select-request-in-parallel-python) – Tomerikoo Feb 01 '21 at 11:59
  • 1
    What's your reason for this requirement? I don't see any in the question. – Klaus D. Feb 01 '21 at 12:01
  • https://medium.com/hackernoon/how-to-run-asynchronous-web-requests-in-parallel-with-python-3-5-without-aiohttp-264dc0f8546 – k0pernikus Feb 01 '21 at 12:03
  • Does this answer your question? [Asynchronous Requests with Python requests](https://stackoverflow.com/questions/9110593/asynchronous-requests-with-python-requests) – k0pernikus Feb 01 '21 at 12:03
  • @Klaus D. My reason for this requirement is to make parsing function run faster when I have more users. Because as I said user selects which pages of the site he want to parse and their parsing parameters adds to my DB. When I have 50-60 users i need to run this function 50-60 times, one by one. It takes time to run all this funcs. I wanted to run all of them at the same time. Is there some other way to make 50-60 functions run fast? – noname Feb 01 '21 at 15:50
  • This is not how parallel processing works. Usually you make sure that you have the capability to run a few tasks at the same time. "A few" should be around the number of available CPUs. All the rest should be done by the OS. It will give time slots on CPUs to the processes handling the tasks taking several factors into account. – Klaus D. Feb 01 '21 at 18:32
  • Run multiple copies of the _program_. But don't try 100 copies. Have each copy pick one user (or whatever) to work on. – Rick James Feb 01 '21 at 20:12

0 Answers0