0

I am running the code blow. I am trying to run multiple web pages at the same time that would go to mail.google.com. For example the code below should open the website mail.google.com on multiple different windows at the same time.

from selenium import webdriver
from multiprocessing import Process, Pipe, Pool
from webdriver_manager.firefox import GeckoDriverManager

def create_browser(num):

    # Defines Browser
    browser = webdriver.Firefox(executable_path=GeckoDriverManager().install())
    browser.get('https://mail.google.com/')

    return browser


pool = Pool(processes=10)  # Maximum number of browsers opened at same time

for i in range(0, 2):  # Five browsers will be created
    async_result = pool.apply_async(create_browser, args=(i))

pool.close()
pool.join()

However the browser never opens. I constantly get this error below:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Admin\Desktop\PycharmProjects\snkrs\Nike\_test.py", line 14, in <module>
    pool = Pool(processes=10)  # Maximum number of browsers opened at same time
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 212, in __init__
    self._repopulate_pool()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
AMC
  • 2,642
  • 7
  • 13
  • 35
Rhkruz03
  • 57
  • 1
  • 4
  • Does this answer your question? [RuntimeError on windows trying python multiprocessing](https://stackoverflow.com/questions/18204782/runtimeerror-on-windows-trying-python-multiprocessing) – AMC Sep 18 '20 at 20:38
  • Im going to try it right now. I was looking all over – Rhkruz03 Sep 18 '20 at 20:48

1 Answers1

-1

The problem is that each of your sub-processes is re-exeecuting the code that creates the process pool and its sub-processes in a recursive, never-ending loop. Try running your process-creating code in a special block as follows:

if __name__ == '__main__':
    pool = Pool(processes=10)  # Maximum number of browsers opened at same time

    for i in range(0, 2):  # Five browsers will be created
        #async_result = pool.apply_async(create_browser, args=(i))
        # args must be an iterable. (i) is not a tuple but (i,) is:
        async_result = pool.apply_async(create_browser, args=(i,))


    pool.close()
    pool.join()

To process results:

if __name__ == '__main__':
    with Pool(2) as pool: # don't create more processes than you need
        # submit tasks:
        result_objects = [pool.apply_async(create_browser, args=(i,) for i in range(2)]
        # wait for tasks to complete and get reurn values
        for result_object in result_objects:
            return_value = result_object.get()
            print(return_value)
        # no need to do close or join

Or simpler:

if __name__ == '__main__':
    with Pool(2) as pool: # don't create more processes than you need
        return_values = pool.map(create_browser, range(2))
        for return_value in return_values:
            print(return_value)
Booboo
  • 38,656
  • 3
  • 37
  • 60