0

Excuse the non-descriptive title, as I don't really understand what is causing the issue. I have this class which extends Thread. The skeleton of this class is:

genetic.py:

class Genetic(Thread):
    def __init__(self, waypoints, msg, landing_time, takeoff_time, flight_speed, max_flight_time, population_size, breeding_pool_size_percent, selection_pool_size):
        Thread.__init__(self)
        self.exit_event = Event()
        ...

    def stop(self) -> None:
        print("Stopping genetic algorithm")

        if self.exit_event.is_set():
            print("Already stopped")
            return

        # set the exit event
        self.exit_event.set()

        # join thread
        self.join()

    def run(self) -> None:        
        # create a population of randomly generated solutions
        print("Initializing population")
        self.population = self.initialize_population()

        # temporary best solution
        self.best_solution = self.population[0]
        last_best_solution = None

        stop_count: int = 0
        iteration: int = 1

        # this is main loop of the algorithm. theoretically, the more iteration it goes through, the better the solutions in the population become
        while stop_count < 8 and not self.exit_event.is_set():
            print(f"Genetic algorithm iteration {iteration}: ".ljust(40) + f"${self.best_solution.total_revenue} at {self.best_solution.total_time} minutes")

            self.population = self.breed()

            def mutate_population(population: list[Solution], new_population) -> list[Solution]:
                for i in range(len(population)):
                    population[i] = self.mutate(population[i])
                new_population.extend(population)

            jobs = []
            process_count = 4
            manager = mp.Manager()
            new_population = manager.list()
            for i in range(0, len(self.population), len(self.population) // process_count):
                p = mp.Process(target=mutate_population, args=(self.population[i:i + len(self.population) // process_count], new_population))
                jobs.append(p)
                p.start()
            for job in jobs:
                job.join()
            self.population = list(new_population)

            self.best_solution = max(self.population)

            if self.best_solution != last_best_solution:
                stop_count = 0
            
            last_best_solution = self.best_solution
            stop_count += 1
            iteration += 1

This class is being instantiated from another file main.py:

print("Creating optimiser")
optimiser = Genetic(waypointDict,
                    msg,
                    LANDING_TIME,
                    TAKEOFF_TIME,
                    FLIGHT_SPEED,
                    MAX_FLIGHT_TIME,
                    population_size=320,
                    breeding_pool_size_percent=0.4,
                    selection_pool_size=15)

print("Starting optimiser")
optimiser.start()

print("Waiting for optimiser to finish")
while optimiser.is_alive():
    time.sleep(0.1)

Note that I have two other classes: Solution and Route, but they are mostly to structure data, so I don't think they are relevant.

On a linux system (WSL or bare metal install), the code works as expected:

Creating optimiser
Starting optimiser
Initializing population
Waiting for optimiser to finish
Genetic algorithm iteration 1:          $3522 at 1749.9238703534118 minutes
Genetic algorithm iteration 2:          $5144 at 1759.3837782009025 minutes
Genetic algorithm iteration 3:          $5144 at 1759.3837782009025 minutes
Genetic algorithm iteration 4:          $5549 at 1745.1499777213191 minutes

However, on my Windows machine, the output is unexpected:

Creating optimiser
Starting optimiser
Initializing population
Waiting for optimiser to finish
Genetic algorithm iteration 1:          $2052 at 1734.0651969910905 minutes
Creating optimiser
Starting optimiser
Initializing population
Waiting for optimiser to finish
Genetic algorithm iteration 1:          $3155 at 1752.2125752177699 minutes
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python39\lib\threading.py", line 954, in _bootstrap_inner
    self.run()
  File "D:\project\genetic.py", line 201, in run
    manager = mp.Manager()
  File "C:\Python39\lib\multiprocessing\context.py", line 57, in Manager
    m.start()
  File "C:\Python39\lib\multiprocessing\managers.py", line 553, in start
    self._process.start()
  File "C:\Python39\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Python39\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "C:\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Average time:  6.617425441741943
Average routes:  13.0
Average money:  3155.0
flight time:  1752.2125752177699
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python39\lib\threading.py", line 954, in _bootstrap_inner
    self.run()
  File "D:\project\genetic.py", line 206, in run
    p.start()
  File "C:\Python39\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Python39\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Python39\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Python39\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Python39\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Genetic.run.<locals>.mutate_population'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python39\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Python39\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid
Average time:  13.808059215545654
Average routes:  11.0
Average money:  2052.0
flight time:  1734.0651969910905

Two things stand out to me in this result:

  1. It seems like somehow, two instances of my Genetic class were created.
  2. The AttributeError: Can't pickle local object 'Genetic.run.<locals>.mutate_population' line seems to suggest that something cannot be pickled into a ListProxy

Both the tests were executed using python 3.9.5 in virtual environments. I tried adding only parts of my project I think are relevant, but let me know if more is needed.

Taking the advice from the error message, I tried adding freeze_support() to my run() function, but nothing changed.

Blobtoe
  • 11
  • 1
  • Does this answer your question? [Python multiprocessing linux windows difference](https://stackoverflow.com/questions/42148344/python-multiprocessing-linux-windows-difference) also [Compulsory usage of if __name__=="__main__" in windows while using multiprocessing](https://stackoverflow.com/questions/20360686/compulsory-usage-of-if-name-main-in-windows-while-using-multiprocessi) – Ahmed AEK Mar 30 '23 at 20:14

0 Answers0