Excuse the non-descriptive title, as I don't really understand what is causing the issue. I have this class which extends Thread. The skeleton of this class is:
genetic.py:
class Genetic(Thread):
def __init__(self, waypoints, msg, landing_time, takeoff_time, flight_speed, max_flight_time, population_size, breeding_pool_size_percent, selection_pool_size):
Thread.__init__(self)
self.exit_event = Event()
...
def stop(self) -> None:
print("Stopping genetic algorithm")
if self.exit_event.is_set():
print("Already stopped")
return
# set the exit event
self.exit_event.set()
# join thread
self.join()
def run(self) -> None:
# create a population of randomly generated solutions
print("Initializing population")
self.population = self.initialize_population()
# temporary best solution
self.best_solution = self.population[0]
last_best_solution = None
stop_count: int = 0
iteration: int = 1
# this is main loop of the algorithm. theoretically, the more iteration it goes through, the better the solutions in the population become
while stop_count < 8 and not self.exit_event.is_set():
print(f"Genetic algorithm iteration {iteration}: ".ljust(40) + f"${self.best_solution.total_revenue} at {self.best_solution.total_time} minutes")
self.population = self.breed()
def mutate_population(population: list[Solution], new_population) -> list[Solution]:
for i in range(len(population)):
population[i] = self.mutate(population[i])
new_population.extend(population)
jobs = []
process_count = 4
manager = mp.Manager()
new_population = manager.list()
for i in range(0, len(self.population), len(self.population) // process_count):
p = mp.Process(target=mutate_population, args=(self.population[i:i + len(self.population) // process_count], new_population))
jobs.append(p)
p.start()
for job in jobs:
job.join()
self.population = list(new_population)
self.best_solution = max(self.population)
if self.best_solution != last_best_solution:
stop_count = 0
last_best_solution = self.best_solution
stop_count += 1
iteration += 1
This class is being instantiated from another file main.py:
print("Creating optimiser")
optimiser = Genetic(waypointDict,
msg,
LANDING_TIME,
TAKEOFF_TIME,
FLIGHT_SPEED,
MAX_FLIGHT_TIME,
population_size=320,
breeding_pool_size_percent=0.4,
selection_pool_size=15)
print("Starting optimiser")
optimiser.start()
print("Waiting for optimiser to finish")
while optimiser.is_alive():
time.sleep(0.1)
Note that I have two other classes: Solution
and Route
, but they are mostly to structure data, so I don't think they are relevant.
On a linux system (WSL or bare metal install), the code works as expected:
Creating optimiser
Starting optimiser
Initializing population
Waiting for optimiser to finish
Genetic algorithm iteration 1: $3522 at 1749.9238703534118 minutes
Genetic algorithm iteration 2: $5144 at 1759.3837782009025 minutes
Genetic algorithm iteration 3: $5144 at 1759.3837782009025 minutes
Genetic algorithm iteration 4: $5549 at 1745.1499777213191 minutes
However, on my Windows machine, the output is unexpected:
Creating optimiser
Starting optimiser
Initializing population
Waiting for optimiser to finish
Genetic algorithm iteration 1: $2052 at 1734.0651969910905 minutes
Creating optimiser
Starting optimiser
Initializing population
Waiting for optimiser to finish
Genetic algorithm iteration 1: $3155 at 1752.2125752177699 minutes
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Python39\lib\threading.py", line 954, in _bootstrap_inner
self.run()
File "D:\project\genetic.py", line 201, in run
manager = mp.Manager()
File "C:\Python39\lib\multiprocessing\context.py", line 57, in Manager
m.start()
File "C:\Python39\lib\multiprocessing\managers.py", line 553, in start
self._process.start()
File "C:\Python39\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Python39\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Average time: 6.617425441741943
Average routes: 13.0
Average money: 3155.0
flight time: 1752.2125752177699
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Python39\lib\threading.py", line 954, in _bootstrap_inner
self.run()
File "D:\project\genetic.py", line 206, in run
p.start()
File "C:\Python39\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Python39\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Python39\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Python39\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
reduction.dump(process_obj, to_child)
File "C:\Python39\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Genetic.run.<locals>.mutate_population'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python39\lib\multiprocessing\spawn.py", line 107, in spawn_main
new_handle = reduction.duplicate(pipe_handle,
File "C:\Python39\lib\multiprocessing\reduction.py", line 79, in duplicate
return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid
Average time: 13.808059215545654
Average routes: 11.0
Average money: 2052.0
flight time: 1734.0651969910905
Two things stand out to me in this result:
- It seems like somehow, two instances of my
Genetic
class were created. - The
AttributeError: Can't pickle local object 'Genetic.run.<locals>.mutate_population'
line seems to suggest that something cannot be pickled into a ListProxy
Both the tests were executed using python 3.9.5 in virtual environments. I tried adding only parts of my project I think are relevant, but let me know if more is needed.
Taking the advice from the error message, I tried adding freeze_support()
to my run()
function, but nothing changed.