I am using the Emcee python package to fit some models to data. My python code is a wrapper to a separate piece of software. My code generates the input file to this software at each step in the MCMC chain.
For each iteration, I need to:
- Copy the folder containing the software to a new folder with a unique path
- Generate the input file within that folder
- Run the software
- Compare the output to the data
I am having trouble with Step 1. I have set up a function called model() which creates the new folder and input file for each iteration:
def model(theta):
id_number = random.randint(100000)
os.system('cp -R original/model/path/ new/model/path/{}/'.format(id_number))
# then actually generate input and run the model using theta...
The idea is that each walker copies the folder over with a unique ID, then generates an input file and runs the software from that particular unique location. However, the code doesn't seem to create a new folder for each individual walker. At the moment, the folder is successfully copied with a unique ID once, then all the walkers attempt to generate input files and run the software from that folder at the same time. I am not sure why this is.
The emcee part of my code is as follows:
from multiprocessing import Pool
import multiprocessing as mp
mp.set_start_method('fork')
def main(p0,nwalkers,niter,ndim,lnprob,data):
with Pool() as pool:
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=data, pool=pool)
pos, prob, state = sampler.run_mcmc(p0, niter, progress=True)
return sampler, pos, prob, state
sampler, pos, prob, state = main(p0, nwalkers, niter, ndim, lnprob, data)
A similar question was asked here, but I wasn't able to get this to work.