I'm working on a project where I calculate multiple models with fbprophet. Fbprohet has the problem, that there is no default way to surpress the messages of pystan. So there is a workaround mentioned on github to use this as decorator:
class suppress_stdout_stderr(object):
def __init__(self):
# Open a pair of null files
self.null_fds = [os.open(os.devnull, os.O_RDWR) for x in range(2)]
# Save the actual stdout (1) and stderr (2) file descriptors.
self.save_fds = (os.dup(1), os.dup(2))
def __enter__(self):
# Assign the null pointers to stdout and stderr.
os.dup2(self.null_fds[0], 1)
os.dup2(self.null_fds[1], 2)
def __exit__(self, *_):
# Re-assign the real stdout/stderr back to (1) and (2)
os.dup2(self.save_fds[0], 1)
os.dup2(self.save_fds[1], 2)
# Close the null files
os.close(self.null_fds[0])
os.close(self.null_fds[1])
I calculate the model with that command:
with suppress_stdout_stderr():
m = Prophet()
m.fit(history)
I'm doing that for every model about 250 times and have about 600 single models. So Prophet() will be called about 150.000 time:
class Model:
def __init__(stockticker):
self.ticker = stockticker
self.data = pd.read_excel(f'{self.ticker}.xlsx')
def get_residuals(my_stock_resid_distr_fitting):
for i in range(len(my_stock_resid_distr_fitting)):
with suppress_stdout_stderr():
m = Prophet()
m.fit(history)
def main():
self.get_residuals(200)
tickers = ['AAPL',....]
for ticker in tickers:
m = model(ticker)
m.main()
- list of stocktickers are about 600
After about 1024 iterations the script stops running with that error code:
OSError: [Errno 24] Too many open files
So as I understand the decorator correctly, it opens empty files to substitute the pystan warning. In my opinion the decorator closes the empty files in the function def __exit__(self, *_):. But this seems not to work correctly.
Does anybody sees the mistake or have another solution?
If not I would increase the max open files on my ubuntu server, but I hope you have a more pythonic way.
BR Lorenz
