0

My problem is time series anomaly detection and I use facebook prophet library. So I have a function called "fit_predict_model" and I have 90 different dataframes that I keep in the dictionary. I mean have 90 different models. Then it takes a long time to train. I wanted to use multiprocessing to train faster.But I am getting memory error. How can I solve this problem?

def fit_predict_model(dataframe, model_name, interval_width = 0.95, changepoint_range = 0.88):
    model = Prophet(yearly_seasonality=False,daily_seasonality=True, 
                    seasonality_mode = "multiplicative",changepoint_range = changepoint_range)
    
    model = model.fit(dataframe)

    forecast = model.predict(forecast)
    
    return forecast


pred = {}

def run(key):
    pred[key] = fit_predict_model(train[key], model_name = key)

pool = Pool(cpu_count())
pool.map(run, list(train.keys()))
pool.close()
pool.join()
klopscience
  • 1
  • Hello klopscience, are you running linux or windows? since it does matter when parallelizing with Pool. However prophet work with Stan, which already parallelize the work. Not sure if you will obtain better time with parallelizing the fitting process. – Ric Oct 26 '22 at 18:01
  • running windows environment – klopscience Oct 26 '22 at 18:44
  • See [this](https://stackoverflow.com/a/18205006/6912817). However I will not work in environments like Jupiter, should run standalone. – Ric Oct 26 '22 at 19:55
  • What is the exception you get? How much memory is used for a single thread? How large is your DataFrame? And what is the number of processes? – Jon Nordby Oct 28 '22 at 11:23
  • Where do the 90 different dataframes come from? Ie what computations is needed to get those? – Jon Nordby Oct 28 '22 at 11:25

0 Answers0