Using threading in flask to train a model parallelly

Question

from threading import Thread

def model_training():
    global status
    for i in range(1, 11):
        status = i
        sleep(1)

@app.route('/retrain/', methods=['GET', 'POST'])
def retrain():
    for th in threading.enumerate():
        if th.name == "training":
            print("Thread still running")
            message = "Training is in already in progress"
            break
    else:
        print("Starting a new job thread")
        message = "Training started"
        t1 = Thread(target=model_training, name="training")
        t1.start()

    return render_template('retrain.html', message=message, models=model_files, num_images=id)

I am trying to use threading in flask to train a model in parallel. Above is the sample code of what I am trying to do. With model_training() called in thread returns a status as its in a far loop and works fine as expected. But, I would like to train a model by using model_train() (below code). In this case it doesn't run in parallel. It waits for the model to complete all epochs.

def model_train():
    dataset, model_config, label_mapping = load_config()
    print("Training: start.")

    model = model_config.model

    model.fit()

    print("Training: end.")

Can someone guide me to a right direction? Am I missing something here? Thanks in advance

does your model.fit() use blocking non-daemon threads ? – Saeid Mohammadi Nejati Jul 18 '23 at 13:57 — Saeid Mohammadi Nejati, Jul 18 '23 at 13:57

score 1 · Answer 1 · answered Jul 18 '23 at 13:35

1

You are trying to achieve CPU-bound multithreading, which means trying to improve performance through multithreading. This is not possible (will not improve performance) in Python default implementation due to the GIL

However, you can perform multiprocessing, using multiprocessing library, or take advantage of your GPU.

answered Jul 18 '23 at 13:35

PoneyUHC

287
1
11

I am not trying to improve the model performance with threading here. I just want to run model when other tasks are performed in parallel. In this case, it would just go to return statement while model is still running. – iamkk Jul 19 '23 at 07:37

score 0 · Answer 2 · answered Jul 18 '23 at 13:00

If you want to train a model in parallel, you will need to use a different function that does not block the thread. like this:

def model_train_parallel(dataset, model_config, label_mapping):
    parts = np.array_split(dataset, 4)
    threads = []
    for part in parts:
        thread = Thread(target=model_train_part, args=(part, model_config, label_mapping))
        threads.append(thread)
        thread.start()

    for thread in threads:
        thread.join()

def model_train_part(part, model_config, label_mapping):
    model = model_config.model
    model.fit(part)

here it splits the dataset into 4 parts, then creates 4 threads that train a model on each part in parallel.

Thanks for sharing how to train a model in parallel. But don't want to train model in parallel. I want to perform other tasks when model in running. In this case, it would be to run return statement. — iamkk, Jul 19 '23 at 07:39

Using threading in flask to train a model parallelly

2 Answers2