How to run limited number of Multithreads in loop using python script?

Question

Consider 20 video file in the raw video folder. Initially take only 5 video for fast forwarding by assigning them to 5 individual threads. The process should wait until the 5 threads are completing their task and after completion, the fast forwarded videos must be saved in converted folder and the raw videos must move to the new folder. But in each loop it has to process only 5 threads wait and process the next set of threads. Here is the code:

from moviepy.editor import *
import os
import glob
from natsort import natsorted
from threading import Thread
import datetime

def fast(path,thread_name):
    if os.path.splitext(path)[1] == '.mp4':
        print("Start : " + str(thread_name) + "  -  " + str(datetime.datetime.now()))
        clip = (VideoFileClip(path).fx(vfx.speedx, 5))
        clip.to_videofile('A:/Internship/Assignment-1/output_folders/multithread_fast_output/'+thread_name + '.mp4', codec='libx264')
        print("End : " + str(thread_name) + "  -  " + str(datetime.datetime.now()))

for i, filename in enumerate(glob.glob("A:/Internship/Assignment-1/videos/*.mp4"), 1):
    thread_name = f"t{i}"
    Thread(target=fast, args=(filename, thread_name)).start()

Have you tried something like ThreadPoolExecutor? https://docs.python.org/3/library/concurrent.futures.html — votelessbubble, Mar 11 '22 at 07:25
create 4 (or n) threads and get them to call themselves back when they have completed the task (i have done this process with myself to keep threads limited). — D.L, Mar 11 '22 at 07:57
I see. Then you can give it a try. A pool is good when you need to reuse threads. — votelessbubble, Mar 11 '22 at 07:58
Thank you sir! It got worked but i have slight modification in my question.. I m not getting how wait the threads until finishing the other threads work — Krushi .K, Mar 12 '22 at 15:08

votelessbubble · Accepted Answer · 2022-03-11T08:10:17.227

I have slightly modified your code to use ThreadPoolExecutor instead of Thread The idea here is to create a pool of 5 worker threads (since you want 5 active at a time) and then add fast function to the job queue. 5 thread will run the fast function and when any thread finishes executing the function, it will take up the next task in the queue.

from moviepy.editor import *
import os
import glob
from natsort import natsorted
import datetime

from concurrent.futures import ThreadPoolExecutor


def fast(path, thread_name):
    if os.path.splitext(path)[1] == '.mp4':
        print("Start : " + str(thread_name) + "  -  " + str(datetime.datetime.now()))
        clip = (VideoFileClip(path).fx(vfx.speedx, 5))
        clip.to_videofile('A:/Internship/Assignment-1/output_folders/multithread_fast_output/' + thread_name + '.mp4', codec='libx264')
        print("End : " + str(thread_name) + "  -  " + str(datetime.datetime.now()))


with ThreadPoolExecutor(5) as pool:
    for i, filename in enumerate(glob.glob("A:/Internship/Assignment-1/videos/*.mp4"), 1):
        pool.submit(fast, filename, f"t{i}")

How to run limited number of Multithreads in loop using python script?

1 Answers1