0

I am developing an app that reads a video, converts it to images and then processes them in pairs. Since the app will be running on a Raspberry Pi, memory there's not much RAM available.

If i run the tasks in a simple for loop without the Executor memory behaves just fine but if i use the Executor scheduling 50 jobs, for example, memory gets stuck around 800Mb and that chunk never gets released. The problematic code is as follows:

executor = ThreadPoolExecutor(max workers = 1)
future_list = []
#Read converted images from directory(around 100 images in gray scale)
images = imread_collection(image_directory)

for i in range(0,len(images)-1,2):
    img1 = images[i][:,:,0]
    img2 = images[i+1][:,:,0]
    future = executor.submit(process,xStart,yStart,xLen,yLen,img1,img2,i)
    future_list.append(future)

#Wait or all tasks to finish
executor.shutdown(wait=True)

Memory consumption without Executor

Memory consumption using ThreadPool Executor

I've tested this in Mac and a Linux laptop before sending the code to the Raspberry. This issue is present in Linux but not in Mac.

Gustavo Gonzalez
  • 43
  • 1
  • 7
  • 1
    Are you `join`-ing the threads before shutdown? AFAIK, there is a little known implementation detail that thread local state (thread local variables) only get released when `join` is called. – danny Jun 16 '17 at 16:23
  • Possible duplicate of https://stackoverflow.com/q/44403132/7414759 – stovfl Jun 16 '17 at 17:59
  • Possibly related, [Will malloc implementations return free-ed memory back to the system?](https://stackoverflow.com/q/2215259/608639) – jww Jun 17 '17 at 11:12

0 Answers0