Python: stop long running threads

Question

I have an example code below where I download several hundred files in different thread from a list of urls. The download of each file can take several minutes. But I want to cancel the download if it takes longer than 5 minutes. How can I modify the code below for that?

def download(url):
    # request below can take several minutes
    request.get(url)

threads = []
for url in [url1, url2, ...]:
    thread = threading.Thread(target=download, args=url)
    thread.start()
    threads.append(thread)
    
for thread in threads:
    thread.join(300)
    if thread.is_alive():
        # need to kill the thread here

Have you searched the internet? First result I got was this: https://www.geeksforgeeks.org/python-different-ways-to-kill-a-thread/ second result: https://stackoverflow.com/questions/323972/is-there-any-way-to-kill-a-thread — Cow, Apr 01 '22 at 10:22

score 1 · Answer 1 · answered Apr 01 '22 at 10:21

This code may can help you. You can call stop_thread to kill the process
Maybe there is a more elegant way, I will follow the question

import time
import ctypes
import inspect
import threading


def _async_raise(tid, exctype):
    """
    raise the exception, performs cleanup if needed
    """
    tid = ctypes.c_long(tid)
    if not inspect.isclass(exctype):
        exctype = type(exctype)
    res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, ctypes.py_object(exctype))
    if res == 0:
        raise ValueError("invalid thread id")
    elif res != 1:
        # if it returns a number greater than one, should call it again with exc=NULL to revert the effect
        ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, None)
        raise SystemError("PyThreadState_SetAsyncExc failed")


def stop_thread(thread):
    _async_raise(thread.ident, SystemExit)

score 0 · Answer 2 · answered Apr 11 '22 at 02:01

You can use 'python-worker' (link)

from worker import worker

@worker
def download(url):
    # request below can take several minutes
    request.get(url)

thread_workers = []
for url in [url1, url2, ...]:
    thread_worker = download(url)
    thread_workers.append(thread_worker)
    
for thread_worker in thread_workers:
    thread_worker.thread.join(300)
    if thread_workers.is_alive():
        thread_workers.abort()

your download function will automatically run as a thread.

Python: stop long running threads

2 Answers2