1

i made a downloader based on a scheduler. So I wanted to download files at starting at a time of and stop before 6 am. I managed to achieve this using apscheduler that starts at 12 am, but i am not sure how to terminate all the task at 6 am. I am doing multi threading as i need to download as many things as possible by 6 am. i tried scheduling a exit at 6 am but it doesn't stop the threads and only exits current thread. i want to exit all threads by 6 am. here is the code i tried ::

executor = ThreadPoolExecutor(max_workers=multiprocessing.cpu_count() * 5)
urls = [...] # list of all urls

def download(url):
    ... # downloader here
    
def main_download():
    futures = [executor.submit(download, url) for url in urls]
    for future in as_completed(futures):
        ... # do something
        
scheduler = apscheduler.schedulers.background.BlockingScheduler(timezone="Asia/Kolkata")
job = scheduler.add_job(main_download, trigger="cron", hour=12)
def kill_all(): # kill everything
    job.remove()
    scheduler.remove_all_jobs()
    scheduler.shutdown()
    quit(1)
    # already tried exit, raise keyboard interrupt, sys.exit
scheduler.add_job(kill_all, trigger="cron", hour=6) # kill everything
scheduler.start()

But it still continues the download everything, is there a good way to stop all threads? As this exists only current thread and other processes continues to run.

The Unknown Bengali
  • 35
  • 8

1 Answers1

1

The best solution might be using os._exit(n). exit, quit, and even sys.exit raises SystemExit which only terminates current thread and can be caught by handlers. Where as os._exit(n) exits the code without calling cleanup handlers, flushing stdio buffers. So your solution might just be using os._exit(n) But be warned it is not a good way to exit and should only be used in special cases. So use it carefully.

The difference between each of them has been explained briefly in the following stackoverflow thread

WarnerStark
  • 172
  • 11