0

I need some help with checking current download progress with yt-dlp being run in separate process (by using subprocess) without halting main program. I have decided to use subprocess, because I would like to run up to 3 (or more) downloads a time. I know that I can use communicate() method to check download progress, but by doing so program execution will be stuck until given process would finish downloading. I know that yt-dlp constantly prints current download progress to the console output, but I don't know how can I retrieve that on demand, when I am using multiprocessing here

At the end of the second code example I've included my pseudo code, which is of course not working, but I would like to achieve something like that.

Here is my code example:

yt_dlp_downloader.py

from yt_dlp import YoutubeDL

import time
import sys


def main(idx, url):
    print('~~~~~~~~~~~~~~~~~~~~~~~~~~~~')
    print(f'Starting process no.{idx}')
    print('url:', url)
    print('~~~~~~~~~~~~~~~~~~~~~~~~~~~~')

    URLS = [url]
    ydl_opts = {
        'format': 'mp4',
        'ffmpeg-location': r'C:\ffmpeg\bin',
        'paths': {
            'home': r'C:\path_to_script\video_downloader\yt_dlp_workdir',
            'temp': r'C:\path_to_script\video_downloader\yt_dlp_workdir\tmp',
        },
        'outtmpl': r'C:\path_to_script\video_downloader\yt_dlp_workdir\{}.%(ext)s'.format(idx)
    }
    try:
        with YoutubeDL(ydl_opts) as ydl:
            ydl.download(URLS)
    except (Exception, ) as e:
        print(e)
        return False
    return True


if __name__== '__main__':
    my_input = sys.argv
    main(idx=my_input[1], url=my_input[2])

And here is a code responsible for launching subprocesses:

downloader.py

import subprocess
import time


def main():
    DETACHED_PROCESS = 0x00000008

    urls = ['https://url1.com/a1', 'https://url2.com/a2', 'https://url3.com/a3', 'https://url4.com/a4','https://url5.com/a5', 'https://url6.com/a6', 'https://url7.com/a7', 'https://url8.com/a8',]

    processes = []
    idx = 0
    while idx < 8:
        while len(processes) < 4:
            print(f'beggining of the iteration no. {idx}')
            process = subprocess.Popen(['python', 'yt_dlp_downloader.py', str(idx), urls[idx], f'filename_{idx}'],
                                         creationflags=DETACHED_PROCESS)
            processes.append(process)
            idx += 1

        # Start of pseudocode
        # start checking download progress every 10 seconds
        check_idx = 0
        while len(processes) == 3:
            if processes[check_idx].status == 'completed':
                del processes[check_idx]

            elif processes[check_idx].status == 'downloading':
                print(processes[check_idx].status['download_percentage'])
                check_idx += 1

            if check_idx == 3:
                check_idx = 0
                time.sleep(10)

if __name__ == '__main__':
    main()

I am just a beginner when it comes to multiprocessing, so please bear with me.

Best regards!

CrocoComp
  • 11
  • 3

0 Answers0