I need some help with checking current download progress with yt-dlp
being run in separate process (by using subprocess) without halting main program. I have decided to use subprocess, because I would like to run up to 3 (or more) downloads a time. I know that I can use communicate()
method to check download progress, but by doing so program execution will be stuck until given process would finish downloading. I know that yt-dlp constantly prints current download progress to the console output, but I don't know how can I retrieve that on demand, when I am using multiprocessing here
At the end of the second code example I've included my pseudo code, which is of course not working, but I would like to achieve something like that.
Here is my code example:
yt_dlp_downloader.py
from yt_dlp import YoutubeDL
import time
import sys
def main(idx, url):
print('~~~~~~~~~~~~~~~~~~~~~~~~~~~~')
print(f'Starting process no.{idx}')
print('url:', url)
print('~~~~~~~~~~~~~~~~~~~~~~~~~~~~')
URLS = [url]
ydl_opts = {
'format': 'mp4',
'ffmpeg-location': r'C:\ffmpeg\bin',
'paths': {
'home': r'C:\path_to_script\video_downloader\yt_dlp_workdir',
'temp': r'C:\path_to_script\video_downloader\yt_dlp_workdir\tmp',
},
'outtmpl': r'C:\path_to_script\video_downloader\yt_dlp_workdir\{}.%(ext)s'.format(idx)
}
try:
with YoutubeDL(ydl_opts) as ydl:
ydl.download(URLS)
except (Exception, ) as e:
print(e)
return False
return True
if __name__== '__main__':
my_input = sys.argv
main(idx=my_input[1], url=my_input[2])
And here is a code responsible for launching subprocesses:
downloader.py
import subprocess
import time
def main():
DETACHED_PROCESS = 0x00000008
urls = ['https://url1.com/a1', 'https://url2.com/a2', 'https://url3.com/a3', 'https://url4.com/a4','https://url5.com/a5', 'https://url6.com/a6', 'https://url7.com/a7', 'https://url8.com/a8',]
processes = []
idx = 0
while idx < 8:
while len(processes) < 4:
print(f'beggining of the iteration no. {idx}')
process = subprocess.Popen(['python', 'yt_dlp_downloader.py', str(idx), urls[idx], f'filename_{idx}'],
creationflags=DETACHED_PROCESS)
processes.append(process)
idx += 1
# Start of pseudocode
# start checking download progress every 10 seconds
check_idx = 0
while len(processes) == 3:
if processes[check_idx].status == 'completed':
del processes[check_idx]
elif processes[check_idx].status == 'downloading':
print(processes[check_idx].status['download_percentage'])
check_idx += 1
if check_idx == 3:
check_idx = 0
time.sleep(10)
if __name__ == '__main__':
main()
I am just a beginner when it comes to multiprocessing, so please bear with me.
Best regards!