Run os.system() multiple times simultaneously in Python?

Question

I've written the following short python script to download flv videos using cclive on a Fedora 17 system.

urls = [line.strip() for line in open("urls.txt")]
for url in urlstoget:
    os.system('cclive %s' % url)

It works fine but the videos are limited to about 80kbps. I have a 39 to download and would like to download 2-4 simultaneously.

How can I run the os.system() command multiple times simultaneously?

score 9 · Accepted Answer · edited May 23 '17 at 11:59

9

use either threading or multiprocessing.

Here's an example using multiprocessing:

def retrieve_url(url):
    os.system('cclive %s' % url)

pool = multiprocessing.Pool(4)
pool.map(retrieve_url, list_of_urls)

And a link to another SO question: Python - parallel commands

edited May 23 '17 at 11:59

Community

answered Aug 24 '12 at 18:17

mgilson

Thanks this works perfectly. Well the screen output progress bars are all stacked on top of each other at the bottom of the screen from cclive but its running 4 times. Thanks. – James Aug 24 '12 at 19:05

score 3 · Answer 2 · answered Aug 24 '12 at 18:27

3

Look at the subprocess module, the Popen() method in particular. You could also use os.fork()

answered Aug 24 '12 at 18:27

James Thiele

subprocess is an excellent solution here (In some ways better than my answer). I don't know why I didn't think of it ... – mgilson Aug 24 '12 at 18:30
Thanks I looked at subprocess but I've only been programming a couple of months and docs.python is as cryptic as ever. – James Aug 24 '12 at 19:06
@James -- you'll get used to the format of the docs if you stick with it. They're actually quite easy to read/use once you get the hang of it. – mgilson Aug 24 '12 at 19:07

2 Answers2