Download multiple files with gevent

Question

I am trying to download a list of files in parallel, making use of [gevent][1]

My code is a slight modification of the code suggested here

monkey.patch_all()

def download_xbrl_files(download_folder, yq, list_of_xbrl_urls):
    def download_and_save_file(url, yr, qtr):
        if url is not None:
            full_url = "http://edgar.sec.gov" + url
            if not os.path.exists(full_url):
                try:
                    content = urllib2.urlopen(full_url).read()
                    filename = download_folder + "/" + str(y) + "/" + q + "/" + url.split('/')[-1]
                    print "Saving: ", filename
                    f_raw = open(filename, "w")
                    f = FileObject(f_raw, "w")
                    try:
                        f.write(content)
                    finally:
                        f.close()
                        return 'Done'
                except:
                    print "Warning: can't save or access for item:", url
                    return None
            else:
                return 'Exists'
        else:
            return None
    (y, q) = yq
    if utls.has_elements(list_of_xbrl_urls):
        filter_for_none = filter(lambda x: x is not None, list_of_xbrl_urls)
        no_duplicates = list(set(filter_for_none))
        download_files = [gevent.spawn(lambda x: download_and_save_file(x, y, q), x) for x in no_duplicates]
        gevent.joinall(download_files)
        return 'completed'
    else:
        return 'empty'

What the code does is:

1. after some cleaning
2. gevent.spawn spawns download_and_save_file which:
3. checks if the file has been already downloaded
4. if not, downloads the content with urllib2.urlopen(full_url).read()
5. saves the file with the help of gevent's FileObject

I have the impression that the download_and_save only works sequentially. Furthermore, my application gets in a stand-by. I could add a timeout but I wanted to handle failures gracefully within my code.

Wondering if I am doing something wrong - it's the first time I write code in python.

EDIT

Here is a version of the code using "Threads"

def download_xbrl_files(download_folder, yq_and_url):
    (yq, url) = yq_and_url
    (yr, qtr) = yq
    if url is not None and url is not '':
        full_url = "http://edgar.sec.gov" + url
        filename = download_folder + "/" + str(yr) + "/" + qtr + "/" + url.split('/')[-1]
        if not os.path.exists(filename):
            try:
                content = urllib2.urlopen(full_url).read()
                print "Saving: ", filename
                f = open(filename, "wb")
                try:
                    f.write(content)
                    print "Writing done: ", filename
                finally:
                    f.close()
                    return 'Done'
            except:
                print "Warning: can't save or access for item:", url
                return None
        else:
            print "Exists: ", filename
            return 'Exists'
    else:
        return None


def download_filings(download_folder, yq_and_filings):
    threads = [threading.Thread(target=download_xbrl_files, args=(download_folder, x,)) for x in yq_and_filings]
    [thread.start() for thread in threads]
    [thread.join() for thread in threads]

Answer 1

I looked into this a little deeper the basic problem is that gevent.spawn() creates greenlets not processes (all greenlets run in a single OS thread).

Try a simple:

import gevent
from time import sleep
g = [gevent.spawn(sleep, 1) for x in range(100)]
gevent.joinall(g)

You'll see the time this takes is 100s. Which proves the point above.

You are really looking for multithreading which can be found in the threading module. Have a look at the the question at: How to use threading in Python?. For a little how to.

---update---

Here is a quick example of how you might do this:

threads = [threading.Thread(target=sleep, args=(1,)) for x in range(10)]
[thread.start() for thread in threads]
[thread.join() for thread in threads]