Use muliprocessing or hadoop to speed up python-pandas script on big data

Question

I am treating a big CSV datafile which fileds are: user_id, timestamp, category, and I am building scores for each category for each user. I start by chunking the CSV file, and apply a groupby (on the two last figures of the user_id) on the chunks files so I can store a total of 100 files containing groups of users, and storing them in a HDF5 store.

Then I make a big for loop on my store to process each stored file, one after the other. For each one of them, I groupby on user_id, and then calculate the score for the user. Then I write an output CSV, with one line for each user and containing all his scores.

I noticed that this main loop takes 4 hours on my personal computer, and I want to accelerate it since it looks perfectly parallelizable. How can I? I thought of multiprocessing or hadoop streaming, what would be best?

Here is my (simplified) code:

def sub_group_hash(x):
    return x['user_id'].str[-2:]

reader = read_csv('input.csv', chunksize=500000)                                  
with get_store('grouped_input.h5') as store:
    for chunk in reader:
        groups = chunk.groupby(sub_group_hash(chunk))
        for grp, grouped in groups:
            store.append('group_%s' % grp, grouped,
                 data_columns=['user_id','timestamp','category'])

with open('stats.csv','wb') as outfile:
    spamwriter = csv.writer(outfile)
    with get_store('grouped_input.h5') as store:
        for grp in store.keys(): #this is the loop I would like to parallelize
            grouped = store.select(grp).groupby('user_id')
            for user, user_group in grouped:
                output = my_function(user,user_group)
                spamwriter.writerow([user] + output)

Answer 1

I would recommend multi-threading. The thread library is pretty easy and intuitive. https://docs.python.org/3/library/threading.html#thread-objects

I am a little confused on what you mean your main loop is but I am assuming its all of the above processes. If this is the case enclose it into a definition and using the more simple context of

import thread
t1 = threading.thread(process, ("any", "inputs"))
t1.start()

A decent tutorial can be found here. Which also shows you a more advanced threading technique if you are familiar enough with python to use it. http://www.tutorialspoint.com/python/python_multithreading.htm

The tricky thing is when you are writing to the file you don't want all of the processes to write to the files at once but luckily you can create a chock point with lock. The acquire() and release() function surrounding this process will make sure only one thread is writing at a time.

Also pay attention to how many cores you have available on your pc. If you run more threads then cores on your pc then each thread will have to wait for cpu time and you are not gaining that much in terms of speed. Also you can fork bomb your computer pretty easily if you create an infinite amount of processes.