multithreading in python for loop

Question

I have a for loop where I am passing list of schema names to the loop. I want to execute these in parallel because there is no relation between these schemas.

I am using Python 2.6.

Creating dictionary here with schema information.

region_by_schema = {}
region_by_schema['UTEST_TT'] = 'tt'
region_by_schema['UTEST_V3'] = 'v3'
region_by_schema['UTEST_WRITE_SERVER'] = ''
region_by_schema['UTEST_PROPERTY'] = ''
region_by_schema['UTEST_PROJECT'] = ''

Looping through schema here:

for base_schema in region_by_schema.keys():
    do_one_user(base_schema, opt_password)

In do_one_user I am creating a directory with base_schema name and I am doing some work there.

So I want to execute these 5 schemas in parallel so that my work will finish fast. I want to put the code below in threads or multiprocessing:

for base_schema in region_by_schema.keys():
    do_one_user(base_schema, opt_password)

Please let me know the approach.

possible duplicate of [Python Multiprocessing a for loop](http://stackoverflow.com/questions/20190668/python-multiprocessing-a-for-loop) — three_pineapples, Jul 24 '14 at 05:55
Use multiprocessing: https://docs.python.org/2/library/multiprocessing.html — fileoffset, Jul 24 '14 at 05:56
It might be worth checking out iPython/iPython notebooks. They can be quite useful for [parallel computing](http://ipython.org/ipython-doc/dev/parallel/). — kylieCatt, Jul 24 '14 at 06:42

score 0 · Accepted Answer · edited Jul 24 '14 at 06:41

0

I suggest you read instruction of threading and learn how to use it. Here is link.

from threading import Thread

yourSchemas = [schema1, schema2, schema3]
t = []
def test(region_by_schema):
    for base_schema in region_by_schema.keys():
        do_one_user(base_schema, opt_password)

    for schema in yourSchemas:
        t.append(Thread(target=test, args=[schema]))

    for i in range(len(yourSchemas)):
        t.start()
        t.join()

edited Jul 24 '14 at 06:41

kylieCatt

10,672
5
43
51

answered Jul 24 '14 at 06:17

Stephen Lin

4,852
1
13
26

I tried this but it is still not executing parallel. – Satheesh Jul 24 '14 at 07:20
@user3743797 Could you please update your question by posting your latest code? – Stephen Lin Jul 24 '14 at 07:39
from threading import Thread yourSchemas = [schema1, schema2, schema3] t = [] def test(region_by_schema): for base_schema in region_by_schema.keys(): do_one_user(base_schema, opt_password) for schema in yourSchemas: myThread = Thread(target=test, args=[schema]) myThread.start() t.append(myThread) for i in range(len(yourSchemas)): t[i].join() This will work but now i am facing different issue, currently looking into this. Thanks for your help – Satheesh Jul 24 '14 at 11:18
@user3743797 If it works, please confirm it as an answer, thanks! – Stephen Lin Jul 24 '14 at 14:20
This is working but now i am facing another problem. – Satheesh Jul 25 '14 at 17:46
This is working but now i am facing another problem. Using some logic i created 5 directories and those directories are having subdirectories with some data. UTEST_TT, UTEST_V3, UTEST_WRITE_SERVER, UTEST_PROPERTY, UTEST_PROJECT, So now i want multithreading on these directories.In one thread i am changing the directory using os.chdir(./UTEST_TT/init) and in another thread i am changing it another directory. Because of this my process is failing. So is there a way i can change excecution of the process to current directory instead of changing the os path. – Satheesh Jul 25 '14 at 17:52
@user3743797 I didn't get your point.I suggest you open another question and post code you have tried so that we could see your problem clearly. It's way of working in Stackoverflow. – Stephen Lin Jul 26 '14 at 00:55

multithreading in python for loop

1 Answers1