1

Specification of the problem:

I am running some complex tasks in python and to speed it up, I decided to use pythons multiprocessing library. It worked pretty well, but after some time I started to wonder, how much time are the Locks, I use, consuming and how much are the processes blocking each other.

The structure of processes is following:

One process that updates shared list between processes. Code of data update is something like this:

lock.acquire()
list_rex[0] = pat.some_list
list_rex[1] = pat.some_dictionary
lock.release()

where list_rex and lock are defined by 

list_rex = manager.list([[some_elements], {some_elements}])
lock = multi.Lock()

And then there are several processes, that once in a while updates their own memory space with these list.The code is as follows:

lock.acquire()
some_list = list_rex[0]
some_dict = list_rex[1]
lock.release()

some_list and some_dict are somehow related so I can not allow processes to have information in some_list from different update than is in some_dict.

And my question is, how fast are the methods acquire() and release()? In my case they can be called within seconds and sometimes milliseconds. And/Or is there some way how to avoid using locks in my case?

Thank you for your time.

EDIT: after considering your comment my question probably should be - how are proxy lists affecting my calculations? I use "some_list" and "some_dict" really a lot to be read from after the update.

Jendas
  • 3,359
  • 3
  • 27
  • 55
  • I don't have any reliable references, but my guess would be that using a proxy list could slow you down much more than the locks. See [this question](http://stackoverflow.com/q/13121790/1258041) of mine for an example. – Lev Levitsky Feb 10 '13 at 12:15
  • Hmm I didn't think of that. I thought that the locks are the bottleneck. I will do some more research on that. – Jendas Feb 10 '13 at 12:19
  • You can use [timeit](http://docs.python.org/2/library/timeit.html) module to measure execution time of a code. – Igonato Feb 10 '13 at 12:10

1 Answers1

1

I was about to write a really complicated answer involving reader-writer locks, atomic reference assignments not needing locks if you just used one object for both the list and dictionary together, and some other stuff about measuring performance before changing anything...

But then I took some time to look at what you are actually doing in your locks.... essentially nothing more than a couple reference assignments. Neither of which are blocking, or doing I/O, or anything "slow". It's just a couple of variable assignments. So the short answer is your acquire/release of the locks is likely negligible. Unless you entering the locks across dozens of threads a hundreds of times a second, the impact is negligible. But don't take my word for it - go measure it.

I'm more concerned what happens when some_list and some_dict are being referenced by the other processes outside of the lock while list_rex is being updated within the lock.

selbie
  • 100,020
  • 15
  • 103
  • 173
  • You are right! The time really is not considerable. I didn't measure it before because I thought that that has to be the problem. So I probably will have to consider comment mentioned upper. And to your concern - Why that should cause any problem? I know, lists are mutable, but I thought, that if if copy - some_dict = list_rex[1] - from proxy list, it will create pure copy of the inside and not just pass reference. – Jendas Feb 10 '13 at 12:40
  • 1
    Variable assignments of object types in Python are not deep copies! They are reference assignments to the **same** list. – selbie Feb 10 '13 at 12:49