Here I've found following example (this is modified version) of using semaphores with multiprocessing module:
#!/bin/env python
import multiprocessing
from time import sleep
import os
max_allowed_processes_in_critical_section=1
semaphore = multiprocessing.Semaphore(max_allowed_processes_in_critical_section)
def do_job(id):
# BEGINNING OF CRITICAL SECTION
with semaphore:
sleep(1)
print "#####################"
print "Inside locked semaphore"
print "PPID: %s" % os.getppid()
print "PID: %s" % os.getpid()
# END OF CRITICAL SECTION
print("Finished job")
def main():
pool = multiprocessing.Pool(6)
for job_id in range(6):
print("Starting job")
pool.apply_async(do_job, [job_id])
pool.close()
pool.join()
if __name__ == "__main__":
main()
As you can see in this context the semaphore is used with with
keyword. In general semaphores has two methods wait()
and signal()
. In python's threading
and multiprocessing
modules those methods are AFAIK equivalent to acquire()
and release()
. I've rewritten the code to this one, which uses acquire()
and release()
methods:
#!/bin/env python
import multiprocessing
from time import sleep
import os
max_allowed_processes_in_critical_section=1
semaphore = multiprocessing.Semaphore(max_allowed_processes_in_critical_section)
def do_job(id):
# BEGINNING OF CRITICAL SECTION
semaphore.acquire()
sleep(1)
print "#####################"
print "Inside locked semaphore"
print "PPID: %s" % os.getppid()
print "PID: %s" % os.getpid()
semaphore.release()
# END OF CRITICAL SECTION
print("Finished job")
def main():
pool = multiprocessing.Pool(6)
for job_id in range(6):
print("Starting job")
pool.apply_async(do_job, [job_id])
pool.close()
pool.join()
if __name__ == "__main__":
main()
From my previous question I've understood that when some method is used with with
keyword then __enter__()
and __exit__()
methods of context manager are called on entry (and exit respectively) from the body of the with
statement. So I assume that acquire()
is called inside __enter__()
and release()
is called inside __exit__()
.
Questions:
Is my assumption about calling
acquire()
from__enter__()
andrelease()
from__exit__()
correct?Can I see somehow what are
__enter__()
and__exit__()
methods doing in this example? I've also noticed that when I access not defined variables inwith
version I don't get any exception (it must be there some exception handling which simply suppress the errors).
E.G. this one does not throw an exception even if ppid
and pid
does not exists
print "#####################"
print "Inside locked semaphore"
print "PPID: %s" % ppid
print "PID: %s" % pid
- Is this approach of using semaphores to ensure exclusivity for one process in critical section correct?
- As a python beginner I did not catch why
BoundedSemaphore([value])
andSemaphore([value])
are nested underclass multiprocessing.managers.SyncManager
in the documentation can you please clarify this?