4

Here I've found following example (this is modified version) of using semaphores with multiprocessing module:

#!/bin/env python

import multiprocessing
from time import sleep
import os

max_allowed_processes_in_critical_section=1
semaphore = multiprocessing.Semaphore(max_allowed_processes_in_critical_section)

def do_job(id):
    # BEGINNING OF CRITICAL SECTION
    with semaphore:
        sleep(1)
        print "#####################"
        print "Inside locked semaphore"
        print "PPID: %s" % os.getppid()
        print "PID:  %s" % os.getpid()

    # END OF CRITICAL SECTION
    print("Finished job")

def main():
    pool = multiprocessing.Pool(6)
    for job_id in range(6):
        print("Starting job")
        pool.apply_async(do_job, [job_id])
    pool.close()
    pool.join()

if __name__ == "__main__":
    main()

As you can see in this context the semaphore is used with with keyword. In general semaphores has two methods wait() and signal(). In python's threading and multiprocessing modules those methods are AFAIK equivalent to acquire() and release(). I've rewritten the code to this one, which uses acquire() and release() methods:

#!/bin/env python

import multiprocessing
from time import sleep
import os

max_allowed_processes_in_critical_section=1
semaphore = multiprocessing.Semaphore(max_allowed_processes_in_critical_section)

def do_job(id):
    # BEGINNING OF CRITICAL SECTION
    semaphore.acquire()
    sleep(1)
    print "#####################"
    print "Inside locked semaphore"
    print "PPID: %s" % os.getppid()
    print "PID:  %s" % os.getpid()
    semaphore.release()

    # END OF CRITICAL SECTION
    print("Finished job")

def main():
    pool = multiprocessing.Pool(6)
    for job_id in range(6):
        print("Starting job")
        pool.apply_async(do_job, [job_id])
    pool.close()
    pool.join()

if __name__ == "__main__":
    main()

From my previous question I've understood that when some method is used with with keyword then __enter__() and __exit__() methods of context manager are called on entry (and exit respectively) from the body of the with statement. So I assume that acquire() is called inside __enter__() and release() is called inside __exit__().

Questions:

  1. Is my assumption about calling acquire() from __enter__() and release() from __exit__() correct?

  2. Can I see somehow what are __enter__() and __exit__() methods doing in this example? I've also noticed that when I access not defined variables in with version I don't get any exception (it must be there some exception handling which simply suppress the errors).

E.G. this one does not throw an exception even if ppid and pid does not exists

print "#####################"
print "Inside locked semaphore"
print "PPID: %s" % ppid
print "PID:  %s" % pid
  1. Is this approach of using semaphores to ensure exclusivity for one process in critical section correct?
  2. As a python beginner I did not catch why BoundedSemaphore([value]) and Semaphore([value]) are nested under class multiprocessing.managers.SyncManager in the documentation can you please clarify this?
Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
Wakan Tanka
  • 7,542
  • 16
  • 69
  • 122

1 Answers1

0
  1. Yes.
  2. You can see this (if only indirectly) in the documentation. with blocks can suppress exceptions, but one for a semaphore shouldn’t do so. Perhaps there’s a complication in the context of (or there is (or was in 2015) a bug in) multiprocessing.
  3. That’s one of the uses of semaphores, yes.
  4. The names like Semaphore under SyncManager are functions to create synchronization objects backed by the manager—they’re named like classes as a hint that, like a class, they are called to manufacture new objects.
Davis Herring
  • 36,443
  • 4
  • 48
  • 76