0

I'm new to inter-process communication and was looking for input and guidance about best way to achieve the following: I have a function that convert an image from one format to another and I want to do batch images conversions using multiple processes.

  • I have 1000 images that i want to convert from one format to another
  • I create 6 child processes and wanted to send them the image file name to process: one child one image to convert.

Should I simply create 6 message queue and send one to each of the process or it's better to create one message queue storing a vector of image names and send the vector to the 6 child process ?

How can I communicate back the result of one child process to the parent process ? And how i can keep sending work to the process until i'm done with the 1000 files ?

I'm using C++, Windows and learning Boost.

Thanks

Jeff

Jeff Lacoste
  • 15
  • 4

1 Answers1

0

Do you need multiple processes or is it possible to use a single process with multiple threads? Here is a solution with multiple threads (check https://computing.llnl.gov/tutorials/pthreads/):

  1. Create a worker thread pool, with the six (or more) worker threads.
  2. Start the threads. They should stop on a mutex, waiting for a condition. The condition is: input queue is not empty.
  3. Load data (e.g. image handles, file names) to the input queue on the main thread (synchronize its access) and notify the worker threads.
  4. Each worker thread should access the queue and remove the last element. You need to synchronize access to the queue.
  5. When a thread finishes processing the image, it should store the processed data (e.g. converted image handle, file name) on an output queue (again the access should also be synchronized).
  6. Job is finished when input queue is empty and there is no more data to add there. output queue has all processed image data.
fhsilva
  • 1,217
  • 1
  • 9
  • 17
  • Thanks 'fhsilva' for your response. Yes I do need multiple process because the file conversion uses a 3rd party library that is not thread safe. This is the reason why i needed multiple process to work around the thread safety issue. The approach you listed appears to apply to processes as well. I was wondering if there was some sample code to do this as i wanted to keep the processes busy and as soon as done with one image, receive next image to process. – Jeff Lacoste Oct 16 '14 at 11:21
  • First I suggest you verify what makes the library not thread safe. Perhaps you can work around that with the threads approach. Note that if the library writes something to the filesystem, you may have problems even with different processes. Other than that I recommend you take a look at [MSDN](http://msdn.microsoft.com/en-us/library/windows/desktop/aa365574(v=vs.85).aspx). Check also this useful [answer](http://stackoverflow.com/questions/7127242/fastest-ipc-method-on-windows-7). – fhsilva Oct 16 '14 at 12:57
  • Thanks again. I do not have source code of the 3rd party library, so i can't look into and try to make it thread safe. – Jeff Lacoste Oct 16 '14 at 13:37
  • I didn't mean for you to modify the library. I mean that if you have some documentation perhaps indicating which calls are not thread safe maybe you could use synchronization on your code to make your threads work properly. – fhsilva Oct 16 '14 at 13:41
  • Basically the call to convert an image by itself is not thread safe, so if i protect it with a mutex for ex., my program will be simply sequential as one thread converting an image will lock the conversion of other images. – Jeff Lacoste Oct 16 '14 at 14:11