0

Hello everyone,

Here is the situation, I have two sets of threads, Mappers, and Reducers

Mappers parse a text file and send individual words with there line numbers to a reducer thread based on a hash function

Reducers have a fixed buffer size and place each word in that buffer to a single list

I have Mappers and Reducer threads into separate Pthread arrays

For example:

[Step 1]  For the word "example" Mapper thread produces a hash value of 3
[Step 2]  Mapper thread puts "example" and its line number in the 4th 
          reducer's threads (0,1,2,3) buffer
[Step 3]  Reducer thread #4 will place "example" with its 
          line number to the list

Please inform me if this is not clear.

The Problem
I can not figure out how to pass information from the mapper thread to the appropriate user thread so Step 2.

Can anyone shed some light on how I can do this ?

I hope my description is clear enough however I can share some code if it helps.

ha9u63a7
  • 6,233
  • 16
  • 73
  • 108
Rstack
  • 45
  • 1
  • 10
  • did you see this? http://stackoverflow.com/questions/11352249/memory-sharing-between-c-threads – Lrrr Oct 29 '14 at 06:15
  • you could use condition variables to design a solution – Nik Oct 29 '14 at 06:15
  • it is almost consumer producer problem. you can solve it using thread-safe queue-s – qwr Oct 29 '14 at 06:21
  • What is wrong with existing MapReduce algorithms? E.g, QtConcurrent is an easy-to-use library that contains beside Threading also MapReduce algorithms. – sfrehse Oct 29 '14 at 06:32

1 Answers1

0

There are many ways to share data between threads. Your example suggests that you need some kind of pipeline. It means one thread push data into a shared data structure while the destination thread will pull data from the same structure. This data structure can be a vector or a list protected by a lock. It is also possible to use a lock free mechanism (see http://msdn.microsoft.com/en-us/library/windows/desktop/ms684121(v=vs.85).aspx). You can read more about pipelines here: http://msdn.microsoft.com/en-us/library/gg663538.aspx

Rogério Ramos
  • 286
  • 1
  • 5