0

I'm using pthreads in a "work crew" pattern where I have a large number of commands to run, and I want to read the output from each command and merge all the outputs. Using pthreads I create one worker thread per core, and each worker thread essentially does the following:

for (;;) {
   const char *command = get_from_work_queue();
   if (command == NULL)
       break;
   FILE *fp = popen("long-running-command", "r");
   // ... loops reading fp until EOF
   // enqueue data read from fp
}

Both get_from_work_queue and "enqueue data" require the thread to temporarily acquire a mutex, as the queues are shared, but these mutexes are released before any call to popen() or fread().

My issue is that although I have four cores and four worker threads, only one long-running command is going at any one time. The other three threads are just sitting around, probably waiting to be scheduled.

I assume that something about popen or fread is blocking all the threads, and not just the caller. How can I read from a command without blocking other threads, so I can have four long-running commands going at the same time?

Norman Ramsey
  • 198,648
  • 61
  • 360
  • 533
  • Why not use `select` with a bunch of file descriptors? – Ed Heal Apr 20 '17 at 22:18
  • The accepted answer here: [popen() alternative](http://stackoverflow.com/questions/6743771/popen-alternative) may be worth a try to see if you get better/desired behavior. – Phil Brubaker Apr 20 '17 at 23:21
  • @PhilBrubaker I can't quite see why forking an extra shell would make a difference---and the truth is I *need* to run the shell. But it could be worth a try. – Norman Ramsey Apr 21 '17 at 00:36
  • @EdHeal coding with select(2) is a huge pain in the ass. I was really hoping for threads with non-blocking I/O. But I may be SOL. – Norman Ramsey Apr 21 '17 at 00:36
  • A [simple test](https://gist.github.com/keaston/ac6dfbdb0d494ab31d67755aeef01dfa) shows that having multiple `popen()` commands running simultaneously from multiple threads works fine. If you backtrace your threads in gdb, where are they stopped? – caf Apr 21 '17 at 12:33

1 Answers1

0

Turns out it works fine if I use read(2) instead of fread(3). For slightly more detail see my answer to What multithreading package for Lua "just works" as shipped?

Community
  • 1
  • 1
Norman Ramsey
  • 198,648
  • 61
  • 360
  • 533