Pipe buffer size in boost process

Question

I am using boost::process to read asynchronously the output of a console application in Windows. I noticed that the reading events is triggered after about 4k of data every-time.

If I set my buffer 'buf' to a small value nothing changes: the event is triggered multiple times ONLY after 4k of data has been transferred.

As per my understanding this could be a safe mechanism used in Windows to avoid dead-lock while reading from the pipe.

Is there any way in boost::process to change the size of the buffer used by the PIPE to transfer the data?

   #include <boost/process.hpp>
   #include <boost/asio.hpp>

   using namespace boost::process;
   boost::asio::io_service ios;

   std::vector<char> buf(200);
   async_pipe ap(ios);

   void read_from_buffer(const boost::system::error_code &ec, std::size_t size)
   {
    if (ec)
    {
        std::cout << "error" << std::endl;
        return;
    }

    std::cout << "--read-- " << size << std::endl;
    for (size_t i = 0; i < size; i++) std::cout << buf[i];
    std::cout << std::endl;


   ap.async_read_some(boost::asio::buffer(buf), read_from_buffer);
   }


   int main()
   {    
    child c("MyApp.exe --args", std_out > ap);  
    ap.async_read_some(boost::asio::buffer(buf), read_from_buffer);
    ios.run();
    int result = c.exit_code(); 
   }

Are you sure the buffer isn't in the console application generating the output? Most standard library implementations change their behavior to buffer more when stdout is not a console window. For a fuller explanation: https://www.codeproject.com/Articles/16163/Real-Time-Console-Output-Redirection — Dark Falcon, Aug 02 '18 at 16:16
Pipes have a maximum size of messages that can be transferred *atomically* - see `PIPE_BUF` - On Linux (for example) it is 4096 bytes afaik. — Jesper Juhl, Aug 02 '18 at 16:27
@DarkFalcon: the application generating the output flushes the text on screen at each line. I haven't seen any use of std::setvbuf to alter the normal behaviour so my assumption is that the application does not do anything esotheric. — Abruzzo Forte e Gentile, Aug 02 '18 at 16:30
@JesperJuhl: ..sorry I am not too experienced with Wndows..what is that PIPE_BUF that you mentioned and where I can find info about it? — Abruzzo Forte e Gentile, Aug 02 '18 at 16:31
PIPE_BUF is a Posix thing (see for example https://linux.die.net/man/7/pipe) but I believe Windows has something similar. As for where to find info, the generic answer to that is; use your favourite search engine. — Jesper Juhl, Aug 02 '18 at 16:33

score 1 · Answer 1 · answered Aug 02 '18 at 16:24

1

You might have to control the "sending" side (so, MyApp.exe).

On UNIX there's stdbuf (using setvbuff), unbuffer and similar. Tools might have some support built-in (e.g. grep --line-buffered).

On Windows, I'm not sure. Here's some pointers: Disable buffering on redirected stdout Pipe (Win32 API, C++)

answered Aug 02 '18 at 16:24

sehe

374,641
47
450
633

I don't have control on that as boost::process does all the calls to used pipes. My question was about how much control do I have on those pipe in boost::process. – Abruzzo Forte e Gentile Aug 02 '18 at 16:28
None. Also see the linked answer that might give control on the pipes connected to the underlying process even without access to the process. Whatever you'll find is proprietary and will not be a portable boost feature. But you can design and submit extensions. – sehe Aug 02 '18 at 16:34
@AbruzzoForteeGentile Yes you do have control, you have access to the pipe handle available through the `native_handle` api. – Michaël Roy Oct 30 '19 at 12:14
1

That doesn't at all imply you control the buffering on the sending side. You might, depending on the platform and implementation of the application. Do you mean you have it working for your case? If so, consider writing a small answer that might help other people in the future – sehe Oct 30 '19 at 21:33

Pipe buffer size in boost process

1 Answers1