Read pipe (C/C++), no error, but not all data

Question

In a C++ program, I want to fetch some data that a python program can easily provide. The C++ program invokes popen(), reads the data (a serialized protobuf) and continues on. This worked fine but has recently begun to fail with a shorter string received than sent.

I am trying to understand why I am not reading what I have written (despite no error reported) and how to generate further hypotheses. Fwiw, this is on linux (64 bit) and both processes are local. Python is 2.7.

(It's true the data size has gotten large (now 17MB where once 500 KB), but this should not lead to failure, although it's a sure signal I need to make some changes for the sake of efficiency.)

On the python side, I compute a dict of group_id mapping to group (a RegistrationProgress, cf. below):

payload = RegistrationProgressArray()
for group_id, group in groups.items():
    payload.group.add().CopyFrom(group)
payload.num_entries = len(groups)
print('{a}, {p}'.format(a=len(groups), p=len(payload.group)),
      file=sys.stderr)
print(payload.SerializeToString())
print('size={s}'.format(s=len(payload.SerializeToString())),
      file=sys.stderr)

Note that a and p match (correctly!) on the python side. The size will be about 17MB. On the C++ side,

string FetchProtoFromXXXXX<string>(const string& command_name) {
    ostringstream fetch_command;
    fetch_command << /* ... */ ;
    if (GetMode(kVerbose)) {
        cout << "FetchProtoFromXXXXX()" << endl;
        cout << endl << fetch_command.str() << endl << endl;
    }
    FILE* fp = popen(fetch_command.str().c_str(), "r");
    if (!fp) {
        perror(command_name.c_str());
        return "";
    }
    // There is, sadly, no even remotely portable way to create an
    // ifstream from a FILE* or a file descriptor.  So we do this the
    // C way, which is of course just fine.
    const int kBufferSize = 1 << 16;
    char c_buffer[kBufferSize];
    ostringstream buffer;
    while (!feof(fp) && !ferror(fp)) {
        size_t bytes_read = fread(c_buffer, 1, kBufferSize, fp);
        if (bytes_read < kBufferSize && ferror(fp)) {
            perror("FetchProtoFromXXXXX() failed");
            // Can we even continue?  Let's try, but expect that it
            // may set us up for future sadness when the protobuf
            // isn't readable.
        }
        buffer << c_buffer;
    }
    if (feof(fp) && GetMode(kVerbose)) {
        cout << "Read EOF from pipe" << endl;
    }
    int ret = pclose(fp);
    const string out_buffer(buffer.str());
    if (ret || GetMode(kVerbose)) {
        cout << "Pipe closed with exit status " << ret << endl;
        cout << "Read " << out_buffer.size() << " bytes." << endl;
    }
    return out_buffer;
}

)

The size will be about 144KB.

The protobuf I'm sending looks like this. The num_entries was a bit of paranoia, since it should be the same as group_size() which is the same as group().size().

message RegistrationProgress { ... }

message RegistrationProgressArray {
required int32 num_entries = 1;
repeated RegistrationProgress group = 2;
}

Then what I run is

array = FetchProtoFromXXXXX("my_command.py");
cout << "size=" << array.num_entries() << endl;
if (array.num_entries() != array.group_size()) {
    cout << "Something is wrong: array.num_entries() == "
         << array.num_entries()
         << " != array.group_size() == " << array.group_size()
         << " " << array.group().size()
         << endl;
    throw MyExceptionType();
}

and the output of running it is

122, 122
size=17106774
Read EOF from pipe
Pipe closed with exit status 0
Read 144831 bytes.
size=122
Something is wrong: array.num_entries() == 122 != array.focus_group_size() == 1 1

Inspecting the deserialized protobuf, it appears that group is an array of length one containing only the first element of the array I expected.

score 1 · Accepted Answer · answered Nov 30 '15 at 06:50

1

This...

buffer << c_buffer;

...requires that c_buffer contain ASCIIZ content, but in your case you're not NUL-terminating it.

Instead, make sure the exact number of bytes read are captured (even if there are embedded NULs):

buffer.write(c_buffer, bytes_read);

answered Nov 30 '15 at 06:50

Tony Delroy

102,968
15
177
252

This point is correct, but does not explain the output. It the stream contains embedded `'\0'` bytes, converting `buffer` to a string via `const string out_buffer(buffer.str());` will cut the output at the first null byte. – chqrlie Nov 30 '15 at 06:56
@chqrlie: seems you're only registering half the problem I described: `char c_buffer[kBufferSize];` isn't NUL terminated after a read, so any garbage characters after it on the stack will be written to `buffer` by `buffer << c_buffer`; it doesn't just potentially truncate early. And `const string out_buffer(buffer.str());` does *not* truncate at the first NUL (perhaps you were thinking of `c_str()`?). – Tony Delroy Nov 30 '15 at 07:00
You are correct., but your explanation is incomplete. The OP invokes undefined behavior *and* may forget portions of the output. – chqrlie Nov 30 '15 at 07:05
@chqrlie: that's true - I explain what's wrong and how to fix, and not what all the possible consequences of the existing wrong code, as I assume from the OP's code that they'd already understand that. But, suit yourself. – Tony Delroy Nov 30 '15 at 07:11
Indeed, how embarrassing. Thanks! – jma Nov 30 '15 at 07:32

score 1 · Answer 2 · edited May 23 '17 at 12:30

1

You catenate each chunk to the output buffer with this:

buffer << c_buffer;

As Tony D explains in his answer, you do not null terminate c_buffer before doing, so you invoke undefined behavior if c_buffer does not contain embedded null characters.

Conversely, if c_buffer does contain embedded null characters, portions of the stream are stripped and ignored.

Are you sure the streaming protocol does not contain embedded '\0' bytes?

You should also read Why is “while ( !feof (file) )” always wrong? although in your case, I don't think this is causing your problem.

edited May 23 '17 at 12:30

Community

1
1

answered Nov 30 '15 at 07:02

chqrlie

131,814
10
121
189

Thanks for the link, which is good reading. (You're correct that it is not at issue here.) – jma Nov 30 '15 at 10:57

Read pipe (C/C++), no error, but not all data

2 Answers2