open a temporary C FILE* for input

Question

I have a legacy function accepting a FILE* pointer in a library. The contents I would like to parse is actually in memory, not on disk.

So I came up with the following steps to work around this issue:

• the data is in memory at this point
• fopen a temporary file (using tmpnam or tmpfile) on disk for writing
• fclose the file
• fopen the same file again for reading - guaranteed to exist
• change the buffer using setvbuf(buffer, size)
• do the legacy FILE* stuff
• close the file
• remove the temporary file
• the data can be discarded

On windows, it looks like this:

int bufferSize;
char buffer[bufferSize];

// set up the buffer here

// temporary file name
char tempName [L_tmpnam_s];
tmpnam_s(tempName, L_tmpnam_s);

// open/close/reopen
fopen_s(&fp, tempName,"wb");
fclose(fp);
freopen_s(&fp, tempName,"rb", fp);

// replace the internal buffer
setvbuf(fp, buffer, _IONBF, bufferSize);
fp->_ptr = buffer;
fp->_cnt = bufferSize;

// do the FILE* reading here

// close and remove tmp file
fclose(fp);
remove(tempName);

Works, but quite cumbersome. The main problem, aside from the backwardness of this approach, are:

• the temporary name needs to be determined
• the temporary file is actually written to disk
• the temporary file needs to be removed afterwards

I'd like to keep things portable, so using Windows memory-mapped functions or boost's facilities is not an option. The problem is mainly that, while it is possible to convert a FILE* to an std::fstream, the reverse seems to be impossible, or at least not supported on C++99.

All suggestions welcome!

Update 1

Using a pipe/fdopen/setvbuf as suggested by Speed8ump and a bit of twiddling seems to work. It does no longer create files on disk nor does it consume extra memory. One step closer, except, for some reason, setvbuf is not working as expected. Manually fixing it up is possible, but of course not portable.

// create a pipe for reading, do not allocate memory
int pipefd[2];
_pipe(pipefd, 0, _O_RDONLY | _O_BINARY);

// open the read pipe for binary reading as a file
fp = _fdopen(pipefd[0], "rb");

// try to switch the buffer ptr and size to our buffer, (no buffering)
setvbuf(fp, buffer, _IONBF, bufferSize);

// for some reason, setvbuf does not set the correct ptr/sizes      
fp->_ptr = buffer;
fp->_charbuf = fp->_bufsiz = fp->_cnt = bufferSize;

Update 2

Wow. So it seems that unless I dive into the MS-specific implementation CreateNamedPipe / CreateFileMapping, POSIX portability costs us an entire memcopy (of any size!), be it to file or into a pipe. Hopefully the compiler understands that this is just a temporary and optimizes this. Hopefully.

Still, we eliminated the silly device writing intermediate. Yay!

int pipefd[2];
pipe(pipefd, bufferSize, _O_BINARY);   // setting internal buffer size

FILE* in  = fdopen(pipefd[0], "rb");    
FILE* out = fdopen(pipefd[1], "wb");   

// the actual copy
fwrite(buffer, 1, bufferSize, out);  
fclose(out);

// fread(in), fseek(in), etc.. 

fclose(in);

Answer 1

You might try using a pipe and fdopen, that seems to be portable, is in-memory, and you might still be able to do the setvbuf trick you are using.

Answer 2

Your setvbuf hack is a nice idea, but not portable. C11 (n1570):

7.21.5.6 The setvbuf function

Synopsis

#include <stdio.h>
int setvbuf(FILE * restrict stream,
            char * restrict buf,
            int mode, size_t size);

Description

[...] If buf is not a null pointer, the array it points to may be used instead of a buffer allocated by the setvbuf function [...] and the argument size specifies the size of the array; otherwise, size may determine the size of a buffer allocated by the setvbuf function. The contents of the array at any time are indeterminate.

There is neither a guarantee that the provided buffer is used at all, nor about what it contains at any point after the setvbuf call until the file is closed or setvbuf is called again (POSIX doesn't give more guarantees).

The easiest portable solution, I think, is using tmpfile, fwrite the data into that file, fseek to the beginning (I'm not sure if temporary files are guaranteed to be seekable, on my Linux system, it appears they are, and I'd expect them to be elsewhere), and pass the FILE pointer to the function. This still requires copying in memory, but I guess usually no writing of the data to the disk (POSIX, unfortunately, implicitly requires a real file to exist). A file obtained by tmpfile is deleted after closing.