3

I'm developing an application for embedded system running Linux.

In my case, I have a quite big file (compared to the capability of the system) as input. The file has a small header whose size is just a few hundred bytes. In my application, I need to remove that header from the file so the file will have no header and contain only relevant data. Normally, I'd implement like following (pseudo code):

char *input_file  = "big_input.bin";
char *tmp_file1 = "header.bin";
char *tmp_file2 = "data.bin";
/* Copy the content of header from input file to tmp_file1 */ 
_copy_header(tmp_file1, input_file); 
/* Copy the data from input file to tmp_file2 */ 
_copy_data(tmp_file2, input_file);
/* Rename temp file to input file */
unlink(input_file);
rename(tmp_file2, input_file);

The problem with this approach is that it creates a temporary file tmp_file2 whose size is almost as big as input file (because the header is very small). In my system, everything is stored on RAM, which is very limited. Creating a big temporary file causes an out-of-memory error.

So how can I avoid creating a big temporary file?

vvvvv
  • 25,404
  • 19
  • 49
  • 81
theman
  • 598
  • 1
  • 7
  • 19

3 Answers3

6

Open the same file twice, once for reading, once for writing.

Seek the read pointer behind the header.

Read from the read-pointer and write to the write-pointer.

Make sure the size you read and write at once is not larger than the header.

Cut off the header's size at the end of the file.

alk
  • 69,737
  • 10
  • 105
  • 255
3

Assuming you know the exact size of the header in advance, something like this should do it:

#define HEADER_SIZE 128

// size the buffer as appropriate for you RAM limits
char buffer[ 4096 ];
int fd = open( filename, O_RDWR );
size_t totalBytes = 0UL;
for ( ;; )
{
    ssize_t bytes_read = pread( fd, buffer,
        sizeof( buffer ), totalBytes + HEADER_SIZE );
    if ( bytes_read <= 0L )
    {
        break;
    }
    pwrite( fd, buffer, bytes_read, totalBytes );
    total_bytes += bytes_read;
}

ftruncate( fd, total_bytes );

close( fd );

You'll need to add the proper header files and some error checking.

Andrew Henle
  • 32,625
  • 3
  • 24
  • 56
1

In your case, you could

  • open the file in read-write
  • read char-by-char from your offset and write to the beginning, loop till the end of file (sounds suboptimal but you're on a RAM drive, and it's simple. Faster methods would imply read more bytes, maybe more complex to implement, and speed gain would have to be measured)
  • in the end you can truncate the file with truncate or ftruncate as explained here: How to truncate a file in C?
Community
  • 1
  • 1
Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
  • 4
    Reading char by char is maybe not the most efficient way. – Jabberwocky Nov 21 '16 at 16:25
  • from a RAM system that would work. And memory size is critical in that question. – Jean-François Fabre Nov 21 '16 at 16:26
  • 1
    The header is small, so I suppose he can have a read buffer of that length. – Jabberwocky Nov 21 '16 at 16:29
  • Or you could allocate a "resonably sized" temporary buffer (say a page long), and read and write that many bytes at a time. An alternative method (if virtual memory is supported) is to `mmap` the file to a virtual address, use `memmove` to move the data, then `munmap` it. – Ian Abbott Nov 21 '16 at 16:34
  • The header is small and I know its size in advance. So I can read to a buffer with length of the header as @MichaelWalz suggest. – theman Nov 22 '16 at 02:04