C - Reading then writing to a file without using memory (in place)?

Question

I need to 'move' a large string X spaces to the left. It is too big to fit in the memory, so I need to do it in place. I need to do it using a minimal amount of system calls.

I am aware that I can use buffers and reuse memory to minimize memory consumption, then use fseek -> Read -> Write till I'm done, but I'm interested to see if doing such a thing in place is possible.

It may depend on what he is considering 'memory'. `mmap` is limited to `size_t` bytes, and would certainly fit in memory unless he is referring to the stack. — David C. Rankin, Apr 22 '17 at 05:17

score 1 · Accepted Answer · edited May 23 '17 at 10:31

You can accomplish this by moving one byte at a time, as show below. However, you'll get much better performance if you allow for a larger buffer (moving 4096 bytes at a time). We obviously use some stack memory, but it makes no allocations based on the size of your prefix or the size of your file, so we can call this 'in place'.

void inPlaceTruncate(
    char const * const filename,
    int shift)
{
  FILE * f;
  if ((f = fopen(filename, "r+")) == NULL) {
    // handle error
  }

  // go to the end
  if (fseek(f, 0, SEEK_END) != 0) {
    // handle error
  }

  // get current file size
  long int const oldFileLen = ftell(f);
  if (oldFileLen < 0) {
    // handle error
  } else if (oldFileLen < shift) {
    // make the file empty
    shift = oldFileLen;
  }

  // go back to the beginning
  if (fseek(f, 0, SEEK_SET) != 0) {
    // handle error
  }

  // move file
  char buffer;
  for (long int pos = 0; pos < oldFileLen-shift; ++pos) {
    // slow to be repeatedly fseeking...
    if (fseek(f, pos+shift, SEEK_SET) != 0) {
      // handle error
    }

    if (fread(&buffer, sizeof(buffer), 1, f) != 1) {
      // handle error
    }

    if (fseek(f, pos, SEEK_SET) != 0) {
      // handle error
    }

    if (fwrite(&buffer, sizeof(buffer), 1, f) != 1) {
      // handle error
    }
  }

  // shrink file -- in a rather unpleasent way
  #ifdef WIN32
  if (_chsize(fileno(f), oldFileLen-shift) != 0) {
    // handle error
  }
  #else
  // we're assuming if its not windows, it's at least posix compliant.
  if (ftruncate(fileno(f), oldFileLen-shift) != 0) {
    // handle error
  }
  #endif

  fclose(f);
}

A related post for file shrinking.

Edited to actually answer the OP's question.

Edited to note places for error handling.

Also, as pointed in the comments, this will only handle files and shifts of less than 2GB. In order to handle larger files and work around the fseek()/ftell()/ftruncate() 32-bit limitation (on Windows you can use _chsize_s), you would need to determine the filesize in a loop using relative offsets, and make multiple calls to ftruncate().

Hmmm. This code moves everything to the right. It appears that OP wants to move the contents of a file to the left, making the file smaller. This is a different, and slightly trickier problem. — ad absurdum, Apr 22 '17 at 04:55
[`ftell()` returns `long`](http://port70.net/~nsz/c/c11/n1570.html#7.21.9.4), not `int`. — Andrew Henle, Apr 22 '17 at 11:58
Better. An `int` may not be able to hold the return value from `ftell()`; this is significant if the file is large enough. Thus, `oldFileLen`, `shift`, and `pos` should all be `long`. Also, the return value of `fopen()` should be checked to be certain the file opened successfully; all of those calls to `fseek()`, `fread()` and `fwrite()` should be checked for errors. — ad absurdum, Apr 22 '17 at 12:29
in general, the file size limit can be gotten around by using `lseek64()` rather than `lseek()` — user3629249, Apr 23 '17 at 18:06

C - Reading then writing to a file without using memory (in place)?

1 Answers1