Write data to start of file when not enough space

Question

The problem

How to write data to start of file if I have not enough space to allocate it in RAM and I have not enough space to make it's copy on current FS partition? I.e. I have a file with 100Mb size, I have 30Mb memory limit in my PHP script (and it can not be adjusted in any way) and I have only 50Mb free on my current FS partition. I want to add 2-10 rows to file (it's definitely less than remaining 50Mb FS space)

Some background

I know about XY-problem and agree that it's true for this case. But to reconsider this case I'll need to change significant part of current application (actually, it went from previous team) and, may be, API of other applications that using this file.

My attempt

I have not found solution for this yet. My previous approach was - to use some network buffer (i.e. to connect to some external storage, such as MySQL, for example - it's located on another machine where there is enough space to write file's copy)

The question

So, is it possible to write data to file's start when I have not enough space to allocate it in RAM and have not enough space to create file's copy on FS? Is using network (external) storage the only solution?

Take a look at these answers: [Q: Using php, how to insert text without overwriting to the beginning of a text file](http://stackoverflow.com/questions/103593/using-php-how-to-insert-text-without-overwriting-to-the-beginning-of-a-text-fil) — ComFreek, Sep 29 '13 at 11:32
@ComFreek - I've not found desired solution there, unfortunately - but thank you. There there are all options I've tried (even not-working case with `fopen($rFile, 'a+')` and `rewind` (same as `fseek`) — Alma Do, Sep 29 '13 at 11:41
FYI, I found a [C implementation](http://stackoverflow.com/a/10468278/603003), it seems that it does *not* load the whole file into memory. — ComFreek, Sep 29 '13 at 12:13

deceze · Answer 1 · 2013-09-29T12:08:46.340

Say you want to write 2K to the beginning of a file, your only real option is to:

open the file
read as much from the end of the file as you can fit into memory
write it back into the file 2K later than you started to read
continue with the previous block of data until you have shifted the entire content of the file 2K towards the end
write your 2K to the beginning

To visualize that:

|------------------------|

|-----------------XXXXXXX|
                  ------>

|-------------------XXXXXXX|

|----------XXXXXXX---------|
           ------>

|------------XXXXXXX-------|

...repeat...

Note that this is a very unsafe operation which edits the file in place. If the process crashes, you're left with a file in an inconsistent state. If you don't have enough room on disk to duplicate a file you arguably shouldn't work with that file and expand your storage capacity first.

Yes, I like this because it will _work_ at least. And I'm thinking exactly about case you've mentioned - what if writing process will be interrupted — Alma Do, Sep 29 '13 at 12:08
Thank you again - I was inspired by your idea and created safe method to do that. — Alma Do, Sep 29 '13 at 13:11

score 1 · Accepted Answer · answered Sep 29 '13 at 13:07

@deceze hint me great idea. So I've finished with:

function reverseFile($sIn, $sOut, $bRemoveSource=false)
{
        $rFile = @fopen($sIn, 'a+');
        $rTemp = @fopen($sOut,'a+');
        if(!$rFile || !$rTemp)
        {
                return false;
        }
        $iPos = filesize($sIn)-1;
        while($iPos>=0)
        {
                fseek($rFile, $iPos, SEEK_SET);
                fwrite($rTemp, $tmp=fread($rFile, 1));
                ftruncate($rFile, $iPos>0?$iPos:0);
                clearstatcache();
                $iPos--;
        }
        fclose($rFile);
        fclose($rTemp);
        if($bRemoveSource)
        {
                unlink($sIn);
        }
        return true;
}

function writeReverse($sFile, $sData, $sTemp=null)
{
        if(!isset($sTemp))
        {
                $sTemp=$sFile.'.rev';
        }
        if(reverseFile($sFile, $sTemp, 1))
        {
                file_put_contents($sTemp, strrev($sData), FILE_APPEND);
                return reverseFile($sTemp, $sFile, 1);
        }
        return false;
}

-it will be quite slow, but recoverable if process is interrupted (simply look to .rev file)

Thanks to all who participated in this.

score 0 · Answer 3 · edited May 23 '17 at 11:46

I've tried code suggested by @AlmaDo, don't try it on real projects, or you will be burn in hell, it is VERY slow. (60MB file - processing 19minutes)

You can run shell script - https://stackoverflow.com/a/9533736/2064576 (processed 420ms, can not understand how much memory does it use) Or try this php script - https://stackoverflow.com/a/16813550/2064576 (160ms, worked with memory_limit=3M, not worked with 2M)

Write data to start of file when not enough space

3 Answers3