0

I have a script processing large text files. I am limited however by the size of the files. I've done some searching on this forum, and i've come to the conclusion that i must process the file line by line, however this brings up quite a bit of issues for me, as i need some detailed info from the file, before i can start processing it.

I've tried adding each line to a variable as below:

$content = "";
$handle = fopen($target_File, "r") or die("Couldn't get handle");
if ($handle) {
    while (!feof($handle)) {
        $buffer = fgets($handle, 256);
        // Process buffer here..
        $content .= $buffer;
    }
    fclose($handle);
}

And just as expected, it did not work.

Could anyone help me out ?

KasperFA
  • 33
  • 4
  • Possible duplicate of [How to increase memory limit for PHP over 2GB?](https://stackoverflow.com/questions/11885191/how-to-increase-memory-limit-for-php-over-2gb) – unalignedmemoryaccess Jul 03 '17 at 09:11
  • 1
    Well just adding the lines one by one to a string variable, is of course going to need roughly the same amount of memory, as if you had read it into a string in one go in the first place ... _“however this brings up quite a bit of issues for me, as i need some detailed info from the file, before i can start processing it”_ – well then you need to give us more details on that ... – CBroe Jul 03 '17 at 09:11
  • The textfile is a gcode and the file contains information regarding how many layers etc. there is. It is this info i need to start doing my calculations. – KasperFA Jul 03 '17 at 09:17
  • Maybe you could read the file line by line (as you already do) and create a "plan" of your file. E.g. "layer1" data starts in file line 23 and so on. Later on you can directly access this line and read only the part you really need. – user1915746 Jul 03 '17 at 09:23

2 Answers2

1

add this on top of your file

ini_set("memory_limit", -1);

Be careful, the page can use all the RAM on the server

MadPapo
  • 495
  • 4
  • 13
  • I've tried that - My Host does not allow any more than 1024M - regardless of my setting :/ And it still times out because of memmory (Out of memory (allocated 811859968) (tried to allocate 72 bytes)) – KasperFA Jul 03 '17 at 09:15
  • Sorry i've done mistake. ini_set("memory_limit", -1) – MadPapo Jul 03 '17 at 09:42
0
$content = "";
$handle = fopen($target_File, "r") or die("Couldn't get handle");
if ($handle) {
    while (($buffer = fgets($handle, 4096))!== false) {
        // Process buffer here..
        $content .= $buffer;
    }
    fclose($handle);
}

OR

$content = "";
$target_File ="/tmp/uploadfile.txt";
$handle = fopen($target_File, "r") or die("Couldn't get handle");

if ($handle) {
    while (!feof($handle)) {
        $buffer = fgets($handle, 4096);
        // Process buffer here.. 
        //$content .= $buffer;
    }
    fclose($handle);
}

NOTE:

If the problem is caused by hitting the memory limit, you can try setting it a higher value (this could work or not depending on php's configuration).

this sets the memory limit to 32 Mb

ini_set("memory_limit","32M");

RïshïKêsh Kümar
  • 4,734
  • 1
  • 24
  • 36