Read and parse contents of very large file

Question

I am trying to parse a tab delimited file that is ~1GB in size.

Where I run the script i get:

Fatal error: Allowed memory size of 1895825408 bytes exhausted  (tried to allocate 1029206974 bytes) ...

My script at the moment is just:

$file = file_get_contents('allCountries.txt') ;

$file = str_replace(array("\r\n", "\t"), array("[NEW*LINE]", "[tAbul*Ator]"), $file) ;

I have set the memory limit in php.ini to -1, which then gives me:

Fatal error: Out of memory (allocated 1029963776) (tried to allocate 1029206974 bytes)

Is there anyway to partially open the file and then move on to the next part so less memory is used up at one time?

I'm surprised nobody mentioned generators, Please check my answer here: https://stackoverflow.com/a/54146788/1016425 — lloiacono, Jan 11 '19 at 12:54

score 13 · Accepted Answer · answered Feb 13 '13 at 08:04

13

Yes, you can read it line by line:

$handle = @fopen("/tmp/inputfile.txt", "r");
if ($handle) {
    while (($buffer = fgets($handle, 4096)) !== false) {
        echo $buffer;
    }
    fclose($handle);
}

answered Feb 13 '13 at 08:04

Ranty

3,333
3
22
24

1

For a tab-separated file, please use fgetcsv() for simplicity – Mark Baker Feb 13 '13 at 08:49

score 2 · Answer 2 · edited May 23 '17 at 10:29

2

You have to use blocks to read the file. Check the answer of this question. https://stackoverflow.com/a/6564818/1572528

You can also try to use this for less large files.

ini_set('memory_limit', '32M'); //max size 32m

edited May 23 '17 at 10:29

Community

1
1

answered Feb 13 '13 at 08:07

Jordi Kroon

2,607
3
31
55

score 1 · Answer 3 · answered Feb 13 '13 at 08:06

Are you sure that it's fopen that's failing and not your script's timeout setting? The default is usually around 30 seconds or so, and if your file is taking longer than that to read in, it may be tripping that up.

Another thing to consider may be the memory limit on your script - reading the file into an array may trip over this, so check your error log for memory warnings.

If neither of the above are your problem, you might look into using fgets to read the file in line-by-line, processing as you go.

$handle = fopen("/tmp/uploadfile.txt", "r") or die("Couldn't get handle");
if ($handle) {
    while (!feof($handle)) {
        $buffer = fgets($handle, 4096);
        // Process buffer here..
    }
    fclose($handle);
}

Edit

PHP doesn't seem to throw an error, it just returns false.

Is the path to $rawfile correct relative to where the script is running? Perhaps try setting an absolute path here for the filename.

If I just load the file into memory it happens in about 2 seconds, the error happens with the str_replace but I will try out the line by line suggestion. — imperium2335, Feb 13 '13 at 08:07
error happens with str_replace because in this case php try to copy same size of memory in to another variable — , Feb 13 '13 at 08:11

score 1 · Answer 4 · answered Feb 13 '13 at 08:07

Yes, use fopen and fread / fgets for this:

http://www.php.net/manual/en/function.fread.php

string fread ( resource $handle , int $length )

Set $length to how many of the file you want to read. The $handle saves the position for new reads then, with fseek you can also set the position later....

Read and parse contents of very large file

4 Answers4

Linked