Array limit issue

Question

I want to upload the csv file to the database so I am using word-press plug-in to do that . I have file size of 350 MB . Although I copied some data and save it to new file and now it has file size of 14 MB and total number of lines are 66872 .

When I try to upload that file the script don’t work after uploading 63296 lines of data in array . I check the forum and mostly say its a memory_limit issue . I even change the memory_limit = 2000M but it didn’t help .

Here is the code from plugin

    function csv_file_data($file, $delim) {
            $this->checkUploadDirPermission ();
            ini_set ( "auto_detect_line_endings", true );

            $data_rows = array ();
            $resource = fopen ( $file, 'r' );
            //print $file;  
            $init = 0;
            while ( $keys = fgetcsv ( $resource, '', $this->delim, '"' ) ) {
                    print $keys;
                    print $init;
                    if ($init == 0) {
                            $this->headers = $keys;
                    } else {
                            array_push ( $data_rows, $keys );
                    }

                    $init ++;
            }
            //print_r($data_rows);
            print $init;
            fclose ( $resource );
            ini_set ( "auto_detect_line_endings", false );
            return $data_rows;
    }

If this occurs during upload I would guess the problem is with [file uploads](http://php.net/upload). — Ja͢ck, Jul 03 '13 at 16:08
Are you trying to read the full file into an array? that's a bad idea for large files obviously, it should be done line by line. It could also be down to max_execution_time if the script runs too long, try upping that. — Andrew, Jul 03 '13 at 16:08
@Andrew yes its doing like that , i increase the max_execution_time but it didnt work — Uahmed, Jul 03 '13 at 16:18
@Jack i dont think its upload issue as i can able to get the data from file — Uahmed, Jul 03 '13 at 16:20
what error do you get, where does it quit? did you check php error log and enable display_errors ? — Andrew, Jul 03 '13 at 16:20
Are you sure the memory limit is actually being upped successfully? — Andrew, Jul 03 '13 at 16:22
@Andrew thanks for your message it give me this error Allowed memory size of 268435456 bytes exhausted (tried to allocate 35 bytes) , memory_limit = 8228M and i even restarted apache too but same error — Uahmed, Jul 03 '13 at 16:26
@jack so now what i did that i remove the array and try to upload the file having size 350MB and browser do processing for 2-3 sec and stop — Uahmed, Jul 03 '13 at 19:19

score 2 · Answer 1 · answered Jul 03 '13 at 16:19

2

You should not load entire file into the memory.

Here a correct example:

if (($handle = fopen("test.csv", "r")) !== FALSE) {
  while (($data = fgetcsv($handle, 2000, ",")) !== FALSE) {
    // Do your staff with  $data array
  }
  fclose($handle);
}

answered Jul 03 '13 at 16:19

Oleg Lemeshenko

68
4

Thanks for the reply , so it will keep loading 2000 rows until the file end ? please correct me if i am wrong – Uahmed Jul 03 '13 at 16:22
You are wrong. This code will read entire file. 2000 is a maximum string length, in php5 you can simply use fgetcsv($handle) to read string without string length limitation. – Oleg Lemeshenko Jul 03 '13 at 16:34
what if string length exceeds 2000 ? – Uahmed Jul 03 '13 at 16:42
fgetcsv will read first 2000. But as mentioned before you can use only 1 parameter to avoid limitation or use 0 in php >= 5.4 – Oleg Lemeshenko Jul 03 '13 at 16:57

Array limit issue

1 Answers1