0

I have a query selects all from the database table and writes it to a text file. If the state is small (say max of 200k rows), the code still works and writes it to the text file. Problem arises when I have a state that has 2M rows when queried, then there's also the fact that the table has 64 columns.

Here's a part of the code:

create and open file

$file = "file2.txt";
$fOpen = fopen($file, "a"); // Open file, write and append

$qry = "SELECT * FROM tbl_two WHERE STE='48'";
         
    $res = mysqli_query($con, $qry);
    if(!$res) {
        echo "No data record" . "<br/>";
    exit;
    }
    
$num_res =mysqli_num_rows($res);
for ($i=0; $i<=$num_res; $i++) {
    $row = mysqli_fetch_assoc ($res);

    $STATE = (trim($row['STATE'] === "") ? " " : $row['STATE']);
    $CTY   = (trim($row['CTY']=== "") ? "  " : $row['CTY']);
    $ST    = (trim($row['ST']=== "") ? "   " : $row['ST']);
    $BLK   = (trim($row['BLK']=== "") ? "      " : $row['BLK']);
   ....
   ....
   //64th column

   
    $data = "$STATE$CTY$ST$BLK(to the 64th variable)\r\n";

    fwrite($f,$data);
    
}

fclose($f);

I tried putting a limit to the query:

$qry = "SELECT * FROM tbl_two WHERE STE='48' LIMIT 200000";

Problem is, it just writes until the 200kth line, and it doesn't write the remaining 1.8m lines.

If I don't put a limit to the query, it encounters the error Out of memory .... . TIA for any kind suggestions.

Community
  • 1
  • 1
LadyWinter
  • 307
  • 6
  • 13
  • All possible answers http://stackoverflow.com/questions/2162497/efficiently-counting-the-number-of-lines-of-a-text-file-200mb – Ahmed Ginani Apr 29 '17 at 04:35
  • If you're doing this copy-paste stuff for 64 different variables you *really* need to look up how to use `array()` and loops. – tadman Apr 29 '17 at 04:37

1 Answers1

0

First you need to use buffer query for fetching the data Read it

Queries are using the buffered mode by default. This means that query results are immediately transferred from the MySQL Server to PHP and then are kept in the memory of the PHP process.

Unbuffered MySQL queries execute the query and then return a resource while the data is still waiting on the MySQL server for being fetched. This uses less memory on the PHP-side, but can increase the load on the server. Unless the full result set was fetched from the server no further queries can be sent over the same connection. Unbuffered queries can also be referred to as "use result".

NOTE: buffered queries should be used in cases where you expect only a limited result set or need to know the amount of returned rows before reading all rows. Unbuffered mode should be used when you expect larger results.

Also optimize the array try to put variable directly and you while loop only 

pdo = new PDO("mysql:host=localhost;dbname=world", 'my_user', 'my_pass');
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);
$uresult = $pdo->query("SELECT * FROM tbl_two WHERE STE='48' LIMIT 200000");
        if ($uresult) {
            $lineno = 0;
           while ($row = $uresult->fetch(PDO::FETCH_ASSOC)) {
               echo $row['Name'] . PHP_EOL;
           // write value in text file  
           $lineno++;
           }
        }
Ahmed Ginani
  • 6,522
  • 2
  • 15
  • 33