1

After read many post about "delete massive amount of rows", I tried several answer for my problem but not work like hope.

Explanation : I've been commendited to transcript a desktop application to a web app. The application have to main goal a update of several sql tables.

When I was working on it, I've been informed to change the truncate method to a delete method. So now we don't do a Truncate and Insert, but a Replace/Insert and Delete all rows not replace/insert into the table.

Here some code :

    $codeliste = "";
    $req2 = "SELECT projection.article.ean, projection.article.artic
    FROM projection.article
    WHERE projection.article.annule is null and projection.article.ean is not null";
    $req_res2 = mssql_query($req2);
    if (mssql_num_rows($req_res2) > 0) {
        $nbc = 0;
        while ($result = mssql_fetch_row($req_res2)) {
            if(trim($result[0]) <> "") {
                if($codeliste == "") $codeliste = "'".$result[0]."'";
                else $codeliste .= ",'". $result[0]."'";
                
                $req = "REPLACE INTO fean ( bar_code , article ) VALUES ('". trim($result[0]) ."','". trim($result[1]) ."')";
                if($connexion->exec($req)) {
                    $nbc++;
                }else{
                    echo "<br/>Fail Replace fean";
                }
            }
        }
        echo "<br/>Exported Ean : ".$nbc."<br/>";
    }else{
        echo "Fail request at projection.article of ean";
    }
    
    if($codeliste <> "") {
        
        $req = "DELETE FROM fean WHERE bar_code NOT IN (".$codeliste .")";
        if($connexion->exec($req)) {
            echo "Success delete Ean not export";
        }else{
            echo "Fail at DELETE fean";
        }
    }else{
        echo "codeliste is empty";
    }

More Information : I'm not allowed to truncate tables, I can't change the configuration of php.ini and conf.ini, I'm working on a delete up to 50.000~100.000 rows.

With the code above I obtain "SQLSTATE[HY000]: General error: 2006 MySQL server has gone away"

After that I tried tempory table, I obtain for result a loading page who never end (to be sure I waited 1h and no result from the page).

Then I tried to delete 5.000 rows by 5.000 rows with a loop, no more success.

Thanks in advance.


EDIT : I do some research and try to change the way I was deleting for a other way to the NOT IN in the code above, like CBroe suggest.

But I've got one more time "Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away'" when I load my page. Have you another idea of way to make this page Delete without gone away ?


EDIT2 : I've had some change in my code in the objectif to send less request to the database mysql. But I've been inform that my request have cause some lag issue because my delete request stay on sleep state in mysql. To be honest i don't know why, if someone could explain me why and maybe how I can avoid this problem in futur I would be grateful.

Here my code now : my goal was to reduce the number of request sql I send to the database.

        $codeliste = array();$codesuppr = array();
        $req2 = "SELECT projection.article.ean, projection.article.artic
        FROM projection.article
        WHERE projection.article.annule is null and projection.article.ean is not null";
        $req_res2 = mssql_query($req2);
        if (mssql_num_rows($req_res2) > 0) {
            $nbc = 0;
            while ($result = mssql_fetch_row($req_res2)) {
                if(trim($result[0]) <> "") {
                    array_push($codeliste, trim($result[0]));
                    $req = "REPLACE INTO fean ( bar_code , article ) VALUES ('". trim($result[0]) ."','". trim($result[1]) ."')";
                    if($connexion->exec($req)) {
                        $nbc++;
                    }else{
                        echo "<br/>Fail Replace fean";
                    }
                }
            }
        echo "<br/>Exported Ean : ".$nbc."<br/>";
    }else{
         echo "Fail request at projection.article of ean"
    }
    
    if (!empty($codeliste)) {
        $req = "SELECT bar_code FROM fean";
        $resultats = $connexion->query($req);
        foreach($resultats AS $resultat) {
            if (!in_array(trim($resultat), $codeliste)) {
                array_push($codesuppr, trim($resultat));
            }
        }
        if (!empty($codesuppr)) {
            $total= 0;
            $nbc = 0;
            $codeliste = array();
            foreach($codesuppr AS $code_barre) {
                if(count($codeliste) < 500 && ($total + count($codeliste)) < count($codesuppr)) {
                    array_push($codeliste, $bar_code);
                }else{
                    array_push($codeliste, $bar_code);
                    $req = "DELETE FROM fean WHERE bar_code IN ('".implode("','",$codeliste)."')";
                    if($connexion->exec($req)) {
                        $nbc++;
                    }else{
                         echo "Fail at DELETE fean";
                    }
                    $total .= count($codeliste);
                    $codeliste = array();
                }
            }
            echo "Success delete Ean not export";
        }
    }else{
        echo "codeliste is empty<br/>";
    }

EDIT3 : After looking mysql, it's seems Replace send too many request at mysql who send back : Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away', and finish in sleep command.

IMaj
  • 11
  • 2
  • Being a total transaction, why not download the database locally and modify it and then make a total upload? – Simone Rossaini Aug 30 '22 at 08:40
  • Because i need to do the same as the desktop app in the web page to abandon the desktop app. – IMaj Aug 30 '22 at 08:51
  • 3
    Not sure this code makes sense...you talk about MySQL and indeed show us a MySQL error message but `mssql_query` is a very old, legacy function for querying Microsoft SQL Server databases, not MySQL. – ADyson Aug 30 '22 at 08:51
  • @ADyson Yes because i get data from a database in Sql server then i send it in the database mysql. – IMaj Aug 30 '22 at 08:53
  • _"MySQL server has gone away"_ is probably the result of trying to stick 50k values into the `NOT IN` clause. But because that is negated, you won't be able to "chunk" this into smaller amounts. Perhaps you can turn the logic around, to get the actual codes of the items that you need to delete intead? – CBroe Aug 30 '22 at 08:59
  • If deletion code works, then move it to Queue/Schedulers – Abdulla Nilam Sep 06 '22 at 12:34
  • Best way is to flag the deleted rows and then run a cron to 'nibble' at them to either physically delete or archive them. Then work out the quickest max number of record to delete in say a 60 to 120 second window and run the cron every 5 mins. As mentioned too mssql_* functions arte for SQL Server, not MySQL! Regardless, should be using PDO. – Brian Sep 06 '22 at 12:58
  • It seems that your `IN()` query can get quite large. Perhaps [this question](https://stackoverflow.com/questions/4514697/mysql-in-operator-performance-on-large-number-of-values) (and its answers) is useful for you (like the suggestion regarding a temporary table for IDs). – RickN Sep 06 '22 at 13:00
  • @AbdullaNilam Sorry but it's seems to be a SQL Server (or Transact-SQL) method and indeed I use SQL Server but only to obtain data who's be `INSERT/REPLACE` in MySQL. – IMaj Sep 06 '22 at 13:01
  • 1
    PDO can be used with SQL Server too. – Brian Sep 06 '22 at 13:04
  • 1
    _"I'm not allowed to truncate tables"_ Why? Please explain the purpose behind this restriction so we can understand what else may be allowed or not allowed. Are you not allowed to truncate *any* table, or just the one in question? If the latter, then I can think of a solution involving temp tables. – kmoser Sep 06 '22 at 15:28
  • @kmoser I don't have permission to truncate existing table, but if I create a temp table I can truncate it. – IMaj Sep 07 '22 at 06:38
  • Yes but you were asked why you don't have permission, can't you ask for permission? It doesn't make much sense as a restriction – ADyson Sep 07 '22 at 06:47
  • @ADyson I'm not allowed to truncate because it's erase foreign keys with this table. Moreover, I'm not the only one working on this table. Example, someone send a request when I truncate, and got an error, where a replace with delete, can let's people send request when this code execute. – IMaj Sep 07 '22 at 07:04

0 Answers0