How can I clear the memory while running a long PHP script? tried unset()

Question

My script imports an excel file into a product database to update quantities new products etc....

I am having memory issue and I have tried raising the memory limit to the max(800MB+). I have tried unsetting the variables in order to release the memory between the loops but I still run out of memory. I have tried setting the timeout to infinite but its definitely a memory issue.

Error msg from log file: Fatal error: Allowed memory size of 851443712 bytes exhausted (tried to allocate 71 bytes)

None of the script is contained in a function. If I create the main for loop inside a function and repeatedly call that function will that help garbage collection and clear up memory? Any help or guidance will be appreciated.

Import Script:

error_reporting( E_ALL & ~E_NOTICE );
ini_set('memory_limit', '812M');
set_time_limit(0);

/* Config Start */
define('BasePath', '/home/xxxxx/public_html');
define('CfgMagentoPath',                    BasePath);
define('CfgCategoryMapDBxls',                   BasePath."/xxxx/Shdddddd.xls");
define('CfgVenderDBxls',                    BasePath."/xxxx/xxxxxx.xls");
define('CfgReportEmail',                    "xxxxxx@gmail.com");
/* Config End */

require_once(CfgMagentoPath . '/app/Mage.php');
Mage::app(); 
//$app = Mage::app('default'); 
//Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);
require_once(BasePath.'/xxxxx/xxxx/libs/mage.func-inc.php');
require_once(BasePath.'/xxxxx/xxxxx/libs/excel-read.class.php');

//Alert Arrays
$AAnotmapped        = array();
$AAnewproducts  = array();
$AApriceupdated = array();
$AAimgerror         = array();
$PriceErrors        = array();

$SkipCat = false;

//Create Mapped Cats - In Magento

$excel = new ExcelReader(CfgCategoryMapDBxls,"UTF-8");
$CM = $excel->getWorksheetData('Sheet1');
if(!$SkipCat){
    echo "========   Generating Catagory Maps   ===========\n\n";
    CatMap_Create($CM);
    echo "======== ============================== ===========\n\n";
}

//Start Item Read
$excel = new ExcelReader(CfgVenderDBxls,"UTF-8");
$IT = $excel->getWorksheetData('New_DATA');
$ITcnt = 0;
$ITtotal = count($IT);

foreach($IT as $ItemRow){
    $ITcnt++;

    $cSKU                   = $ItemRow['ITEM'];
    $cProductName   = Clean_Data($ItemRow['ALTSHORTDESC']);
    $cCatName           = Clean_Data($ItemRow['CATEGORY']);
    $cManuf                 = Clean_Data($ItemRow['MANUFACTURER']);
    $cShortDesc         = Clean_Data($ItemRow['SHORTDESC']);
    $cLongDesc          = Clean_Data($ItemRow['LONGDESC']);
    $cUPC                       = Prod_GetUPC($ItemRow['UPC'], $ItemRow['ALTUPC']);
    $cStockQty          = $ItemRow['QTY'];
    $cWeight                = Prod_GetWeight($ItemRow['WEIGHT'], $ItemRow['ALTWEIGHT']);
    $cPrice                 = Prod_FigurePrice($ItemRow['COST'], $ItemRow['MSRP'], $ItemRow['MAP']);
    $cCost                  = $ItemRow['COST'];


    //Locate Catagory Map Magento ID
    $mCatId = CatMap_Search($CM, $ItemRow['CATEGORY']);

    //Now Create Product
    if($mCatId > 0 && $cProductName != ""){

        echo date("m.d.y g:i a")."\t($ITcnt / $ITtotal) Working On: " . $cProductName . " - SKU: $cSKU\n";
        $ProdID = Prod_GetIDfromSKU($cSKU);


        if($ProdID > 0){
            if(Prod_Update($ProdID, $cCost, $cStockQty, $cWeight, $cUPC)){
                echo "Updated: $cProductName\n";
                $ITindex++;
            }
        }else{
            Prod_Create($cSKU, $cProductName, $cManuf, $cPrice, $cCost, $cWeight, $cShortDesc, $cLongDesc, $cStockQty, $cUPC, $mCatId);
            echo "Created: $cProductName to Catagory: $mCatId\n";
            echo "$cShortDesc\n\n";
            $ProdID = Prod_GetIDfromSKU($cSKU);
        }


        if($cPrice <= $cCost){
            array_push($PriceErrors, "[$cSKU] $cProductName > Cost: $cCost | Price: $cPrice");  
            echo "Price Lower than Cost : Auto Inactive : Cost: $cCost | Price: $cPrice\n";
        }   

        Prod_AddImg($ProdID, $cSKU);

    }


    unset($ItemRow, $ProdID, $cSKU, $cProductName, $cManuf, $cPrice, $cCost, $cWeight, $cShortDesc, $cLongDesc, $cStockQty, $cUPC, $mCatId);
    echo "\n";  

}


echo "======== Disabling 0 Product Catagories ===========\n\n";
Cat_Disable_Empty($CM);
echo "======== ============================== ===========\n\n";

unset($CM, $IT, $excel);

//array_push($AAnotmapped, 'Cat not Mapped');
//array_push($AApriceupdated, '### Price Updated');
//array_push($AAimgerror , 'Image Error');

Send_Status_Email();

Mage_Reindex();


echo date("m.d.y g:i a")."\tCompleted\n\n";

//print_r($AAnotmapped);

//print_r($AApriceupdated);

//print_r($AAimgerror);

Unsetting the variables right before you reassign them gains you nothing here. What version of PHP? What's inside of some of these functions? Is your excel reader pulling data eager or lazy? Does it run out of memory before reaching mage_reindex? — Cory Carson, May 11 '12 at 03:32
Hi Cory, the PHP version is 5.2.9, the script gets through about ~3700-4000 products before it dies. I am not sure if it is lazy/eager and yes it runs out of memory before before calling mage_reindex. — Bob Jones, May 11 '12 at 13:31

score 30 · Answer 1 · edited May 23 '17 at 11:54

30

Use functions.
Use $var = null; instead of unset($var);. Unset simply kills the variable reference.

As mentioned on this comment:

When you are using unset, the memory will only be freed whenever garbage collector decides, but when you are setting a variable to a different value (null in this case), then you might get some memory freed of course with the cost of CPU.

edited May 23 '17 at 11:54

Community

1
1

answered May 11 '12 at 02:58

Hameed

2,227
1
21
31

11

@Michael That's why it's nicely called *unset*, not `kill_with_extreme_prejudice()`. :) – deceze May 11 '12 at 03:31
The point is, the memory won't get cleared with 'unset' until the garbage collection happens. – Hameed May 11 '12 at 03:56
1

that depends on paying taxes for the 'munis'. sorry, guys. working night. no more off-top. – Michael Sazonov May 11 '12 at 04:12
1

There's no difference whether you `unset` or `null` a variable, garbage collection is always responsible for removing the old data from memory. – deceze May 11 '12 at 05:57
1

When you are using unset, the memory will only be freed whenever garbage collector decides, but when you are setting a variable to a different value (null in this case), then you might get some memory freed of course with the cost of CPU. – Hameed May 11 '12 at 06:36
4

+1 After a long run of tests, using `null` presents better results than `unset`. Nonetheless, the issue remains, small chunks of objects by the hundreds per minute still crash the script. – Zuul Oct 01 '12 at 19:31
thanks, what i forgot ;) i have a debug log with 1 milion rows + sub arrays, the script its taken more then 10gb of memeory ;) now when i empty the array cache its max used 1,9gb when the script are running. – ParisNakitaKejser Aug 24 '15 at 08:47
In the PHP GC docs, it says this: "If we now call unset($a);, the variable container, including the type and value, will be removed from memory." I /think/ this is badly written english. I think it will NOT be removed from memory immediately (and the example right above suggests the same), but instead makes it available for collection when and if GC is run. Is that correct? reference https://www.php.net/manual/en/features.gc.refcounting-basics.php – Kevin Newman Feb 12 '20 at 17:26

Gilly · Answer 2 · 2016-06-29T10:55:19.373

Even when you use functions, you would expect the garbage collector to clean up everything in the scope of the function when the function returns.. This is not a guarantee and using functions might even work against you if you're fighting memory usage. Because of the scope, php has to create copies of the variables passed as parameters, which will only add up to the memory usage. You could look into passing references.

The garbage collector will only free up memory when there are CPU cycles available. Usually in loops it wont get a chance, so it will try to do this after the loop, in which case it might already be too late.

However you can force the garbage collector to do his round by calling gc_collect_cycles.

Also you can try debugging your code using memory_get_usage()

Passing by reference is a good idea for the large data parameters you may be passing. Just be careful not to edit that parameter in the function if you didn't intend to. Also, debugging is always.... a good idea. — arikin, Oct 06 '16 at 04:32

score -1 · Answer 3 · edited May 04 '23 at 23:30

Both unset($var) and $var = null can be used to free up memory in PHP, but they have different effects.

unset($var) removes the reference to the variable $var and frees up the memory that was used by the variable. This means that any subsequent attempts to access the variable will result in an "Undefined variable" notice.

On the other hand, $var = null assigns a null value to the variable $var and still keeps the variable allocated in memory. This means that the memory used by the variable is not immediately freed up, but it can still be accessed later in the code.

In general, using unset($var) is preferable if you want to free up memory immediately, while using $var = null is useful if you want to keep the variable allocated in memory but don't need to use its value anymore.

How can I clear the memory while running a long PHP script? tried unset()

3 Answers3

Linked

Related