Very long script keeps failing

Question

I have a script that updates my database with listings from eBay. The amount of sellers it grabs items from is always different and there are some sellers who have over 30,000 listings. I need to be able to grab all of these listings in one go.

I already have all the data pulling/storing working since I've created the client side app for this. Now I need an automated way to go through each seller in the DB and pull their listings.

My idea was to use CRON to execute the PHP script which will then populate the database.

I keep getting Internal Server Error pages when I'm trying to execute a script that takes a very long time to execute.

I've already set

    ini_set('memory_limit', '2G');
    set_time_limit(0);
    error_reporting(E_ALL);
    ini_set('display_errors', true);

in the script but it still keeps failing at about the 45 second mark. I've checked ini_get_all() and the settings are sticking.

Are there any other settings I need to adjust so that the script can run for as long as it needs to?

_“I keep getting Internal Server Error pages”_ – the standard ones … those that say _“more information about this error might be available in the server error log”_ …? — CBroe, Oct 30 '13 at 07:45
I was actually looking at the wrong error log file and wondering why nothing useful was showing up. — Vitaliy Isikov, Oct 30 '13 at 22:00

score 0 · Answer 1 · edited May 23 '17 at 11:50

Note the warnings from the set_time_limit function:

This function has no effect when PHP is running in safe mode. There is no workaround other than turning off safe mode or changing the time limit in the php.ini.

Are you running in safe mode? Try turning it off.

This is the bigger one:

The set_time_limit() function and the configuration directive max_execution_time only affect the execution time of the script itself. Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real.

Are you using external system calls to make the requests to eBay? or long calls to the database?

Look for particularly long operations by profiling your php script, and looking for long operations (> 45 seconds). Try to break those operations into smaller chunks.

safemode is off. There are no long calls to eBay or to the database. Each eBay call takes about 1-2 seconds to complete, it insert/updates the database, then asks for page 2 of eBay call. It's already all segmented to make sure no one specific action takes a long time. There's just a lot of actions in sequence. — Vitaliy Isikov, Oct 30 '13 at 05:49

score 0 · Accepted Answer · answered Oct 30 '13 at 21:58

Well, as it turns out, I overlooked the fact that I was testing the script through the browser. Which means Apache was handling the PHP process, which was executed with mod_fcgid, which had a timeout of exactly 45 seconds.

Executing the script directly from shell and CRON works just fine.

Very long script keeps failing

2 Answers2