0

I've written a PHP script which performs web scraping from one site and parse input for my website.

The script is driven by cronjob periodically, and everything is hosted in a shared web-server.

The problem is: my script terminates several times a day, with no error message and in random location in code each time.

The script is long, performing 2 HTTP Gets and 4 HTTP Posts to other country website, each HTTP request takes ~3 seconds to complete; it also writes to files and r/w to/from MySql database.

I'm stuck on it after trying the following things:

1) Talking with my hosting support (IxWebHosting) - they just wasted my time, denying their responsibility and advised me to limit the cronjob periodicy to 5 minute rate maximum (before it was 3 minutes interval, however it didn't change anything.)

2) Instead of running from cronjob context, I've switched to the following method: a. Cronjob calls a "loader PHP script" every 5 minutes. b. The "loader PHP script" calls the real PHP script using HTTP Get and terminates before waiting for an answer. c. The real PHP script perform its ~20 seconds job (here is where the program terminates in random location).

3) Put some log file timestamp writing in many places along the code in order to see where program terminated each run - this showed me the program terminates everywhere in the code.

4) In order to prove it's not my code fault I've performed the following test: a. Cronjob calls another loader PHP script. b. The PHP script performs HTTP request to a different testing-purpose PHP script and terminates without waiting for response. c. The 2nd PHP script will perform dummy 20 seconds task: sleep for a second and write timestamp into log file for 20 times.

Result: the test succeeded! the 2nd program didn't fail... which means it has something to do with my code and the webserver I'm running at - however since it fails everytime in different place and only ~10 times a day (from 288 times it runs a day) then I can't tell where it is (and no error message of PHP).

Thanks in advance, sorry for long description - I'll be happy to provide more details upon request.

Yizhar
  • 875
  • 9
  • 11
  • Do you have the same version of PHP on your host and test environment? – Ryan Dec 21 '12 at 15:00
  • How are you making these requests? Maybe your source is returning a 404 occasionally? – phpisuber01 Dec 21 '12 at 15:00
  • Answers: a) different PHP versions on the remote host and local test environment. b) I'm using CURL for HTTP requests, and always getting 200 response. It's just that program crashes sometime, no error code returened... – Yizhar Dec 23 '12 at 01:09

3 Answers3

1

Are you logging the actual process, rather than writing logs from within the process ? e.g. does your cron job look like:

* * * * * /home/user/myTroublesomeJob.php 2>&1 >/tmp/crash.log

This will catch the stdout/sterr of the process itself. It may also be worth invoking your script from a parent shell script, and that can catch the PHP process exiting, and dump out the exit code (which would indicate a core dump, a signal being caught etc.). See here for more info.

Community
  • 1
  • 1
Brian Agnew
  • 268,207
  • 37
  • 334
  • 440
  • Thanks, I've tried logging the cronjob stderr, it always empty. On the other hand the link you've pointed out for me might be quite useful - thanks! – Yizhar Dec 22 '12 at 00:48
0

Try setting the timeout at the start of the script. 

set_time_limit(1800);

I found recently that if I ran a script manually, it went fine, but if it was run by Cron, it would throw timeout errors. Putting this limit in helped.

aynber
  • 22,380
  • 8
  • 50
  • 63
0

If you are running a script on a Shared server then it will not allow you to run long running Scripts.

If you script takes time then please use a dedicated server. Because in shared server many user are using shared resources so server automatically kills a script which is using extra resources.

..I will suggest you to use amazon EC2 free package. There you will be able to run long running scripts.

Thanks

Jay.D
  • 21
  • 4