2

I need to run a python script (a log parser) on hundreds of log files, but I'm a PHP guy so I figured I'd write a little PHP script to grab the list of files from a directory and call the python script dynamically in a foreach loop.

I've set variables in my PHP script using the full system paths to the python binary, the full path to the python script and checked that everything seems correct. I echo the output of the script I'm trying to run to check it over:

<?php
// batch.php (modified for SO post)
$python = '/usr/bin/python';
$script = '/mnt/data/scripts/the/python/script.py';
// $folder contains the full system path to the dir containing
// the files I need to pass as args to script.py
$files = scandir($folder);
foreach($files as $file){
    if($file=='..'||$file=='.')
    {
        continue;
    }
    $system = $python.' '.$script.' '.$folder.$file.' 2>&1';
    echo "Running ". $system ."\n";
   // I also tried passthru( )
   system($system);
}

Next I do

php batch.php

All I get is the first line from PHP:

Running /usr/bin/python /mnt/data/scripts/the/python/script.py /path/to/data/file/one.log

I can copy, paste and run the output echoed in shell (after 'Running ') directly with python and there's my output, no problem, so I know the PHP script has no syntax problems.

But when running the php script wrapping it, it produces no output other than my echo( ) statement from PHP. It just hangs there (I am thinking that my long-running Python script is actually working, but I'm not sure how to tell.) There's nothing in the error log, and the script never exits until I Ctrl-C.

I've seen a lot of discussions about exec( ), system( ) and passthru( ) and from what I can tell I should be seeing output using system( ) but for some reason I'm not.

I even tried to 

root:~# ps aux | grep php

and then 

root:~# strace -p <process_id>

of the PHP script, but all I get is

root:~# strace -p 14232
Process 14232 attached - interrupt to quit
read(4,

Note: I added the 2>&1 bit from this question but it didn't help; that references Apache however I'm running PHP on the CLI.

Note: 

root:~# echo $PYTHONPATH

in shell produces no output.

What am I missing?

Community
  • 1
  • 1
phpguru
  • 2,351
  • 2
  • 23
  • 33
  • codes look fine to me, maybe the python script doesn't output anything at all? – hago Jul 26 '13 at 07:24
  • @hago Actually, I checked back on the script running after several hours and it does actually produce output -- at the end of running the script -- it's the in-progress updates I am not seeing. I guess the python script does some magic to stdout while running that is different from normal echo. I was hoping to be able to monitor it but at least I can tell it's working. Maybe I'll dig into the python script a bit and see what its doing as it's working. – phpguru Jul 26 '13 at 14:36

1 Answers1

1

python buffers output by default. If your script terminates prematurely (possibly thanks to a PHP script timeout), the buffer is not flushed.

Call set_time_limit() to extend the timeout, and set environment variable PYTHONUNBUFFERED to a nonempty string, or run python with -u.

Seva Alekseyev
  • 59,826
  • 25
  • 160
  • 281