1

We have a Python Selenium Webdriver script that works perfectly when run from command line. However, the code seems to work differently when run by a cronjob, and the script hangs.

The python script scrolls the webpage through and stores the information. When running via cronjob the browser is scrolled to the bottom of the page and then the browser freezes and the script hangs there. No error messages are produced from this.

The cronjob fires a shell script, which contains the following:

#!/bin/sh
adddate() {
    while IFS= read -r line; do
        echo "$(date) $line"
    done
}

cd /home/user
export DISPLAY=:0.0
python webdriver.py 2>&1 | adddate >> /home/user/log

I doubt the problem is in the python code because it works perfectly when run from the command line. The cronjob belongs to the same user who is able to run the script from the command line. All the imports in python code seem to work just fine.

Edit: I'm scrolling down a page and javascript loads the content as you scroll down the page:

browser = webdriver.Firefox()
--
--
keep_scrolling = None
while keep_scrolling is None:
            try:
                keep_scrolling = browser.find_element_by_class_name('name_of_the_desired_class')
            except:
                browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")

It scrolls down the page, and when it finds the class name, it freezes... That is, when the script is run by cron (but it works just fine when run from command line).

Jaakko
  • 584
  • 6
  • 13
  • What specifically is the python script doing when it freezes? – Kenster May 06 '14 at 19:11
  • try simulating cron environment as described here: http://stackoverflow.com/questions/2135478/how-to-simulate-the-environment-cron-executes-a-script-with – isobolev May 06 '14 at 19:32
  • @isobolev I extracted the environmental variables with env command and then exported them in the script running the python file. That did the trick – Jaakko May 09 '14 at 10:47

0 Answers0