How to tell what a python script is doing while running?

Question

I wrote a script to scrape some web data in Python27. But with certain websites, the script gets stuck and seems to run forever.

Is there a way to follow what it is working on while the script is running?

I would like to understand where it is getting stuck, or if in fact it is doing something.

I'm new python. But I'm a little more familiar with Ruby on Rails. So I'm imagining something like watching the console in Rails while navigating the app after launching with 'rails s'.

http://stackoverflow.com/questions/1623039/python-debugging-tips any use? — JCx, Nov 26 '15 at 14:45

score 1 · Answer 1 · answered Nov 26 '15 at 21:50

You can set up and use a transparent proxy (burpsuite) and check the logs where the script gets stuck. Incase you are using requests module, use the following code.

http_proxy  = "localhost:8080"
https_proxy = "localhost:8080"
proxyDict = { 
              "http"  : http_proxy, 
              "https" : https_proxy
            }

r = requests.get(url, headers=headers, proxies=proxyDict)

Luis González · Answer 2 · 2015-11-27T07:43:42.393

If you are running the python script from the console, you could do something like this

example.py

while condition: 
    print "I'm inside the while loop"

After that, you are able to run

python example.py

And you must see "I'm inside the while loop" in the console till the condition become false. If you want print the value of a varible you have to follow the following steps:

variable      = 45
text_variable = "hello"

while condition: 
    print "{0} -- {1}".format(variable,text_variable)

Then you will see in the console

45 -- hello
45 -- hello 
# And so on, until the condition returns false

How to tell what a python script is doing while running?

2 Answers2