Skip Connection Interruptions (Site & BeautifulSoup)

Question

I'm currently doing this with my script:

Get the body (from sourcecode) and search for a string, it does it until the string is found. (If the site updates.)

Altough, if the connection is lost, the script stops.

My 'connection' code looks something like this (This keeps repeating in a while loop every 20 seconds):

opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]

url = ('url')
openUrl = opener.open(url).read()

soup = BeautifulSoup(openUrl)

I've used urllib2 & BeautifulSoup.

Can anyone tell me how I could tell the script to "freeze" if the connection is lost and look to see if the internet connection is alive? Then continue based on the answer.(So, to check if the script CAN connect, not to see if the site is up. If it does checkings this way, the script will stop with a bunch of errors.)

Thank you!

score 1 · Answer 1 · edited Dec 21 '17 at 12:47

1

Found the solution!

So, I need to check the connection every LOOP, before actually doing stuff.

So I created this function:

def check_internet(self):
    try:
        header = {"pragma" : "no-cache"}
        req = urllib2.Request("http://www.google.ro", headers=header)
        response = urllib2.urlopen(req,timeout=2)
        return True
    except urllib2.URLError as err:
        return False

And it works, tested it with my connection down & up!

For the other newbies wodering:

while True:
     conn = check_internet('Site or just Google, just checking for connection.')
     try:
         if conn is True:
         #code
         else:
         #need to make it wait and re-do the while.
         time.sleep(30)
     except: urllib2.URLError as err:
         #need to wait
         time.sleep(20)

Works perfectly, the script has been running for about 10 hours now and it handles errors perfectly! It also works with my connection off and shows proper messages.

Open to suggestions for optimization!

edited Dec 21 '17 at 12:47

Community

1
1

answered Dec 04 '13 at 05:15

Daniel Crangu

107
2
12

1

Checking the connection is up is not important, checking if the _website is running_ is more important. So what you should really do is attempt to connect to your target website (not Google); and if the status code is anything that begins with 4 or 5, then wait. Otherwise, attempt to process the request. – Burhan Khalid Dec 04 '13 at 05:21
That's what the script is doing.Well, just without the response handling. But good note. I just forgot to change it to my site. I was a bit in the mindset of "I want to check my connection." Thanks a lot, I'll look up Responses and how to read them! – Daniel Crangu Dec 04 '13 at 05:27
Without response Handling...I think @BurhanKhalid means that you must check response (...website is running -> more important), if you don't then you're just checking connection (not so important) – Adriano Repetti Dec 13 '13 at 16:28

score 0 · Answer 2 · edited May 23 '17 at 12:25

0

Rather than "freeze" the script, I would have the script continue to run only if the connection is alive. If it's alive, run your code. If it's not alive, either attempt to reconnect, or halt execution.

while keepRunning:
   if connectionIsAlive():
      run_your_code()
   else:
      reconnect_maybe()

One way to check whether the connection is alive is described here Checking if a website is up via Python

If your program "stops with a bunch of errors" then that is likely because you're not properly handling the situation where you're unable to connect to the site (for various reasons such as you not having internet, their website is down, etc.).

You need to use a try/except block to make sure that you catch any errors that occur because you were unable to open a live connection.

try:
   openUrl = opener.open(url).read()
except urllib2.URLError:
   # something went wrong, how to respond?

edited May 23 '17 at 12:25

Community

1
1

answered Dec 03 '13 at 06:54

MxLDevs

19,048
36
123
194

Hello - Thank you but that's not what I was looking for and for good, I think I wasn't clear enough. I want Python to check, before all - if my internet connection is UP. If it checks the site's status without an internet connection, it again stops with a bunch of errors. I need it to say "Okay, connection's up" then it continues to do whatever I asked it to do. Thank you. – Daniel Crangu Dec 03 '13 at 13:35
You don't need to know whether you have internet or not, and that's not important. What's important is whether you're able to connect to that site or not. Checking whether you have internet connection or not can be as simple as trying to access a website that you know should be up 100% of the time like Google. – MxLDevs Dec 03 '13 at 16:41
Thanks - it was of much help but forgot to reply. – Daniel Crangu Dec 04 '13 at 05:20

Skip Connection Interruptions (Site & BeautifulSoup)

2 Answers2