IndexError: list index out of range [Python 3.x Web scraping]

Question

I am trying to grab geo locations using URLs from a csv by searching the twitter, tweets urls. The input file has more than 100K rows with bunch of columns.

I am using python 3.x anaconda with all the updated version and I am getting following error:

Traceback (most recent call last):
  File "__main__.py", line 21, in <module>
    location = get_location(userid)
  File "C:path\twitter_location.py", line 22, in get_location
    location = html.select('.ProfileHeaderCard-locationText')[0].text.strip()
IndexError: list index out of range

The code below :

#!/usr/env/bin python
import urllib.request
import urllib3
from bs4 import BeautifulSoup

def get_location(userid):
    '''
    Get location as string ('Paris', 'New york', ..) by scraping twitter profils page.
    Returns None if location can not be scrapped
    '''

    page_url = 'http://twitter.com/{0}'.format(userid)

    try:
        page = urllib.request.urlopen(page_url)
    except urllib.request.HTTPError:
        print ('ERROR: user {} not found'.format(userid))
        return None

    content = page.read()
    html = BeautifulSoup(content)
    location = html.select('.ProfileHeaderCard-locationText')[0].text.strip()

    if location.strip() == '':
        return None
    return location.strip()

I am looking for a quick fix so that I can execute the whole input files with more than 100k rows.

Edit: I As mentioned in the answer below, After including the try block the outputs have stopped grabbing geo location.

Before the inclusion of try block after certain count list out of range error.

After Including the try block the error is gone and so the coordinates. I am getting all none values.

Here is the DropBox link with Input, Before & After Output & entire code bundle.

Edit: II

Entire code and inputs are in the dropbox I am searching for some help where we can eliminate the entire API thing and find an alternative to pull geo locations of twitter usernames.

Appreciate the help in fixing the problem. Thanks in advance.

Answer 1

Well, you have an exception handling for HTTPErrors but there's no handling if there's no .ProfileHeaderCard-locationText. That's probably the issue. Now you can import/implement

import logging
logging.warning('Watch out!')  # will print a message to the console
logging.info('I told you so')  # will not print anything
logging.exception()

You can use this in all of your programs (and you should !). Just like you added a try, except block for ` try:

try:
    page = urllib.request.urlopen(page_url)
except urllib.request.HTTPError:
    print('ERROR: user {} not found'.format(userid))
    return None

you can do the same for

try:
    location = html.select('.ProfileHeaderCard-locationText')[0].text.strip()
except Exception:
    print("Error,Hey dud couldn't find Profile...")

The main problem might be that Google limits the usage of their API Using it your way. Much more convenient way is using Google-Maps-Python-API < CLICK FOR DETAILS Usage Example:

from geolocation.google_maps import GoogleMaps

address = "New York City Wall Street 12"

google_maps = GoogleMaps(api_key='your_google_maps_key') 

location = google_maps.search(location=address) # sends search to Google Maps.

print(location.all()) # returns all locations.

my_location = location.first() # returns only first location.

print(my_location.city)
print(my_location.route)
print(my_location.street_number)
print(my_location.postal_code)

EDIT:

    if location.strip() == '':
        return None
return location.strip()

I think you meant:

if location.strip()==None:
    return None
else:
    return location.strip()

Answer 2

While handling the 'index out of range' exception as suggested in Faulty Fuse's answer is important, this will only fix the symptom.

The root of the problem is that after a certain number of requests twitter blocks your IP(s) and stops sending any usable content. (they don't like such mass-queries).

Potential solutions:

1. Go slower. This will delay being blocked by Twitter. You need to go so slow you don't get blocked at all. While this might not be possible for 100k records, this would be an easy fix if you have the time to wait for results.

2. Use rotating proxies. Use many of them. ... or combine a handful of proxies with going somewhat slower.