Twitter Streaming: Try/Except for IndexError

Question

Background: I want to download specific parts of users tweets (e.g. username, id, expanded url, etc) from the twitter api. I am able to do so successfully.

Problem: Because not all users have expanded_url, I sometimes receive the following error:

IndexError: list index out of range

Goal: If such IndexError occur, skip and proceed to collect new tweets

I think one way to solve this problem is by using a try/except statement

Questions: Is a try/except statement a valid way to do achieve this goal? If so, how do I properly apply a try/except statement?

I have tried the following:

class StdOutListener(StreamListener):
         def on_data(self, data):
             t = json.loads(data)
             tweet_id = t['id_str'] 
             user_name = t['user']['name'] 
             try:
                 expanded_url = t['entities']['urls'][0]['expanded_url']
             except:
                 pass

But I get the following error:

UnboundLocalError: local variable 'expanded_url' referenced before assignment

I have searched around SO, and have a few examples similar to my question (UnboundLocalError: local variable 'url_request' referenced before assignment, UnboundLocalError: local variable 'url' referenced before assignment)

But I'm not sure how to directly change my code so that I can implement the try/except statement. I am also open to other ways to solve this problem. Thank you!

Your traceback probably shows the error on a different line because `expanded_url` never comes into existence if the `try` fails. One way would be to define a value for `expanded_url` in your `except` block e.g. `expanded_url = ''`. Then you'd need to change code further downstream to handle `url` being an empty string or some default value. Depends on your program in general. — roganjosh, Apr 17 '17 at 19:11

francisco sollima · Accepted Answer · 2017-04-17T21:32:36.800

2

When the block inside the try has an error, the block inside the except gets executed. If nothing happens in your except (it only passes) expanded_url is never defined. You should assign expanded_url inside the except block to a default value or empty strings, whatever it should be:

         try:
             expanded_url = t['entities']['urls'][0]['expanded_url']
         except:
             expanded_url = ''

edited Apr 17 '17 at 21:32

answered Apr 17 '17 at 19:11

francisco sollima

7,952
4
22
38

3

In some cases `t['entities']['urls'][0].get('expanded_url', None)` makes more sense than try/except statement – VeGABAU Apr 17 '17 at 19:13
Yes, but in this case, isn't the error caused because t['entities']['urls'] is empty? Since it's IndexError? – francisco sollima Apr 17 '17 at 19:19
thanks for the suggestion, Francisco! Since I am new to python, I'm not sure what warrants an appropriate default value. Do you have any suggestions or toy examples? – SFC Apr 17 '17 at 19:21
Thanks for the suggestion @VeGABAU. I tried it, and I receive an AttributeError: 'NoneType' object has no attribute 'get' – SFC Apr 17 '17 at 19:24
I would need to know how you're using the expanded_url afterwards. Like roganjosh said in the comment, you could go with an empty string and handle it later. Whenever you get that empty string you will know you had no data and you will collect new tweets. – francisco sollima Apr 17 '17 at 19:26

score 0 · Answer 2 · answered Apr 17 '17 at 19:14

The problem is that you are probably trying to use the expanded_url after the try/exceptblock but when your code throws the exception you don't create the variable. Try to either:

expanded_url = ''
try:
    expanded_url = ....
except:
    pass

Or:

try:
    expanded_url = ....
except:
    expanded_url = ''

Twitter Streaming: Try/Except for IndexError

2 Answers2