Invalid response (404) with Twitter API in Python (and in general)

Question

tl;dr:

I am trying to set headers in a Python HTTP request for the first time and am getting a 404. I would appreciate any help (see bottom).

I have been experimenting with the Twitter API and have not been having much luck. Eventually I am trying to get all of the media (photos) a user has posted (20 or 50 or whatever per fetch)

In my experience with other APIs, this process would go as follows: Get The userID, Make a get request to some endpoint using that userId, get a JSON feed response.

It seems to be much more complicated in Twitter.

For instance, I do not see any URLs where I can attach an access token or client ID. Instead, in their documentation they show a place where I can retrieve my

Comsumer Key, Consumer Secret, Access Token, and, Access Token Secret

If I enter my request URI and "query", it generates an oAuth Signature, which in this case consists of a

Signature base string, Authorization header and cURL command

This is where things get confusing. It says

Important: This will only be valid for a few minutes. Also remember the cURL command will actually execute the request.

So:

Question 1: right away I am wondering how I can use these credentials to retrieve media over an entire day or a weekend if they become invalid only a few minutes later?

Question 2: Using their "exploring API" console, I can test this query (where I am trying to get the user ID for the Ford" twitter account I use GET https://api.twitter.com/1.1/users/lookup.json?screen_name=hyundai

Typing that into the browser alone returns a 404 {"errors": [{"message": "Bad Authentication data","code": 215}]}

But using their little console APP I can pick "O Auth 1 authentication" (using a twitter app I made) and I get the JSON response I want. Examining the request object in the console shows:

GET /1.1/users/lookup.json?screen_name=hyundai HTTP/1.1
Authorization:
OAuth oauth_consumer_key="555SECRET555",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1421370510",oauth_nonce="1869828628",oauth_version="1.0",oauth_token="333DONTHACKMEPLEASE333",oauth_signature="444SECRET444"
Host:
api.twitter.com
X-Target-URI:
https://api.twitter.com
Connection:
Keep-Alive

tl;dr Version:

So, I thought this was the headers object I would need to send from Python (3) to make the identical request. So here is that code:

    import urllib.request
    header = {
        "Authorization" : "OAuth",
        "oauth_consumer_key" :"555SECRET555",
        "oauth_signature_method": "HMAC-SHA1",
        "oauth_timestamp" : "1421362844",
        "oauth_nonce":"1201915763",
        "oauth_version": "1.0",
        "oauth_token":"333CHANGINGTHIS33333",
        "oauth_signature":"222CHANGEDTHIS222",
        "Host": "api.twitter.com",
        "X-Target-URI": "https://api.twitter.com",
        "Connection": "Keep-Alive"
    }


endpoint = 'https://api.twitter.com/1.1/users/lookup.json?screen_name=hyundai'

q = urllib.request.Request(endpoint, headers=header)
a = urllib.request.urlopen(q)
print(a.read().decode('utf-8'))

But I get a bad, 404 response.

Have I formatted my headers wrong here or is there another way to do this?

score 0 · Answer 1 · edited May 23 '17 at 11:50

If you capture the network traffic from your request (use http not https), you will see that the headers you send are not the same as the header that are expected. This is why you are getting a 404 response.

What you want is something like

header = {
"Authorization": 'OAuth oauth_consumer_key="555SECRET555",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1421362844",oauth_nonce="1201915763",oauth_version="1.0",oauth_token="333CHANGINGTHIS33333",oauth_signature="222CHANGEDTHIS222"',
"Host": "api.twitter.com",
"X-Target-URI": "https://api.twitter.com",
"Connection": "Keep-Alive"
}

Of course, you could always use an OAuth library, such as RAuth or similar, see Python: OAuth Library for a discussion

Invalid response (404) with Twitter API in Python (and in general)

1 Answers1