Github-api giving 404 when passing json-data with python + urllib2

Question

I have the following code, which should perform the first part of creating a new download at github. It should send the json-data with POST.

jsonstring = '{"name": "test", "size": "4"}'
req = urllib2.Request("https://api.github.com/repos/<user>/<repo>/downloads")
req.add_header('Authorization', 'token ' + '<token>')
result = urllib2.urlopen(req, jsonstring)

If I remove the , jsonstring from the urlopen(), it does not fail, and gives me the list of available downloads. However, if I try to POST the json-string, I get 404 error.

The problem has to be with the json, or in the way I send it, but I can't figure out what the problem is. The strings at <...> are right in the actual code, I just removed them from the post

I tried roughly the same with curl on the command-line, with slightly different method of authentication, and it worked.

Tested:

Works(returns the wanted json):

curl -u "user:password" --data "json..." https://api.github.com/repos/<user>/<repo>/downloads

Works:

curl -H 'Authorization: token <token>' https://api.github.com/

Does not work (returns "invalid credentials"):

curl -H 'Authorization: token <invalid_token>' https://api.github.com/

Does not work ("not found"):

curl -H 'Authorization: token <valid_token>' --data "json..." https://api.github.com/repos/<user>/<repo>/downloads

This does not seem to be an issue specific to the python code. The json POST data seems to be fine, and the OAuth token authorization seems to be (atleast partly) working. But when these are put together, it stops working.

Answer 1

I was finally able to work out, why it did not work.

I did not have the autorization scopes for the authorization token set correctly. The token I was using, was not "authorized" to do any modifications, and every action using it, that tried to modify something (add a download), failed.

I had to add the correct scopes to the authorization to make it work.

Answer 2

I have provided an answer as to how to POST JSON data to the V3 API without any authentication, but seen as you have identified that the original problem was with not setting up your OAUTH tokens correctly, I thought I would provide a programatic solution to getting one (this implementation gets a token every time the script is run, whereas in practice it would be done just once, and the token would be stored localy).

import urllib2
import json
import getpass
import base64

# Generate a token from the username and password.
# NOTE: this is a naive implementation. Store pre-retrieved tokens if possible.
username = 'pelson'
passwd = getpass.getpass() # <- this just puts a string in passwd (plaintext)

req = urllib2.Request("https://api.github.com/authorizations")

# add the username and password info to the request
base64string = base64.encodestring('%s:%s' % (username, passwd)).replace('\n', '')
req.add_header("Authorization", "Basic %s" % base64string)

data = json.dumps({"scopes":["repo"], "note":"Access to your repository."})
result = urllib2.urlopen(req, data)
result = json.loads('\n'.join(result.readlines()))
token = result['token']

Once you have this token, it can be used for any "repo" scope actions. Lets add a new issue to a repository:

# add an issue to the tracker using the new token
repo = 'name_of_repo'
data = json.dumps({'title': 'My automated issue.'})
req = urllib2.Request("https://api.github.com/repos/%s/%s/issues" % (username, repo))
req.add_header("Authorization", "token %s" % token)
result = urllib2.urlopen(req, data)

result = json.loads('\n'.join(result.readlines()))
print result['number']

Hope that helps somebody.

Answer 3

The v3 github api has a nice feature where it can convert markdown to html. No authentication is needed to request this information, hence it makes a nice example to try before delving into the more tricky world of authentication.

The API docs state:

Render an arbritrary Markdown document

POST /markdown Input

text Required string - The Markdown text to render

And even give an example of a markdown string to be converted:

{"text": "Hello world github/linguist#1 **cool**, and #1!"}

Given this information, lets build an urllib2 Request for the html-ified version of that markdown.

import urllib2
import json


data = {"text": "Hello world github/linguist#1 **cool**, and #1!"}
json_data = json.dumps(data)

req = urllib2.Request("https://api.github.com/markdown")
result = urllib2.urlopen(req, json_data)

print '\n'.join(result.readlines())

And the result is some html representing the markdown.

Given this example, it is not the JSON which is causing the 404 when posting your request, but more likely the authentication (as you have already answered).

Answer 4

Isn't the data meant to be URL-encoded, not JSON? Try:

data = {"name": "test", "size": "4"}
req = urllib2.Request("https://api.github.com/repos/<user>/<repo>/downloads")
req.add_header('Authorization', 'token ' + '<token>')
result = urllib2.urlopen(req, urllib.parse.urlencode(data))