How to implement curl -u in Python?

Question

I am trying to use http://developer.github.com/v3/ to retrieve project issues. This works:

curl -u "Littlemaple:mypassword" https://api.github.com/repos/MyClient/project/issues

It returns all private issues of my client's project. However, I am not able to find out how to implement this in Python. Both ways I have found (e.g. Python urllib2 Basic Auth Problem) doesn't work, they return 404 or 403 errors:

def fetch(url, username, password):
    """Wonderful method found on forums which does not work.""""
    passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
    passman.add_password(None, url, username, password)
    urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))

    req = urllib2.Request(url)
    f = urllib2.urlopen(req)
    return f.read()

...and:

def fetch(url, username, password):
    """Wonderful method found on forums which does not work neither.""""
    request = urllib2.Request(url)
    base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
    request.add_header("Authorization", "Basic %s" % base64string)   
    return urllib2.urlopen(request).read()

Any ideas? Thanks in advance!

you can peek at what curl is actually up to to see if you are missing any important interactions by passing it the `-v` option. That might give you some clues to closer emulating its behavior in python. — SingleNegationElimination, Sep 04 '11 at 21:24

score 18 · Answer 1 · edited Apr 13 '20 at 10:02

18

r = requests.get('https://api.github.com', auth=('user', 'pass'))

Python requests is the way to go here. I've been using requests extensively at work and at home for various web service interactions. It is a joy to use compared to what came before it. Note: the auth keyword arg works on any call that requires auth. Thus, you can use it sparingly, i.e. you don't need it for every call against GitHub, only those that require logins. For instance:

r = requests.get('https://api.github.com/gists/starred', auth=('user', 'pass'))

The GitHub login is documented here:

http://pypi.python.org/pypi/requests/0.6.1

edited Apr 13 '20 at 10:02

akesfeden

480
2
7
12

answered Dec 11 '11 at 18:13

David Watson

3,394
2
36
51

1

Requests are a great library. Wish I knew about it when I needed to solve this problem. It's so much better way to go in Python then using cURL if you don't really NEED to use specifically cURL. Thanks for mentioning, however, it doesn't reply exactly to given question, so I have to accept Ken's answer. – Honza Javorek Dec 12 '11 at 11:09
No problem. You are correct about Ken's answer. I just wanted to make sure that people landing here, as I did, trying to figure it out for the first time without a hard-stop curl requirement had the short answer. – David Watson Dec 12 '11 at 14:42

score 6 · Accepted Answer · answered Jun 01 '11 at 19:25

If it's 404, you probably just have the wrong URL. If it's 403, perhaps you have the realm wrong.

For starters, you're passing the URL to add_password, when in fact you should only be passing the base URL. Also, instead of install_opener, you should probably just create a new opener.

See this recipe for an example:

class NoOpHandler(urllib2.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newUrl):
        return None

passmanager = urllib2.HTTPPasswordMgrWithDefaultRealm()
passmanager.add_password(None, baseurl, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(passmanager)
opener = urllib2.build_opener(auth_handler, NoOpHandler())

Best answer, but it just didn't work. I think it's a problem somewhere in GitHub API. Because it was only a simple synchronization script, I finally used just subprocess and called curl itself. Dirty, but simplest and quickest solution. Thanks anyway! Especially for the link :) — Honza Javorek, Jun 02 '11 at 16:09

score 3 · Answer 3 · answered Mar 16 '17 at 12:19

3

You can also do it this way

 r = requests.get('https://user:pass@api.github.com')

answered Mar 16 '17 at 12:19

Aminah Nuraini

18,120
8
90
108

score 1 · Answer 4 · answered Jun 01 '11 at 17:39

1

Use pycurl which is python interface to libcurl.

answered Jun 01 '11 at 17:39

ssapkota

3,262
19
30

How to implement curl -u in Python?

4 Answers4