curl vs python "requests" when hitting APIs

Question

I am trying to hit the Bitbucket API for my account, and a successful attempt looks like:

curl --user screename:mypassword https://api.bitbucket.org/1.0/user/repositories

in the command line. In python, I try:

import requests
url = 'https://api.bitbucket.org/1.0/user/repositories'

then

r = requests.post(url, data={'username': myscreename, 'password':mypassword})

and

r = requests.post(url, data="myscreename:mypassword")

and

r = requests.post(url, data={"user": "myscreename:mypassword"})

all get 405 error. The API is https://confluence.atlassian.com/bitbucket/rest-apis-222724129.html.

I wonder:

What am I doing wrong in the requests version, they all look similar to my curl attempt
What is the difference between requesting with curl and python requests module? What general pattern can I recognize when reading an API with a curl example and then writing it in python?

Thank you

ANSWER:

it required the right headers

https://answers.atlassian.com/questions/18451025/answers/18451117?flashId=-982194107

UPDATE:

# ===============
# get user
# ===============
import requests
import json

# [BITBUCKET-BASE-URL], i.e.: https://bitbucket.org/
url = '[BITBUCKET-BASE-URL]/api/1.0/user/'
headers = {'Content-Type': 'application/json'}

# get user
# [USERNAME], i.e.: myuser
# [PASSWORD], i.e.: itspassword
r = requests.get(url, auth=('[USERNAME]', '[PASSWORD]'), headers=headers)
print(r.status_code)
print(r.text)
#print(r.content)

`--user` in curl uses HTTP auth headers. Your `data={"username"...` solution includes them as post data. The two are not the same thing, and it's unlikely that Bitbucket looks for the ones in the post data. — Colonel Thirty Two, Jun 25 '15 at 21:49

jlhonora · Accepted Answer · 2015-06-26T14:02:04.590

Here's a way to do basic HTTP auth with Python's requests module:

requests.post('https://api.bitbucket.org/1.0/user/repositories', auth=('user', 'pass'))

With the other way you're passing the user/pass through the request's payload, which is not desired since HTTP basic auth has its own place in the HTTP protocol.

If you want to "see" what's happening under the hood with your request I recommend using httpbin:

>>> url = "http://httpbin.org/post"
>>> r = requests.post(url, data="myscreename:mypassword")
>>> print r.text
{
  "args": {}, 
  "data": "myscreename:mypassword", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "22", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.5.1 CPython/2.7.6 Darwin/14.3.0"
  }, 
  "json": null, 
  "origin": "16.7.5.3", 
  "url": "http://httpbin.org/post"
}

>>> r = requests.post(url, auth=("myscreename", "mypassword"))
>>> print r.text
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Authorization": "Basic bXlzY3JlZW5hbWU6bXlwYXNzd29yZA==", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.5.1 CPython/2.7.6 Darwin/14.3.0"
  }, 
  "json": null, 
  "origin": "16.7.5.3", 
  "url": "http://httpbin.org/post"
}

And with curl:

curl -X POST --user myscreename:mypassword http://httpbin.org/post
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Authorization": "Basic bXlzY3JlZW5hbWU6bXlwYXNzd29yZA==", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.37.1"
  }, 
  "json": null, 
  "origin": "16.7.5.3", 
  "url": "http://httpbin.org/post"
}

Notice the resemblance between the last python example and the cURL one.

Now, getting right the API's format is another story, check out this link: https://answers.atlassian.com/questions/94245/can-i-create-a-bitbucket-repository-using-rest-api

The python way should be something like this:

requests.post('https://api.bitbucket.org/1.0/repositories', auth=('user', 'pass'), data = "name=repo_name")

I like that httpbin site, but I edited the question to show my recent attempts, they did not work — codyc4321, Jun 26 '15 at 13:39
@codyc4321 That's out of the scope of this question, since it is not curl/python related — jlhonora, Jun 26 '15 at 13:55
question 1 was what is my requests attempt I tried based on their curl example doing wrong, and the point is to take a curl example and be able to get a 200 using python requests. none of these similar attempts are working in requests — codyc4321, Jun 26 '15 at 14:02
`requests.post('https://api.bitbucket.org/1.0/repositories', auth=('user', 'pass'), data = "name=repo_name")` is the attempt that got 405 — codyc4321, Jun 26 '15 at 14:06
No, that request used 'https://api.bitbucket.org/1.0/user/repositories' as an url, you need to remove "user" from it — jlhonora, Jun 26 '15 at 14:09
using that url correctly, I now get several 400s instead of 405, trying the original way then toying around. that url will make a new repo, which I did using the forum poster's exact syntax with curl, but I'm trying to grab existing repo info, which uses the other URL — codyc4321, Jun 26 '15 at 14:17
your answer cleared up the difference, and the best way to get the exactly request was asking bitbucket directly. thanks! — codyc4321, Jun 26 '15 at 15:04
@codyc4321 Great! It'd be nice if you shared the right way to do it — jlhonora, Jun 26 '15 at 15:15

score 2 · Answer 2 · answered Feb 28 '20 at 19:05

With python3, you can use json={...} instead of data={...}, and it will set the header automatically to application/json:

import requests
url = 'https://api.bitbucket.org/1.0/user/repositories'

data = {
    'data1': 'asd',
    'data2': 'asd'
}
req = requests.post(url, auth=('user', 'password'), json = data)
data = req.json()
# data['index']

curl vs python "requests" when hitting APIs

2 Answers2

Linked