python requests.put() fails when urllib3 http.request('PUT', ...) succeeds. What gives?

Question

I am trying to hit the Atlassian Confluence REST API using python requests.

I've successfully called a GET api, but when I call the PUT to update a confluence page, it returns 200, but didn't update the page.

I used chrome::YARC to verify that the API was working properly (which it was). After a while trying to debug it, I reverted to try using urllib3, which worked just fine.

I'd really like to use requests, but I can't for the life of me figure this one out after hours and hours of trying to debug, Google, etc.

I'm running Mac/Python3:

$ uname -a
Darwin mylaptop.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64
$ python3 --version
Python 3.6.1

Here's my code that shows all three ways I'm trying this (two requests and one urllib3):

def update(self, spaceKey, pageTitle, newContent, contentType='storage'):
    if contentType not in ('storage', 'wiki', 'plain'):
        raise ValueError("Invalid contentType={}".format(contentType))

    # Get current page info
    self._refreshPage(spaceKey, pageTitle) # I retrieve it before I update it.
    orig_version = self.version

    # Content already same as requested content. Do nothing
    if self.wiki == newContent:
        return

    data_dict = {
        'type' : 'page',
        'version' : {'number' : self.version + 1},
        'body'  : {
            contentType : {
                'representation' : contentType,
                'value' : str(newContent)
            }
        }
    }
    data_json = json.dumps(data_dict).encode('utf-8')

    put = 'urllib3' #for now until I figure out why requests.put() doesn't work
    enable_http_logging()
    if put == 'requests':
        r = self._cs.api.content(self.id).PUT(json=data_dict)
        r.raise_for_status()
    elif put == 'urllib3':
        urllib3.disable_warnings() # I know, you can quit your whining now!!!
        headers = { 'Content-Type' : 'application/json;charset=utf-8' }
        auth_header = urllib3.util.make_headers(basic_auth=":".join(self._cs.session.auth))
        headers = {**headers, **auth_header}
        http = urllib3.PoolManager()
        r = http.request('PUT', str(self._cs.api.content(self.id)), body=data_json, headers=headers)
    else:
        raise ValueError("Huh? Unknown put type: {}".format(put))
    enable_http_logging(False)

    # Verify page was updated
    self._refreshPage(spaceKey, pageTitle) # Check for changes
    if self.version != orig_version + 1:
        raise RuntimeError("Page not updated. Still at version {}".format(self.version))
    if self.wiki != newContent:
        raise RuntimeError("Page version updated, but not content.")

Any help would be great.

Update 1: Adding request dump

-----------START-----------
PUT http://confluence.myco.com/rest/api/content/101904815
User-Agent: python-requests/2.18.4
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 141
Content-Type: application/json
Authorization: Basic <auth-token-here>==

b'{"type": "page", "version": {"number": 17}, "body": {"storage": {"representation": "storage", "value": "new body here version  version 17"}}}'

Answer 1

requests never went back to PUT (Bug???)

What you're observing is requests behaving consistently with web browsers: reacting to HTTP 302 redirect with a GET request.

From Wikipedia:

The user agent (e.g. a web browser) is invited by a response with this code to make a second, otherwise identical, request to the new URL specified in the location field.

(...)

Many web browsers implemented this code in a manner that violated this standard, changing the request type of the new request to GET, regardless of the type employed in the original request (e.g. POST)

(...)

As a consequence, the update of RFC 2616 changes the definition to allow user agents to rewrite POST to GET.

So this behaviour is consistent with RFC 2616. I don't think we can say which of the two libraries behaves "more correctly".

Answer 2

Looks like a difference in how the requests and urllib3 modules deal with switching from http to https. (See @Kos answer above). Here's what I found when I checked the debug logs.

So I got to thinking after @JonClements suggested I send him the Response dump. After doing some research I found the magic runs to enable debugging for requests and urllib3 (See here).

In looking at the diffs from both, I noticed that they were being redirected from http to https for my companies confluence site:

urllib3:

DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): confluence.myco.com
DEBUG:urllib3.connectionpool:http://confluence.myco.com:80 "PUT /rest/api/content/101906196 HTTP/1.1" 302 237
DEBUG:urllib3.util.retry:Incremented Retry for (url='http://confluence.myco.com/rest/api/content/101906196'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
INFO:urllib3.poolmanager:Redirecting 
    http://confluence.myco.com/rest/api/content/101906196 -> 
    https://confluence.myco.com/rest/api/content/101906196
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): confluence.myco.com
DEBUG:urllib3.connectionpool:https://confluence.myco.com:443 "PUT /rest/api/content/101906196 HTTP/1.1" 200 None

while requests tried with my PUT and then after redirecting went to GET:

DEBUG:urllib3.connectionpool:http://confluence.myco.com:80 "PUT /rest/api/content/101906196 HTTP/1.1" 302 237
DEBUG:urllib3.connectionpool:https://confluence.myco.com:443 "GET /rest/api/content/101906196 HTTP/1.1" 200 None

requests never went back to PUT

I changed my initial url from http: to https: and everything worked fine.