Proxy-Authorization header is removed when using https

Question

I am trying to access a https url through a proxy that requires authentication in java, and the Proxy-Authorization header is not being passed to the proxy.

I know the networking aspect is working because I can perform exactly what I need using curl:

$ curl -H "Proxy-Authorization: Basic ##########" -x my_proxy_host:80 my_https_url -v

My code seems to work when I access an http url, however when I try to access a https url I get a 403 Forbidden, and I see in the logs that the Proxy-Authorization header is not passed from Java to the proxy.

Here is my code:

public static void main(String args[]) {
    try {

        HttpHost proxy = new HttpHost(my_proxy_host, 80, "http");

        DefaultHttpClient cli = new DefaultHttpClient();

        cli.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);

        HttpHost target = new HttpHost(my_https_endpoint, 443, "https");

        HttpGet get = new HttpGet("/");

        get.setHeader("Proxy-Authorization", "Basic ##############");

        HttpResponse response = cli.execute(target, get);

        HttpEntity entity = response.getEntity();

        System.out.println("----------------------------------------");
        System.out.println(response.getStatusLine());
        if (entity != null) {
            System.out.println("Response content length: " + entity.getContentLength());
        }

        EntityUtils.consume(entity);                        
    } catch (Exception e) {
        e.printStackTrace();
    }
}

Note if you change the following line the code above works:

HttpHost target = new HttpHost(my_https_endpoint, 80, "http");

Any ideas would be greatly appreciated.

Thanks!

Here are the logs that the apache httpclient generates.

Here is the version that doesn't work, trying to access https endpoint on port 443

09:43:43.574 [main] DEBUG o.a.h.i.conn.SingleClientConnManager - Get connection for route HttpRoute[{tls}->my_proxy_host:80->my_https_endpoint:443]
09:43:43.653 [main] DEBUG o.a.h.i.c.DefaultClientConnectionOperator - Connecting to my_proxy_host:80
09:43:43.716 [main] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
09:43:43.716 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Sending request: CONNECT my_https_endpoint:443 HTTP/1.1
09:43:43.717 [main] DEBUG org.apache.http.wire - >> "CONNECT my_https_endpoint:443 HTTP/1.1[\r][\n]"
09:43:43.720 [main] DEBUG org.apache.http.wire - >> "Host: my_https_endpoint:443[\r][\n]"
09:43:43.721 [main] DEBUG org.apache.http.wire - >> "Proxy-Connection: Keep-Alive[\r][\n]"
09:43:43.721 [main] DEBUG org.apache.http.wire - >> "User-Agent: Apache-HttpClient/4.1.2 (java 1.5)[\r][\n]"
09:43:43.721 [main] DEBUG org.apache.http.wire - >> "[\r][\n]"
09:43:43.721 [main] DEBUG org.apache.http.headers - >> CONNECT my_https_endpoint:443 HTTP/1.1
09:43:43.721 [main] DEBUG org.apache.http.headers - >> Host: my_https_endpoint:443
09:43:43.721 [main] DEBUG org.apache.http.headers - >> Proxy-Connection: Keep-Alive
09:43:43.721 [main] DEBUG org.apache.http.headers - >> User-Agent: Apache-HttpClient/4.1.2 (java 1.5)
09:43:43.762 [main] DEBUG org.apache.http.wire - << "HTTP/1.1 403 Forbidden[\r][\n]"
09:43:43.766 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Receiving response: HTTP/1.1 403 Forbidden
09:43:43.766 [main] DEBUG org.apache.http.headers - << HTTP/1.1 403 Forbidden
09:43:43.768 [main] DEBUG o.a.h.c.p.ResponseProcessCookies - Cookie spec not specified in HTTP context
09:43:43.772 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Connection closed
09:43:43.773 [main] DEBUG o.a.h.impl.client.DefaultHttpClient - CONNECT refused by proxy: HTTP/1.1 403 Forbidden
09:43:43.773 [main] DEBUG o.a.h.i.conn.SingleClientConnManager - Releasing connection org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter@49869a03
HTTP/1.1 403 Forbidden
Response content length: 0

And here is the version that works, accessing a http endpoint through port 80 (obviously)

09:34:11.510 [main] DEBUG o.a.h.i.conn.SingleClientConnManager - Get connection for route HttpRoute[{}->my_proxy_host:80->my_http_endpoint:80]
09:34:11.646 [main] DEBUG o.a.h.i.c.DefaultClientConnectionOperator - Connecting to my_proxy_host:80
09:34:11.720 [main] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: best-match
09:34:11.742 [main] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
09:34:11.743 [main] DEBUG o.a.h.impl.client.DefaultHttpClient - Attempt 1 to execute request
09:34:11.743 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Sending request: GET my_http_endpoint:80/ HTTP/1.1
09:34:11.744 [main] DEBUG org.apache.http.wire - >> "GET my_http_endpoint:80/ HTTP/1.1[\r][\n]"
09:34:11.746 [main] DEBUG org.apache.http.wire - >> "Proxy-Authorization: Basic ###################[\r][\n]"
09:34:11.746 [main] DEBUG org.apache.http.wire - >> "Host: my_http_endpoint:80[\r][\n]"
09:34:11.747 [main] DEBUG org.apache.http.wire - >> "Proxy-Connection: Keep-Alive[\r][\n]"
09:34:11.747 [main] DEBUG org.apache.http.wire - >> "User-Agent: Apache-HttpClient/4.1.2 (java 1.5)[\r][\n]"
09:34:11.747 [main] DEBUG org.apache.http.wire - >> "[\r][\n]"
09:34:11.747 [main] DEBUG org.apache.http.headers - >> GET my_http_endpoint:80/ HTTP/1.1
09:34:11.747 [main] DEBUG org.apache.http.headers - >> Proxy-Authorization: Basic ###################
09:34:11.747 [main] DEBUG org.apache.http.headers - >> Host: my_http_endpoint:80
09:34:11.747 [main] DEBUG org.apache.http.headers - >> Proxy-Connection: Keep-Alive
09:34:11.747 [main] DEBUG org.apache.http.headers - >> User-Agent: Apache-HttpClient/4.1.2 (java 1.5)
09:34:11.880 [main] DEBUG org.apache.http.wire - << "HTTP/1.1 200 OK[\r][\n]"
09:34:11.889 [main] DEBUG org.apache.http.wire - << "Date: Fri, 29 Mar 2013 14:34:10 GMT[\r][\n]"
09:34:11.889 [main] DEBUG org.apache.http.wire - << "Expires: -1[\r][\n]"
09:34:11.889 [main] DEBUG org.apache.http.wire - << "Cache-Control: private, max-age=0[\r][\n]"
09:34:11.889 [main] DEBUG org.apache.http.wire - << "Content-Type: text/html; charset=ISO-8859-1[\r][\n]"
09:34:11.889 [main] DEBUG org.apache.http.wire - << "Set-Cookie: PREF=ID=918523062b63f55d:FF=0:TM=1364567650:LM=1364567650:S=T8IIsiH2_cw1UMLI; expires=Sun, 29-Mar-2015 14:34:10 GMT; path=/; domain=.google.com[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "Set-Cookie: NID=67=OHgx0xi1JDegF7uPjPUfW_pCC5Yn0H9S8-CMxyKbtgrF2vPQ-svFv1G4h5yxwCVEi2gDA23tywtEiYQvbPCHxDpAkefUlDJrWK94fEmXaiwuvua5w50eAej0yy3ysI4N; expires=Sat, 28-Sep-2013 14:34:10 GMT; path=/; domain=.google.com; HttpOnly[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "P3p: CP="This is not a P3P policy! See my_http_endpoint/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "Server: gws[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "X-Xss-protection: 1; mode=block[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "X-Frame-options: SAMEORIGIN[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "Transfer-Encoding: chunked[\r][\n]"
09:34:11.890 [main] DEBUG org.apache.http.wire - << "Connection: close[\r][\n]"
09:34:11.891 [main] DEBUG org.apache.http.wire - << "[\r][\n]"
09:34:11.892 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
09:34:11.892 [main] DEBUG org.apache.http.headers - << HTTP/1.1 200 OK
09:34:11.892 [main] DEBUG org.apache.http.headers - << Date: Fri, 29 Mar 2013 14:34:10 GMT
09:34:11.892 [main] DEBUG org.apache.http.headers - << Expires: -1
09:34:11.892 [main] DEBUG org.apache.http.headers - << Cache-Control: private, max-age=0
09:34:11.892 [main] DEBUG org.apache.http.headers - << Content-Type: text/html; charset=ISO-8859-1
09:34:11.892 [main] DEBUG org.apache.http.headers - << Set-Cookie: PREF=ID=918523062b63f55d:FF=0:TM=1364567650:LM=1364567650:S=T8IIsiH2_cw1UMLI; expires=Sun, 29-Mar-2015 14:34:10 GMT; path=/; domain=.google.com
09:34:11.892 [main] DEBUG org.apache.http.headers - << Set-Cookie: NID=67=OHgx0xi1JDegF7uPjPUfW_pCC5Yn0H9S8-CMxyKbtgrF2vPQ-svFv1G4h5yxwCVEi2gDA23tywtEiYQvbPCHxDpAkefUlDJrWK94fEmXaiwuvua5w50eAej0yy3ysI4N; expires=Sat, 28-Sep-2013 14:34:10 GMT; path=/; domain=.google.com; HttpOnly
09:34:11.892 [main] DEBUG org.apache.http.headers - << P3p: CP="This is not a P3P policy! See my_http_endpoint/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
09:34:11.892 [main] DEBUG org.apache.http.headers - << Server: gws
09:34:11.892 [main] DEBUG org.apache.http.headers - << X-Xss-protection: 1; mode=block
09:34:11.892 [main] DEBUG org.apache.http.headers - << X-Frame-options: SAMEORIGIN
09:34:11.892 [main] DEBUG org.apache.http.headers - << Transfer-Encoding: chunked
09:34:11.892 [main] DEBUG org.apache.http.headers - << Connection: close
09:34:11.916 [main] DEBUG o.a.h.c.p.ResponseProcessCookies - Cookie accepted: "[version: 0][name: PREF][value: ID=918523062b63f55d:FF=0:TM=1364567650:LM=1364567650:S=T8IIsiH2_cw1UMLI][domain: .google.com][path: /][expiry: Sun Mar 29 09:34:10 CDT 2015]"
09:34:11.917 [main] DEBUG o.a.h.c.p.ResponseProcessCookies - Cookie accepted: "[version: 0][name: NID][value: 67=OHgx0xi1JDegF7uPjPUfW_pCC5Yn0H9S8-CMxyKbtgrF2vPQ-svFv1G4h5yxwCVEi2gDA23tywtEiYQvbPCHxDpAkefUlDJrWK94fEmXaiwuvua5w50eAej0yy3ysI4N][domain: .google.com][path: /][expiry: Sat Sep 28 09:34:10 CDT 2013]". 
HTTP/1.1 200 OK
Response content length: -1
09:34:11.925 [main] DEBUG o.a.h.i.conn.SingleClientConnManager - Releasing connection org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter@303459ed
09:34:11.925 [main] DEBUG o.a.h.i.conn.SingleClientConnManager - Released connection open but not reusable.
09:34:11.926 [main] DEBUG o.a.h.i.conn.DefaultClientConnection - Connection shut down

Why are you not using the credentials provider to set this up and let the framework do it? — Dave G, Mar 26 '13 at 17:30
I couldn't get the framework to authenticate it correctly, it adds the Authorization header successfully, but the proxy that I am using is looking for the Proxy-Authorization header. Do you have an example on how to do that? — user2209017, Mar 26 '13 at 17:47
Thank you for adding the logs - Ok looks like your proxy is EXPECTING an auth header to be sent even without a challenge. HttpClient 4.x doesn't by default do pre-emptive authentication - but we can tweak it to do that - let me code something up. — Dave G, Mar 29 '13 at 18:41
Did anyone find a solution using the Heroku Proximo addon? I'm running into this same issue. — RC., May 11 '13 at 13:36

score 3 · Answer 1 · edited May 23 '17 at 12:06

3

Try this variant and see if it might meet your needs:

public static void main(String args[]) {
    try {

        HttpHost proxy = new HttpHost(my_proxy_host, 80, "http");

        DefaultHttpClient cli = new DefaultHttpClient();

        cli.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
        cli.getCredentialsProvider().setCredentials(
            new AuthScope(proxy.getHostName(), proxy.getPort()),
            new UsernamePasswordCredentials("username", "password"));

        HttpHost target = new HttpHost(my_https_endpoint, 443, "https");

        HttpGet get = new HttpGet("/");

        // Example from: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html#d5e1032
        BasicAuthCache authCache= new BasicAuthCache();
        // Target the proxy not target host
        authCache.put(proxy, new BasicScheme(ChallengeState.PROXY));
        BasicHttpContext context= new BasicHttpContext();
        context.setAttribute(ClientContext.AUTH_CACHE, authCache);
        HttpResponse response = cli.execute(target, get, context);

        HttpEntity entity = response.getEntity();

        System.out.println("----------------------------------------");
        System.out.println(response.getStatusLine());
        if (entity != null) {
            System.out.println("Response content length: " + entity.getContentLength());
        }

        EntityUtils.consume(entity);                        
    } catch (Exception e) {
        e.printStackTrace();
    }
}

Ideally, the proxy should have responded with an HTTP 407 Proxy Authentication Required on the initial request instead of a 403 Forbidden. The 403 basically is saying GO-AWAY! While the 407 says, "hey - you wanna come through? give me a reason why?" by responding with a "Proxy-Authenticate: " header, to which you must respond with your credentials via a "Proxy-Authorization: " header.

By preemptively authenticating, you're essentially sending credentials to a host when they have not been asked, kind of like stapling your Social Security number in large letters to your forehead.

EDIT I think I may have found something that MIGHT get you over the hump on this one: HttpClient 4.2.2 and proxy with username/password see if that kicks it into gear.

edited May 23 '17 at 12:06

Community

1
1

answered Mar 26 '13 at 23:25

Dave G

9,639
36
41

Dave, thank you for you help! I tried this code but the Proxy-Authorization parameter did not get passed in, so I am still receiving the 403 ... – user2209017 Mar 27 '13 at 15:42
I'm wondering if there is something wonky with the fact that you're proxying HTTPS over HTTP – Dave G Mar 27 '13 at 16:14
Ok I'll dig a little deeper and see if I can spot something – Dave G Mar 27 '13 at 17:23
Can you provide a wire debug log from the apache httpclient? be careful to mask or replace any password hashes that may be in transit. – Dave G Mar 28 '13 at 00:46
The log levels to put on debug org.apache.http, org.apache.http.wire – Dave G Mar 28 '13 at 00:47
Dave, please see above, I have added the logs, is that what you were referring to? – user2209017 Mar 29 '13 at 14:46
With regard to the proxy - do you have control of it or is it a company entity? – Dave G Mar 29 '13 at 18:31
I've placed some edits to my answer as well as reformatted your question slightly – Dave G Mar 30 '13 at 15:36
The proxy is a service, I do not have any control over it. – user2209017 Apr 01 '13 at 14:47
Dave, I executed your new code, and it is now passing an Authorization header which is progress, unfortunately the proxy is looking for a Proxy-Authentication header. Potentially that could be changed on the service side however my target endpoint also requires authentication and reads the signature from the Authorization header, and needs to be different than the Proxy authentication ... BTW, thank you very much for you help so far, I haven't voted your answer up because I lack the rep. sorry! – user2209017 Apr 01 '13 at 15:01
The only other thing I can think to try is adding an AuthState on the proxy to CHALLENGED in the context - my suggestion is get that proxy to return a 407 to correctly challenge the client that way it will automatically handle it. I unfortunately don't have an environment to test that theory with. – Dave G Apr 01 '13 at 16:22
I did a little more digging on this - unfortunately it doesn't work like a 401 auth challenge does - I think you'll need to try preemptively configuring the AuthState and adding that to the local context that was created. Without having the 407 response from the proxy, I don't know any other way to configure http client to preemptively authenticate a proxy without the challenge. – Dave G Apr 01 '13 at 17:29
Just verified it - the ChallengeState.PROXY works as advertised. – Dave G Apr 02 '13 at 20:34
Thank you so much! BTW; you need to be using at least version 4.2 of httpclient to have the ChallengeState.PROXY functionality – user339047 Aug 06 '13 at 22:17

score 1 · Accepted Answer · answered Apr 03 '13 at 15:02

I solved my problem by creating my own proxy on amazon EC2.

My goal was to be able to access an https service that required IP whitelisting, from an application running on Heroku, so there is no static IP or IP range that I can whitelist, therefore I needed a proxy with a static IP.

I was trying to use the proximo heroku addon and was having the problem I described above.

Thanks to Dave G for the help!

Proxy-Authorization header is removed when using https

2 Answers2

Linked