Jena Proxy Authentication

Question

I get a HTTPException 407 when trying to query a SPARQL Service. I have to use a proxy which requires username and password.

My code for the authentification looks as follows:

HttpAuthenticator authenticator = new PreemptiveBasicAuthenticator(new SimpleAuthenticator(USERNAME, PASSWORD),true);

Then I try to query as follows:

QueryExecution qexec = QueryExecutionFactory.sparqlService(serverUrl, query,
                                                           authenticator);
qexec.execSelect();

In the jvm arguments I pass -Dhttp.proxyHost and -Dhttp.proxyPort.

The confusing thing is that if I try to connect to google, it works with my proxy setting. Even without username/password.

String strUrl = "http://www.google.com"; 
URL url = new URL(strUrl);
HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
urlConn.connect();

If I try to connect to the sparql endpoint I get a HTTP 407 error. I can reach the endpoint in the browser and also my colleague has access to the endpoint.

In the documentation they say that the boolean in the PreemptiveBasicAuthenticator Constructor is for proxy useage. That's why I tried it with this code.

Edit: Logging Information

DEBUG 27139 [2014-03-18 14:05:37,970] [main] org.apache.http.impl.conn.PoolingClientConnectionManager  - Connection request: [route: {}->http:*:81->http:*.com][total kept alive: 0; route allocated: 0 of 5; total allocated: 0 of 10]
DEBUG 27146 [2014-03-18 14:05:37,977] [main] org.apache.http.impl.conn.PoolingClientConnectionManager  - Connection leased: [id: 0][route: {}->http:*:81->http:*.com][total kept alive: 0; route allocated: 1 of 5; total allocated: 1 of 10]
DEBUG 27148 [2014-03-18 14:05:37,979] [main] org.apache.http.impl.conn.DefaultClientConnectionOperator  - Connecting to proxy.*:81
DEBUG 27162 [2014-03-18 14:05:37,993] [main] org.apache.http.client.protocol.RequestAddCookies  - CookieSpec selected: best-match
DEBUG 27164 [2014-03-18 14:05:37,995] [main] org.apache.http.client.protocol.RequestAuthCache  - Auth cache not set in the context
DEBUG 27164 [2014-03-18 14:05:37,995] [main] org.apache.http.client.protocol.RequestTargetAuthentication  - Target auth state: UNCHALLENGED
DEBUG 27165 [2014-03-18 14:05:37,996] [main] org.apache.http.client.protocol.RequestProxyAuthentication  - Proxy auth state: UNCHALLENGED
DEBUG 27166 [2014-03-18 14:05:37,997] [main] org.apache.http.impl.client.SystemDefaultHttpClient  - Attempt 1 to execute request
DEBUG 27166 [2014-03-18 14:05:37,997] [main] org.apache.http.impl.conn.DefaultClientConnection  - Sending request: GET http:* HTTP/1.1
DEBUG 27168 [2014-03-18 14:05:37,999] [main] org.apache.http.wire  -  >> "GET http:* HTTP/1.1[\r][\n]"
DEBUG 27169 [2014-03-18 14:05:38,000] [main] org.apache.http.wire  -  >> "Accept: application/sparql-results+xml[\r][\n]"
DEBUG 27169 [2014-03-18 14:05:38,000] [main] org.apache.http.wire  -  >> "User-Agent: Apache-Jena-ARQ/2.11.1[\r][\n]"
DEBUG 27170 [2014-03-18 14:05:38,001] [main] org.apache.http.wire  -  >> "Accept-Encoding: gzip,deflate[\r][\n]"
DEBUG 27170 [2014-03-18 14:05:38,001] [main] org.apache.http.wire  -  >> "Host: *[\r][\n]"
DEBUG 27171 [2014-03-18 14:05:38,002] [main] org.apache.http.wire  -  >> "Proxy-Connection: Keep-Alive[\r][\n]"
DEBUG 27171 [2014-03-18 14:05:38,002] [main] org.apache.http.wire  -  >> "[\r][\n]"
DEBUG 27171 [2014-03-18 14:05:38,002] [main] org.apache.http.headers  - >> GET http:* HTTP/1.1
DEBUG 27172 [2014-03-18 14:05:38,003] [main] org.apache.http.headers  - >> Accept: application/sparql-results+xml
DEBUG 27173 [2014-03-18 14:05:38,004] [main] org.apache.http.headers  - >> User-Agent: Apache-Jena-ARQ/2.11.1
DEBUG 27173 [2014-03-18 14:05:38,004] [main] org.apache.http.headers  - >> Accept-Encoding: gzip,deflate
DEBUG 27173 [2014-03-18 14:05:38,004] [main] org.apache.http.headers  - >> Host: *
DEBUG 27173 [2014-03-18 14:05:38,004] [main] org.apache.http.headers  - >> Proxy-Connection: Keep-Alive
DEBUG 27178 [2014-03-18 14:05:38,009] [main] org.apache.http.wire  -  << "HTTP/1.1 407 Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied.  )[\r][\n]"
DEBUG 27179 [2014-03-18 14:05:38,010] [main] org.apache.http.wire  -  << "Via: 1.1 *[\r][\n]"
DEBUG 27179 [2014-03-18 14:05:38,010] [main] org.apache.http.wire  -  << "Proxy-Authenticate: Negotiate[\r][\n]"
DEBUG 27179 [2014-03-18 14:05:38,010] [main] org.apache.http.wire  -  << "Proxy-Authenticate: Kerberos[\r][\n]"
DEBUG 27180 [2014-03-18 14:05:38,011] [main] org.apache.http.wire  -  << "Proxy-Authenticate: NTLM[\r][\n]"
DEBUG 27180 [2014-03-18 14:05:38,011] [main] org.apache.http.wire  -  << "Connection: Keep-Alive[\r][\n]"
DEBUG 27180 [2014-03-18 14:05:38,011] [main] org.apache.http.wire  -  << "Proxy-Connection: Keep-Alive[\r][\n]"
DEBUG 27181 [2014-03-18 14:05:38,012] [main] org.apache.http.wire  -  << "Pragma: no-cache[\r][\n]"
DEBUG 27181 [2014-03-18 14:05:38,012] [main] org.apache.http.wire  -  << "Cache-Control: no-cache[\r][\n]"
DEBUG 27181 [2014-03-18 14:05:38,012] [main] org.apache.http.wire  -  << "Content-Type: text/html[\r][\n]"
DEBUG 27182 [2014-03-18 14:05:38,013] [main] org.apache.http.wire  -  << "Content-Length: 894   [\r][\n]"
DEBUG 27182 [2014-03-18 14:05:38,013] [main] org.apache.http.wire  -  << "[\r][\n]"
DEBUG 27183 [2014-03-18 14:05:38,014] [main] org.apache.http.impl.conn.DefaultClientConnection  - Receiving response: HTTP/1.1 407 Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied.  )
DEBUG 27183 [2014-03-18 14:05:38,014] [main] org.apache.http.headers  - << HTTP/1.1 407 Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied.  )
DEBUG 27183 [2014-03-18 14:05:38,014] [main] org.apache.http.headers  - << Via: 1.1 *
DEBUG 27184 [2014-03-18 14:05:38,015] [main] org.apache.http.headers  - << Proxy-Authenticate: Negotiate
DEBUG 27184 [2014-03-18 14:05:38,015] [main] org.apache.http.headers  - << Proxy-Authenticate: Kerberos
DEBUG 27184 [2014-03-18 14:05:38,015] [main] org.apache.http.headers  - << Proxy-Authenticate: NTLM
DEBUG 27185 [2014-03-18 14:05:38,016] [main] org.apache.http.headers  - << Connection: Keep-Alive
DEBUG 27185 [2014-03-18 14:05:38,016] [main] org.apache.http.headers  - << Proxy-Connection: Keep-Alive
DEBUG 27185 [2014-03-18 14:05:38,016] [main] org.apache.http.headers  - << Pragma: no-cache
DEBUG 27185 [2014-03-18 14:05:38,016] [main] org.apache.http.headers  - << Cache-Control: no-cache
DEBUG 27186 [2014-03-18 14:05:38,017] [main] org.apache.http.headers  - << Content-Type: text/html
DEBUG 27186 [2014-03-18 14:05:38,017] [main] org.apache.http.headers  - << Content-Length: 894   
DEBUG 27187 [2014-03-18 14:05:38,018] [main] org.apache.http.impl.client.SystemDefaultHttpClient  - Connection can be kept alive indefinitely
DEBUG 27187 [2014-03-18 14:05:38,018] [main] org.apache.http.impl.client.SystemDefaultHttpClient  - Authentication required
DEBUG 27187 [2014-03-18 14:05:38,018] [main] org.apache.http.impl.client.SystemDefaultHttpClient  - proxy*:81 requested authentication
DEBUG 27188 [2014-03-18 14:05:38,019] [main] org.apache.http.impl.client.ProxyAuthenticationStrategy  - Authentication schemes in the order of preference: [negotiate, Kerberos, NTLM, Digest, Basic]
DEBUG 27189 [2014-03-18 14:05:38,020] [main] org.apache.http.impl.auth.SPNegoScheme  - Received challenge '' from the auth server
DEBUG 27190 [2014-03-18 14:05:38,021] [main] org.apache.http.impl.auth.KerberosScheme  - Received challenge '' from the auth server
DEBUG 27190 [2014-03-18 14:05:38,021] [main] org.apache.http.impl.client.ProxyAuthenticationStrategy  - Challenge for Digest authentication scheme not available
DEBUG 27190 [2014-03-18 14:05:38,021] [main] org.apache.http.impl.client.ProxyAuthenticationStrategy  - Challenge for Basic authentication scheme not available
DEBUG 27198 [2014-03-18 14:05:38,029] [main] org.apache.http.wire  -  << "<HTML><HEAD>[\r][\n]"
DEBUG 27198 [2014-03-18 14:05:38,029] [main] org.apache.http.wire  -  << "<TITLE>Proxy Error Message</TITLE>[\r][\n]"
DEBUG 27198 [2014-03-18 14:05:38,029] [main] org.apache.http.wire  -  << "</HEAD>[\r][\n]"
DEBUG 27199 [2014-03-18 14:05:38,030] [main] org.apache.http.wire  -  << "<frameset cols="*">[\r][\n]"
DEBUG 27199 [2014-03-18 14:05:38,030] [main] org.apache.http.wire  -  << "[0x9]<frame src="http:* Proxy Authentication Required&err=Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied. &proxy=*&time=18.03.2014 13:05:38 [GMT]&isa=1&errorhtml=default.htm" name="ProxyError"> [\r][\n]"
DEBUG 27200 [2014-03-18 14:05:38,031] [main] org.apache.http.wire  -  << " <noframes>[\r][\n]"
DEBUG 27200 [2014-03-18 14:05:38,031] [main] org.apache.http.wire  -  << " If this site does not automatically reload, click <a href="*">here</a> [\r][\n]"
DEBUG 27201 [2014-03-18 14:05:38,032] [main] org.apache.http.wire  -  << " </noframes>[\r][\n]"
DEBUG 27201 [2014-03-18 14:05:38,032] [main] org.apache.http.wire  -  << "</frameset>[\r][\n]"
DEBUG 27202 [2014-03-18 14:05:38,033] [main] org.apache.http.wire  -  << "</HTML>[\r][\n]"
DEBUG 27202 [2014-03-18 14:05:38,033] [main] org.apache.http.impl.conn.PoolingClientConnectionManager  - Connection [id: 0][route: {}->http:*:81->http:* can be kept alive indefinitely
DEBUG 27203 [2014-03-18 14:05:38,034] [main] org.apache.http.impl.conn.PoolingClientConnectionManager  - Connection released: [id: 0][route: {}->http:*:81->http:*][total kept alive: 1; route allocated: 1 of 5; total allocated: 1 of 10]

score 1 · Answer 1 · edited May 23 '17 at 11:49

1

HttpAuthenticator applies to the remote endpoint you are querying, not the proxy.

You need to set http.proxyUser and http.proxyPassword as JVM arguments or set the proxy authenticator.

See examples from http 407 proxy authentication required : how to handle in java code which leads to http://blog.vinodsingh.com/2008/05/proxy-authentication-in-java.html

edited May 23 '17 at 11:49

Community

1
1

answered Mar 17 '14 at 22:41

AndyS

16,345
17
21

score 0 · Answer 2 · answered Mar 18 '14 at 09:10

The confusing thing is that if I try to connect to google, it works with my proxy setting. Even without username/password.

String strUrl = "http://www.google.com"; 
URL url = new URL(strUrl);
HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
urlConn.connect();

If I try to connect to the sparql endpoint I get a HTTP 407 error. I can reach the endpoint in the browser and also my colleague has access to the endpoint.

In the documentation they say that the boolean in the PreemptiveBasicAuthenticator Constructor is for proxy useage. That's why I tried it with this code.

If you have additional information to add to your question please use the edit button under your question, I have done this for you this time — RobV, Mar 18 '14 at 10:26

RobV · Answer 3 · 2014-03-18T15:43:15.610

When Preemptive Authentication works for proxies

Using PreemptiveBasicAuthenticator with the forProxy argument set to true is only relevant in certain circumstances:

Your proxy server requires authentication (it appears this is the case)
Your proxy server uses Basic HTTP authentication

From the fact that you get a HTTP 407 this would imply that your proxy server does not support Basic HTTP authentication and is likely asking you to do digest or some other kind of HTTP authentication.

Debugging what's going wrong

Note that ARQ uses Apache Http Client behind the scenes so you can turn up logging for that in order to see detailed HTTP traces. Set the log level for the org.apache.http package to DEBUG to see general information about the requests and responses and turn it all the way up to TRACE to see detailed HTTP traces which will allow you to see precisely the response the proxy returns.

In particular you want to look at what the Proxy-Authenticate header contains in the servers response, if the contents of this header does not contain the word Basic then you cannot use the PreemptiveBasicAuthenticator.

Using non-basic authentication

If the server does not support Basic authentication then you may be able to use the SimpleAuthenticator directly instead, this will support the Basic/Digest authentication schemes so may resolve the authentication problem e.g.

HttpAuthenticator authenticator = new SimpleAuthenticator(USERNAME, PASSWORD);

Edit

However in your case the problem looks to be that your proxy expects to use NTLM authentication and it turns out our authentication API only actually works out of the box with the Basic and Digest authentication schemes (it does support Form based authentication but that is not relevant in this discussion).

In order to support NTLM authentication we need to create the credentials we pass slightly differently because we additionally need to know the workstation and the domain. I've filed JENA-658 to record this problem and am working on a fix. I'll edit the answer again once I have a proposed fix that you can test.

Edit 2

So I have committed my fix but I don't have an NTLM protected server I can test against so I'll need you to test the fix out and let me know.

To get a version of ARQ with the fix change your Maven dependency version for it to 2.11.2-SNAPSHOT which will cause maven to grab the very latest code. You can then replace use of SimpleAuthenticator with SimpleNTAuthenticator - this requires knowing a couple of bits of extra information e.g.

HttpAuthenticator authenticator = new SimpleNTAuthenticator(USERNAME, PASSWORD, 
                                                            WORKSTATION, DOMAIN);

The first is your workstation ID (i.e. computer name) and the second is the NT domain you are authenticating to. You can find both pieces of info by looking at the Computer item in Windows Explorer, on Windows 7 the status bar displays this information. In older versions of Windows you may need to Right-Click on Computer and select Properties.

If you could try this out and let me know if it resolves the problem that'd be much appreciated.

thanks for the tip! I added some lines from the logger to my question above. Using the SimpleAuthenticator directly also did not work for me. — user3429986, Mar 18 '14 at 12:35
Can you please turn on `TRACE` level logging which will show even more detail, we really need to see the actual HTTP request and response headers to help you further — RobV, Mar 18 '14 at 12:54
I set: LogManager.getLogger("org.apache.http").setLevel(Level.TRACE);I hope this is what you need! Thanks for your help! — user3429986, Mar 18 '14 at 13:42
@user3429986 Yes that helped me see what the root problem is, see my updated answer for details — RobV, Mar 18 '14 at 14:47
@user3429986 Ok I have a fix ready for you to test, see my latest edit for details on trying out the proposed fix — RobV, Mar 18 '14 at 15:43
I changed my maven dependency for jena-arq To 2.11.2-SNAPSHOT. Unfortunately I get the following error: Failure to find org.apache.jena:jena-arq:jar:2.11.2 in http://repo.maven.apache.org/maven2 — user3429986, Mar 18 '14 at 15:58
@user3429986 You'll need to add the Apache snapshots repository to your POM - see http://jena.apache.org/download/maven.html - specifically the bit about specifying dependencies on snapshots — RobV, Mar 18 '14 at 16:18
I updated arq and changed the authenticator as you suggested, but I get exectly the same error and trace, as before. — user3429986, Mar 18 '14 at 16:35
@user3429986 If you are still having problems then there isn't much more we can do to help you. You may need to talk to whoever administers your proxy to understand exactly how it expects you to authenticate since an incorrect workstation/domain entry would cause NT authentication to fail. It is also possible that the proxy is not sending an appropriate NTLM challenge so HTTP Client never even tries to use your credentials — RobV, Mar 19 '14 at 09:23
It may also be the case that actually you need to be using Kerberos authentication which is going to require configuration on your part and not something we can help you with — RobV, Mar 19 '14 at 09:25

Jena Proxy Authentication

3 Answers3

When Preemptive Authentication works for proxies

Debugging what's going wrong

Using non-basic authentication