How to cleanly intercept a request URL over java.net.HttpURLConnection in Java 8?

Question

About

Trying to see if it's possible to rewrite/mutate the request URL along with ability to intercept the headers, request and response cleanly over java.net.HttpURLConnection in Java 8. Closest approach is to use setURLStreamHandlerFactory to register a custom handler at the URL level:

URL.setURLStreamHandlerFactory(new URLStreamHandler() {
    protected URLConnection openConnection(URL url) throws IOException {
        // Mutate the URL object to reqrite
        URL rewrittenUrl = new URL(urlEndpoint);

        // Return a connection object with custom connect()
        return new URLConnection(rewrittenUrl) {
            public void connect() throws IOException {
                // Let the default HttpUrlConnection take care rest of the flow
            }
        };
    }
});

The existing usage of URLConnection can remain same:

        HttpURLConnection httpConnection = (HttpURLConnection) requestUrl.openConnection();
        httpConnection.setRequestMethod("GET");

        // Pass Auth Header
        httpConnection.setRequestProperty("Authorization", BEARER_TOKEN);

        int status = httpConnection.getResponseCode();

Problem Statement

The part that's not clear is how do we delegate the connect() method. Ideally, this should be handled by the existing http stack. Looks like HttpURLConnection is an abstract class and cannot to simply be delegated upon.

References

I would use a [http proxy](https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html) if I needed that. [Fiddler](https://www.telerik.com/fiddler) for example. — Elliott Frisch, Mar 25 '23 at 23:31
Does this answer your question? [How to write a http proxy using netty](https://stackoverflow.com/questions/46157002/how-to-write-a-http-proxy-using-netty) — life888888, Mar 26 '23 at 07:29
What I am looking for is an inproc solution. Out of proc proxy is something that works and not a solution for the original question. — PseudoAj, Mar 27 '23 at 02:25
Random thought: Have you tried a call to `HttpURLConnection.super.connect()`? (That's how the line would be written as-is in the code). — Rogue, Apr 04 '23 at 16:34

Raniz · Answer 1 · 2023-03-30T19:10:05.493

What you can do is copy the existing implementation in the JDK and modify that.

Note that this uses internal APIs and might only work on OpenJDK.

I initially wrote this on OpenJDK 16 so the copied implementation is a bit fancier than the one in OpenJDK 8, but there's no harm in that.

public class Main {

    private static String PREFIX = "sun.net.www.protocol.";

    public static void main(String[] args) throws IOException {


        URL.setURLStreamHandlerFactory((protocol) -> {
            // Avoid using reflection during bootstrap
            switch (protocol) {
                case "file":
                    return new sun.net.www.protocol.file.Handler();
                case "jar":
                    return new sun.net.www.protocol.jar.Handler();
               // Inject our own http and https handlers
                case "http":
                    return new HttpUrlHandler();
                case "https":
                    return new HttpsUrlHandler();
            }
            // The rest of this method is copied from the JDK 19 implementation
            String name = PREFIX + protocol + ".Handler";
            try {
                Object o = Class.forName(name).getDeclaredConstructor().newInstance();
                return (URLStreamHandler)o;
            } catch (Exception e) {
                // For compatibility, all Exceptions are ignored.
                // any number of exceptions can get thrown here
            }
            return null;
        });

        byte[] buffer = new byte[1024];
        new URL("http://stackoverflow.com").openStream().read(buffer);
        System.out.println(new String(buffer, StandardCharsets.UTF_8));

        new URL("https://stackoverflow.com").openStream().read(buffer);
        System.out.println(new String(buffer, StandardCharsets.UTF_8));
    }

    private static class HttpUrlHandler extends sun.net.www.protocol.http.Handler {
        @Override
        protected URLConnection openConnection(URL u) throws IOException {
            System.out.println("Overriding HTTP url");
            return super.openConnection(new URL("http://httpbin.org/user-agent"));
        }
    }

    private static class HttpsUrlHandler extends sun.net.www.protocol.https.Handler {
        @Override
        protected URLConnection openConnection(URL u) throws IOException {
            System.out.println("Overriding HTTPS url");
            return super.openConnection(new URL("https://httpbin.org/user-agent"));
        }
    }

}

What this does is extend the normal protocol handle by overriding openConnection. The new implementation prints a message to standard out and then replace the URL with http(s)://httpbin.org/user-agent which responds with your user agent. The actual connection is then handled by the normal handler that we extend.

Note that if you do this on more recent JDK you may have to handle the jrt protocol in the switch and you definitely will have to run and compile your application with a bunch of --add-exports to enable access to the internal packages.

$ ./gradlew run

> Task :compileJava
/home/raniz/src/playground/src/main/java/playground/Main.java:20: warning: sun.net.www.protocol.file.Handler is internal proprietary API and may be removed in a future release
                    return new sun.net.www.protocol.file.Handler();
                                                        ^
/home/raniz/src/playground/src/main/java/playground/Main.java:22: warning: sun.net.www.protocol.jar.Handler is internal proprietary API and may be removed in a future release
                    return new sun.net.www.protocol.jar.Handler();
                                                       ^
/home/raniz/src/playground/src/main/java/playground/Main.java:47: warning: sun.net.www.protocol.http.Handler is internal proprietary API and may be removed in a future release
    private static class HttpUrlHandler extends sun.net.www.protocol.http.Handler {
                                                                         ^
/home/raniz/src/playground/src/main/java/playground/Main.java:55: warning: sun.net.www.protocol.https.Handler is internal proprietary API and may be removed in a future release
    private static class HttpsUrlHandler extends sun.net.www.protocol.https.Handler {
                                                                           ^
4 warnings

> Task :run
Overriding HTTP url
{
  "user-agent": "Java/1.8.0_362"
}

Overriding HTTPS url
{
  "user-agent": "Java/1.8.0_362"
}


BUILD SUCCESSFUL in 1s
2 actionable tasks: 2 executed

Another way to do this would be to use a HTTP client library that allows these kinds of extensions instead - Java's URL API is far from the most ergonomical.

I found these examples with a quick search:

Modify request URI before it is sent with apache http client
How to add headers to OkHttp request interceptor? (this one's for adding headers but the approach for changing the URL is the same)

@rainz realized this doesn't allow mutating request/response streams and also if there is already a delegate doesn't quite have a way to pipeline them. But since this is the closest and very comprehensive, I will award bounty if there are no new answers. — PseudoAj, Apr 04 '23 at 05:27
In that case I recommend you take a look at OkHttp since it's rather easy to do what you want there. If you _really_ can't move away from URLConnection you can always subclass HttpUrlConnection and wrap the connection returned from the Sun stream handler in that. That gives you the possibility to overload and/or manipulate the input/output streams as well. Though using OkHttp would be _much_ easier — Raniz, Apr 04 '23 at 10:39

How to cleanly intercept a request URL over java.net.HttpURLConnection in Java 8?

About

Problem Statement

References

1 Answers1