22

I have an Apache running that is only accessible via HTTPS. I want to serve websockets from an additional server application which runs on the same machine, but since it is not possible for clients to connect on another port than 443 to our server, those websocket connections need to be proxied through the Apache.

Now, I've installed mod_proxy and configured it as follows:

SSLProxyEngine on
ProxyPass /ws https://127.0.0.1:9001

This does not work however. I can connect to https://server/ws in my browser now, but the apache seems to swallow part of the websockets headers, so that real websocket connections do not work.

How can I accomplish tunneling my websocket connections through the Apache server?

Kijewski
  • 25,517
  • 12
  • 101
  • 143
mstud
  • 413
  • 1
  • 4
  • 13
  • Any progress with this problem? – Olle Härstedt Jun 27 '13 at 10:59
  • There seem to be new Apache modules which might make this easier. I have, however, resorted to using stunnel+HAproxy, stunnel to accept the SSL connection and pass through the unencrypted traffic to HAproxy, which then decides on the presence of the "Upgrade: WebSocket" header whether it should redirect it to the websocket server or to Apache via plain HTTP. With version 1.5 of HAproxy (which is currently in development) the use of stunnel might not even be necessary anymore. – mstud Jul 02 '13 at 09:05

2 Answers2

29

I've got it working.

Scenario

-------------       ----------------       ----------
| Browser   |<----->| Apache httpd |<----->| Tomcat |
|           |  SSL  |    2.4.9     |  SSL  | 7.0.52 |
-------------       ----------------       ----------

Browser WebSocket through Apache httpd, reverse proxying to the web app in Tomcat. All SSL front-to-back.

Here's the configuration for each piece:

Browser Client

Note the trailing "/" in the url: wss://host/app/ws/. It was necessary to match the correct wss ProxyPass directive (shown further down in the Apache config section) and preventing a 301 redirect to https://host/app/ws. That is, it was redirecting using the https scheme and not the wss scheme for the back-end.

Test Page
<!doctype html>
<body>

<script type="text/javascript">
    var connection = new WebSocket("wss://host/app/ws/");

    connection.onopen = function () {
        console.log("connected");
    };

    connection.onclose = function () {
        console.log("onclose");
    };

    connection.onerror = function (error) {
        console.log(error);
    };
</script>

</body>
</html>

Apache httpd

I am using Apache httpd 2.4.9, which out of the box provides mod_proxy_wstunnel. However, the mod_proxy_wstunnel.so provided does not support SSL when using wss:// scheme. It ends up trying to connect to the back-end (Tomcat) in plaintext, which fails the SSL handshake. See bug here. So, you have to patch mod_proxy_wstunnel.c yourself by following the suggested correction in the bug report. It's an easy 3 line change.

Suggested correction,
314a315
>     int is_ssl = 0;
320a322
>         is_ssl = 1;
344c346
<     backend->is_ssl = 0;
---
>     backend->is_ssl = is_ssl;

Then rebuild the module and replace in your new mod_proxy_wstunnel.so with the old one.

Building Apache httpd

Here's the (2.4.9) command I used to build in the modules I wanted. You might not need them all.

./configure --prefix=/usr/local/apache --with-included-apr --enable-alias=shared
--enable-authz_host=shared --enable-authz_user=shared 
--enable-deflate=shared --enable-negotiation=shared 
--enable-proxy=shared --enable-ssl=shared --enable-reqtimeout=shared
--enable-status=shared --enable-auth_basic=shared
--enable-dir=shared --enable-authn_file=shared
--enable-autoindex=shared --enable-env=shared --enable-php5=shared
--enable-authz_default=shared --enable-cgi=shared
--enable-setenvif=shared --enable-authz_groupfile=shared
--enable-mime=shared --enable-proxy_http=shared
--enable-proxy_wstunnel=shared

Note the very last switch: --enable-proxy_wstunnel=shared At first, I was incorrectly using --enable-proxy-wstunnel=shared, which seemed to build fine, but ultimately wouldn't work when I used the resultant .so file. See the difference? You want to make sure to use an underscore in "proxy_wstunnel" rather than a dash.

Apache httpd config

httpd.conf
...
LoadModule proxy_module modules/mod_proxy.so
...
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
...
LoadModule ssl_module modules/mod_ssl.so
...
Include conf/extra/httpd-ssl.conf
...
LogLevel debug
ProxyRequests off

# Note, this is the preferred ProxyPass configuration, and *should* be equivalent
# to the same inline version below, but it does NOT WORK!
#<Location /app/ws/>
#        ProxyPass wss://localhost:8443/app/ws
#        ProxyPassReverse wss://localhost:8443/app/ws
#</Location>
#<Location /app/>
#        ProxyPass https://localhost:8443/app/
#        ProxyPassReverse https://localhost:8443/app/
#</Location>

# NOTE: Pay strict attention to the slashes "/" or lack thereof!
# WebSocket url endpoint
ProxyPass /app/ws/ wss://localhost:8443/app/ws
ProxyPassReverse /app/ws/ wss://localhost:8443/app/ws

# Everything else
ProxyPass /app/ https://localhost:8443/app/
ProxyPassReverse /app/ https://localhost:8443/app/

If you didn't see my note in the above config, here it is again: Pay strict attention to the slashes "/" or lack thereof!

Also, if you are seeing debug log statements in your apache log that says a wss connection was made then closed, it is possible that you have mod_reqtimeout enabled as I did, so make sure it not loaded:

#LoadModule reqtimeout_module modules/mod_reqtimeout.so

Tomcat

Assuming your HTTP connector is setup correct, there's not much to configure in tomcat. Though to aid in debugging, I found it useful to create a $CATALINA_HOME/bin/setenv.sh that looked like this:

setenv.sh
CATALINA_OPTS=$CATALINA_OPTS" -Djavax.net.debug=all -Djavax.net.debug=ssl:handshake:verbose"

This allowed me to see if the mod_proxy_wstunnel.so that I modified was working or not for wss://. When it wasn't working, my catalina.out log file would show:

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
http-nio-8443-exec-1, SEND TLSv1 ALERT:  fatal, description = internal_error
http-nio-8443-exec-1, WRITE: TLSv1 Alert, length = 2
http-nio-8443-exec-1, called closeOutbound()
http-nio-8443-exec-1, closeOutboundInternal()

Final Thoughts

Though I am using Apache httpd 2.4.9, I've seen where backports of mod_proxy_wstunnel can be applied to versions 2.2.x. Hopefully my notes above can be applied to those older versions.

Tom Cawley
  • 301
  • 3
  • 5
  • 3
    It's worth noting that mod_proxy_wstunnel also works to wrap non-SSL websockets behind an SSL proxy. – Phil Lello Jan 15 '15 at 11:04
  • @PhilLello, then I would assume if you connect to WS via a non-ssl websocket you wouldn't need the modified mod_proxy_wstunnel, but can use the one "out of the box." Any compelling reason to make Apache connect vis wss instead of ws if it's running on localhost? Seems like a good solution. – davidethell Mar 09 '15 at 10:42
  • @Tom, do you think, its a good solution security wise? I'm not a WebSocket expert but I think, redirecting secure connection to non secure is not good thing (Please correct me if I'm wrong) Thanks! – Muaaz Khalid Jan 04 '16 at 18:55
  • I'm actually trying to wrap a non-ssl websocket behind an SSL proxy, but can't seem to get it to work.. any pointers? – Ketzak May 17 '16 at 15:35
  • NOTE: The apache bug mentioned has been fixed since 2.4.10. – boweeb Oct 18 '17 at 14:27
1

If you don't want Apache to terminate the SSL connection (and forward unencrypted WebSocket traffic), but have the SSL terminated on the final target WebSocket server and exlusively want to use WSS on the WebSocket traffic coming into Apache, then mod_proxy_connect may be able to just connect through the raw traffic. Not sure. I'd be also interested if that works.

If above does not hold, here is more information:

In any case, using Apache will severly limit the scalability regarding number of concurrently served WebSocket connections, since every WS connection will consume 1 process/thread on Apache.

Community
  • 1
  • 1
oberstet
  • 21,353
  • 10
  • 64
  • 97
  • Thank you for your reply. However, I do not think mod_proxy_connect is going to help here, because that only handles CONNECT requests coming from the browser. What I need is transparent reverse proxying. By the way, it all works fine as long as I do not use SSL. – mstud Jul 13 '12 at 13:06
  • I have to redirect the only WebSocket requests and handle https request by apache virtual host itself where I'm redirecting this WebSockets. For this, do I need to `ProxyPass` and `ProxyPassReverse` for `https` as well? – shaik moeed Dec 05 '19 at 06:58