6

I just performed a basic DDOS from my computer:

websocket-bench -a 2500 -c 200 wss://s.example.com

Which to my total dismay crashed my server! The WS works by connecting to my nginx proxy:

    location / {
            proxy_pass http://sock;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header        X-Real-IP               $remote_addr;
            proxy_read_timeout 60;
    }

    upstream sock {
            server 127.0.0.1:1203 fail_timeout=1s;
    }

and locally on the server on port 1203 is ratchet. The setup for ratchet is that I allow any connection and the first onMessage performs authentication and if invalid the connection is closed.

I also have tried authentication by passing headers on the first connection and if invalid the socket closes but this has not helped at all and nginx still reaches 100% resources and then crashes.

What should I be analysing to prevent these crashes?

When changing the upstream to another closed port (i.e disabling it) the server still crashes.

maxisme
  • 3,974
  • 9
  • 47
  • 97
  • You can't prevent DDOS by "tweaking" some config variables or installing a program that prevens it. Your problem isn't the ddos, it's the fact that once crashed - service doesn't come back up. That's why we use `supervisord`. Also, using nodejs instead of ratchet would yield significant gains. – N.B. Jul 29 '18 at 12:52
  • 1
    Yeah but the point of my question is that a DDOS is caused when it shouldn't be. My network with 6mb/s upload shouldn't be able to crash a gigabit server with requests - meaning there is something wrong with my config. – maxisme Jul 29 '18 at 13:02
  • If 10,000 people reached my http web server now it would just slow down the server not crash it. – maxisme Jul 29 '18 at 13:04
  • 10k for nginx and node wouldn't be a problem. You'd use services like Cloudflare to prevent it. If you *need* 10k people connected, you use multiple nginx instances (your DNS points to multiple IP addresses that handle your domain) and each nginx load balances to several ratchet/nodejs instances that handle web sockets. That's how scaling is done. – N.B. Jul 29 '18 at 13:13

1 Answers1

1

Either change your authentication logic so Nginx handles it, or implement request and connection limits within Nginx to control how many connections are accepted and passed to the upstream server

miknik
  • 5,748
  • 1
  • 10
  • 26
  • limits does not seem to affect anything unfortunately. I can't quite believe that it is nginx crashing my server! I am worried that it may be my ratchet php script. – maxisme Jul 29 '18 at 00:03
  • Hmm, ok. What happens if you remove your upstream block altogether and change your proxypass directive to `proxy_pass http://127.0.0.1:1203` ? You normally use an upstream directive to define a group of servers, if your ratchet instance fails then `fail_timeout=1s` will make Nginx mark the server as down for 10 seconds. Usually it would route requests to other servers in the upstream block, but you dont have any. Personally though I would recommend the Nchan websocket module for Nginx, works a treat. – miknik Jul 29 '18 at 17:03