4

When reading blog posts about WAFs and Kubernetes, it seems 90+ % of the posts are written by WAF-providers, while the remaining posts seem to be sceptical. So I would like to hear what your experiences are with WAFs, do they make sense, and if so can you recommend any good open-source WAFs? We are currently not allowed to used American cloud providers, as we work with "person data", and the Schrems II judgement has indicated that unencrypted "person data" is not allowed on their platforms (even if on EU servers).

To my understanding WAF help with the following:

  1. IP-whitelists/blacklists
  2. Rate Limits
  3. Scanning of HTTPS requests for SQLi and XSS
  4. Cookie Poisoning and session-jacking
  5. DDOS (requires a huge WAF cluster)

But I would also think that these problems can be handled elsewhere:

  1. IP-whitelists/blacklists can be handled by the Loadbalancer or NetworkPolicies
  2. Rate Limits can be configured in the Ingress
  3. Handling of SQLi and XSS is done by input sanitization in the application
  4. Server-side sessions bound to IPs can prevent poisoning and jacking
  5. DDOS are hard to absorb, so I have no native solution here (but they are low risk?)

Sure, I can see the advantage in centralizing security at the access gate to the network, but from what I have read WAFs are hard to maintain, they have tons af false positives and most companies mainly use them to be compliant with ISO-standards, and mainly in "monitoring mode". Shouldn't it be secure enough to use SecurityPolicies, NetworkPolicies, Ingress Rules and Loadbalancer Rules rather than a WAF?

Esben Eickhardt
  • 3,183
  • 2
  • 35
  • 56
  • Whether it is appropriate to use a WAF on Kubernetes or not depends on your requirement, which is an open-ended question and has no one-size-fit-all answer. You might do some handling like rate limits or IP-whitelist inside Kubernetes, but this means that the traffic has gone into your K8s cluster. If you would like to stop specific unwanted traffic from reaching your K8s cluster, then you could use WAF. – victor6510 Jul 14 '22 at 04:15
  • Sure the traffic has gone "into" the K8s cluster, but if it is rejected at the top-layer services, then it shouldn't matter. If we install a WAF in kubernetes, e.g. modsecurity, then the traffic will also get "into" the cluster before it is dropped. – Esben Eickhardt Jul 15 '22 at 09:24

3 Answers3

2

A WAF is not strictly necessary on Kubernetes — or on any other deployment platform. Honestly, even after consulting for dozens of companies, I've seldom encountered any site that used a WAF at all.

You're right that you could duplicate the functions of a WAF using other technology. But you're basically reinventing the wheel by doing so, and the programmers you assign to do it are not as expert in those security tasks than the developers of the WAF are. At least they are probably doing it as one of many other tasks they are working on, so they can't devote full-time to implementation and testing of the WAF.

There is also a valid argument that defense in depth in computing is a good thing. Even if you have other security measures in place, they might fail. It's worth creating redundant layers of security defense, to account for that possibility.

There's a tradeoff between implementing security (or any other feature) yourself versus paying someone else for their expert work. This is true for many areas of software development, not only a WAF.

For example, it has become popular to use a web application framework. Is it possible to develop your own framework? Of course it is, and sometimes it's necessary if you want the code to have very specific behavior. But most of the time you can use some third-party framework off the shelf. It saves you a lot of time, and you get the instant benefit from years of development and testing done by someone else.

Bill Karwin
  • 538,548
  • 86
  • 673
  • 828
  • I'm not posting this to advocate for or against using a WAF. I don't discount their value, but as I said, most sites I have visited don't use one. I am not a WAF vendor myself, nor have I worked for any WAF vendor. – Bill Karwin Jul 12 '22 at 17:52
  • Thanks for sharing your experience. Our problem is that most major commercial WAFs are hosted by companies in non-EU countries that are considered unsafe third countries (primarily the USA). Due to the GDPR and SCHREMS II judgement this means that our company cannot utilize their services. We are already using layered security and are protected against the OWASP Top 10 without a WAF, and as we have a zero-trust model for security within our "perimiter", the addition of a WAF mostly seems like a way to push the security responsibility away to someone else, which is something big companies love – Esben Eickhardt Jul 13 '22 at 08:21
  • 1
    I wouldn't choose to use a WAF implemented as a remote service, and I'm _in_ the USA. For two reasons: I don't want my security handled externally, and also I'm assuming it would introduce a round-trip over the WAN for every request. If I used a WAF at all, it would be one I can run on my infrastructure. – Bill Karwin Jul 13 '22 at 12:38
  • use EnvoyProxy @BillKarwin. Its open source fast and reliable. – Shahid Roofi Khan Jul 15 '22 at 13:26
  • @ShahidRoofiKhan, Thanks, I used EnvoyProxy at my last job. – Bill Karwin Jul 15 '22 at 15:33
1

A good waf does a lot more than that, and it is independent of the deployment model (kubernetes or else).

A waf can

  • Detect and prevent application level exploits far beyond sqli and xss. Sure, you can make a secure application... but can you actually make a secure application? (A team of sometimes changing developers usually cannot.)
  • Detect and prevent vulnerabilities in underlying layers, like nginx or the OS - or maybe even kubernetes itself.
  • Provide hotfixing known vulnerabilities until they are actually fixed in the code or patched in the underlying component (like for example preventing certain values for certain parameters you know are vulnerable and so on).

So in short, yes, a waf does make sense with k8s too, in fact it is not dependent on the deployment model. A waf is just a layer 7 firewall that understands http, and can look into traffic to find flaws and prevent exploits.

Update:

For example a recent vulnerability was log4shell, in log4j. In a request it was possible to run arbitrary stuff on servers due to a framework level (3rd party) vulnerability. A good, regularly updated waf would prevent that probably even before you read about the problem.

Spring4shell was a somewhat similar vulnerability in Spring, that can also be prevented by wafs. So could Heartbleed, a vulnerability in openssl.

There was a php vulnerability quite a while ago that involved a magic number, sent as any parameter.

Command injection vulnerabilities in any application or component follow specific patterns, and so on.

A waf also has more generic patterns for usual application vulnerabilities including (but not limited to) sql injection and xss. Sure, your application could be secure and not have these. But especially over time, it will for sure be vulnerable, even the best team cannot produce bug free code, and that applies to security bugs too.

As a web application is usually only accessible through http, ALL of that is available for capture for a component that understands http. All application layer attacks (and that's a lot) will come through http and a waf at least in theory is capable of preventing them. Surely it will not always recognize everything, it's not magic, and again, you could all implement it yourself. But it would be very difficult and time consuming. The same as you would not implement an API gateway or a network firewall, you would want to use a WAF to provide a layer of protection to your application and it's underlying components.

On the other hand, it's true that it takes some time to configure for your specific scenario and application. At first, it will probably produce false positives. Then you can decide how to manage those, you can disable entire rules, or remove certain pages or parameters from checks and so on. It does involve some work, maybe a lot for a very complex application. But when it's configured, it provides an additional layer of protection against threats you may not even have currently, but will in the future.

WAF suggestions:

If you are running managed kubernetes (AWS EKS, Azure AKS and the like) then probably your cloud provider's waf is the best choice due to easy setup and good integration (though I understand that might not be an option for you). I don't know of a good one apart from modsecurity if you are running your own. Naxsi would come to mind, and while I don't have experience with it, its functionality seems very limited compared to other options and what's described above.

Esben Eickhardt
  • 3,183
  • 2
  • 35
  • 56
Gabor Lengyel
  • 14,129
  • 4
  • 32
  • 59
  • 2
    So again this question turned out as a WAF promo (: – Your Common Sense Jul 06 '22 at 04:06
  • I appreciate your answer, but would it be possible for you to elaborate? How can a WAF prevent application exploits, when all it does is to scan the HTTP requests? How can it prevent vulnerabilities in nginx and the OS, when all it does is to scan the HTTP requests? I know that @YourCommonSense is teasing, but his comment also has some truth to it. When you read about WAFs they state that WAFs can do EVERYTHING, but when you read into how a WAF works it pretty much scans HTTP requests (sometime using machine learning), blocks IPs, limits rates and sometimes provides dashboards over calls. – Esben Eickhardt Jul 06 '22 at 07:00
  • 1
    @YourCommonSense It did! :) The thing is, sure, you can implement most of this in several different places. But why would you use an api gateway then? A proxy? An authentication gateway? You could just implement any component. A waf is a complex component that's already implemented and you can just use it. – Gabor Lengyel Jul 06 '22 at 07:08
  • @EsbenEickhardt I added a few examples. – Gabor Lengyel Jul 06 '22 at 07:24
  • @GaborLengyel thanks for specifying. I can see how it is more maintanable to scan for these types of attacks centrally, and even that it would be nice to hand the responsibility over to an external vendor that has security as an expertice. I have always seen WAFs as a way to protect applications from developer mistakes, but also as magical golden bullets that fixes all security. At our company we have the issue that we cannot use American WAFs due to the SCHREMS II judgement, so no AWS, GOOGLE, AZURE og Cloudflare WAFs for us. Do you know of any open source or European WAFs? – Esben Eickhardt Jul 06 '22 at 07:48
  • 1
    @EsbenEickhardt There are the owasp rulesets for modsecurity, which iirc started out as an apache project, but now can be used with nginx too. The benefit of commercial solutions is ongoing updates, like for example log4j would be prevented hours after it was published, and you will not get that (at least not that fast) with something like pure modsecurity. However, some commercial solutions are also based on that, so the foundation is very strong - but the point unfortunately is ongoing, timely support and updates. But for something like hotpatching, it's great. – Gabor Lengyel Jul 06 '22 at 07:54
  • The Modsecurity plugin for NGINX ingress controller is also the one I have run into. I will check if I can plug it into the ingress controller, we have on our managed kubernetes. Thanks for your input. Before closing this thread, I will still make a bounty to get a bit more input. – Esben Eickhardt Jul 07 '22 at 06:54
  • @GaborLengyel I awarded you the 100 points due to your nice answer and good points. The only think I need before I can accept an answer is a suggestion for an "open-source waf" for Kubernetes :) – Esben Eickhardt Jul 15 '22 at 09:48
  • 1
    @EsbenEickhardt Thank you! If you are running managed kubernetes (AWS EKS, Azure AKS and the like) then probably your cloud provider's waf is the best choice due to easy setup and good integration (though I understand that might not be an option for you). I don't know of a good one apart from modsecurity if you are running your own. [Naxsi](https://github.com/nbs-system/naxsi) would come to mind, and while I don't have experience with it, its functionality seems very limited compared to other options and what's described above. – Gabor Lengyel Jul 15 '22 at 10:16
  • @GaborLengyel It looks like we will have to due with modesecurity on top of our layered security. I agree with your point about protecting the developers, as it is hard to code without making any bugs! It is a balancing act. As a developer myself, I often feel suffercated by all the redundant security layers that often prevent me from carrying out my job. – Esben Eickhardt Jul 15 '22 at 10:49
  • 1
    @EsbenEickhardt I as a security guy sometimes code my hobby projects, some of which are actually public and used online by people. I find it interesting that putting on my developer hat is entirely different from putting on my security hat - when reviewing my own code, I sometimes find security weaknesses that I then fix, and push myself hard not to make the easy shortcuts. :) So yeah, I totally see your point. – Gabor Lengyel Jul 15 '22 at 11:08
  • @GaborLengyel I started as a Data Scientist, but as models are worth nothing if they are not put into production, I have had to learn a lot about infrastructure, security and architectures. I developed a pretty decent setup in Azure, but due to Schrems II, we now have to move to a EU cloud. This means that I now had to learn Kubernetes and how to harden that. EU clouds are not as mature as Azure/AWS, so you e.g. do not get a WAF by default. But at least their documentation is better than Azure's :) – Esben Eickhardt Jul 15 '22 at 11:23
  • @GaborLengyel I just added your WAF suggestions, and accepted your answer. – Esben Eickhardt Jul 15 '22 at 11:53
0

WAF &/or API Gateway you may call, play a very vital role in a web application that many developers fail to understand initially.

First and foremost note that its another "out of process" component of your application that assumes all your attack surface

Least it can provide is to play the role as a "Circuit Breaker". For example your main kubernetes based deployment is down, for multiple reasons, this layer can provide some maintenance pages to your users

Further to that, it can provide caching of response, aggregation of responses from different microservices, buffering, prevention of injection types of attacks, centralized request logging, request analysis, TLS termination, Authentication decoupling, TLS translations, HTTP translations, OWASP protection and the list goes on. See this brief video for one reference implementation: link

There is a reason why a web application like Google Search and all other big similar ones rely on a WAF/API Gateway!

Shahid Roofi Khan
  • 957
  • 1
  • 8
  • 19
  • Do you have any suggestions for WAFs? Do you have experience with Nginx's ModSecurity? – Esben Eickhardt Jul 14 '22 at 08:25
  • use Azure or AWS cloud versions. They are most powerful these days. There is one as WAF plugin for EnvoyProxy (free open source API gateway based on C++, developed and used by Google) there are certain free features as well in that @EsbenEickhardt – Shahid Roofi Khan Jul 14 '22 at 11:30
  • We have to be compliant with the Schrems II judgement, which means that no "person data" is allowed through American companies unencrypted. This also means that we cannot use American cloud providers for this, even if their servers are located in Europe. – Esben Eickhardt Jul 15 '22 at 08:57