Kubernetes has become the platform of choice for deploying modern applications. A Web Application Firewall (WAF) is the most common solution to providing run-time protection for applications (well, second most common, after blind -faith and protective amulets). The question is, how do you put a WAF in place for applications running on Kubernetes?
As for most IT questions, the obvious answer is, of course, “it depends.” But on what?
In this webinar, we look at how a WAF works, where to insert a WAF in your infrastructure, and the best way for a platform engineering team to create self-service WAF configuration on Kubernetes. We explore some sample configurations, and provide a demo of NGINX App Protect WAF in action.
This is research on the opinions of on the biggest challenges to running applications in containers, Kubernetes and or serverless (from now on I’ll call these modern apps).
As you can see ## security is number 1. Complexity is also a problem, along with the skill to manage it.
These are opinion based, so maybe they are more emotional than factual?
Well no, it looks like application layer security is still a big problem - with web app and software vulnerabilities (which have a lot of overlap) scoring very high on this survey on the causes of compromise.
So application layer exploits are a serous threat. I’m not going to talk about the costs of compromise, those figures are quite specific to an organization – but we all know you don’t want to get hacked.
As if that’s not bad enough, DDoS is a significant ptoblems.
What we are seeing in the current Marketplace for 2022, is that DDoS attacks are at an all-time high. Just a quick review that a distributed denial of service or DDoS attack is a cyberattack where multiple connected devices, known as a botnet, are controlled from a central point with the goal of the attacker sending malicious requests to consume all available network bandwidth or resources on a target network, system or website making them slow to respond, or shutting them down entirely and preventing legitimate requests from being fulfilled.
Why the increase in DDoS attacks? Several security trade publications including a recent report from Secure List point to the spike in DDoS attacks this year due to the war in the Ukraine and the current geopolitical situation which has resulted in subsequent attacks on businesses specifically in the government and financial sectors. These types of DDoS attacks are being motivated by what is called patriotic hacktivism. The Secure List also notes these trends, that DDoS attacks have grown by 4.5x in volume in Q1 of 2022 than at the same time last year and that DDoS attacks larger than 250 Gbps grew by 1300%. Additionally, according to F5 Labs 2022 report on DDoS Attack Trends based on the analysis of attack data collected by F5s Silverline team from 2021, DDoS attacks are continuing to grow in both size and complexity. In recent years bad actors have increasingly been launching what you see in this graph here, a multi-vector attack, which shows a combined volumetric and application attack at the same time. In November 2021, Silverline observed and mitigated the largest attack it had ever seen. The attack was targeted at an ISP/hosting customer and lasted for just four minutes and reached its maximum attack bandwidth of almost 1.4 Tbps in only 1.5 minutes. Even though the volumetric attack size dwarfs the application attack size, the application attack is just as serious. The resources being requested by the application attack could have easily consumed a web or database server.
OK, here we are going to compare a traditional application running on a virtual machine, or maybe even a physical server. Is it vulnerable to an application layer hack? Yes. How about a DDoS attack. - well if it’s publicly facing, then yes too.
What about a modern app? Built of multiple services running in Kubernetes? Well- honestly there is no difference. Your app needs exactly the same protection – but probably delivered in a different place and managed a different way.
We’re not going to cover massive volumetric attacks here, but F5 does have a DDoS service with tens of terabits of capacity, globally distributed and ready to protect you: F5 distributed cloud. While massive volumes and huge numbers of connections get the headlines, layer 7 (as we saw earlier) is often part of the attack, and needs different protection tools.
Let’s talk about defending against application layer compromise. One solution is to write perfect software. Software that always sanitizes user input and isn’t vulnerable to any of the other attacks out there. Oh, and don’t forget to only use perfect software libraries and frameworks in your app too. OK, so that’s not going 0t happen, so what you need is a device that ca inspect (and decrypt) ## your application traffic, then assess ## whether it’s a treat, then take action## Sounds simple, but anyone who has been working with traditional enterprise security products like a Web Application firewall knows that’s not always the case. Or ever the case, in fact.
OK, now let’s talk about application layer DDoS attacks. This tends to work a bit differently because behavioral detection is the preferred way . So first we need to build a baseline of the site behavior and performance. So that wen an attack occurs ## we can detect a change, and then act## - again this sounds easy, but accurately detecting an attack, and creating a way to detect the specific traffic causing the problem, and then blocking it in a way that still allows good traffic through is far from simple.
NGINX offers solutions for both application layer attacks and application DDoS – inventively called NGINX app Protect WAF and NGINX App Protec DoS. Between then they can help prevent both application layer DDoS attacks and Application layer exploits like SQL injection (yes still a thing, despite being old enough to drink), or cross site scripting.
Lets take a moment to see how they work in a bit more detail.
First app protect waf – when we talked about the inspect capability a while ago this answers the “inspect for what” question you might have had ##
I’m not going to go through all the details here – it’s very well documented, but suffice it to say we look for obvious things like signatures, but also compliance to protocol standards, request sizs, the values of parameters etc. We also look at the repose content to ensure that doesn’t contain thinks like credit cards numbers.
Of course, just because a request has triggered a signature, that doesn’t mean that we automatically want to block it, we need to be a bit more careful if we are not going to overwhelm the alerting system, or block traffic we shouldn’t. ##
This is where the violation rating engine comes it. The engine takes a number of factors into account to build a model and decide what the rating is, from 0-5.
OK, the final step is to take action ## Obviously we can do mothing, alert or block the request – and by default only ratings of 4 or over cause a blocking action, but you can configure your policy how you like to block or alert at different levels.
So that’s WAF, what about DoS?
DoS works in a slightly different way – by building baseline model of load, ## and then noticing na anomaly – art which point the traffic can be examined ## rules created to detect the problematic traffic, then used to take action on the incoming traffic.
The taking action part is adaptive – we want to do the least invasive thing first, so as to block only the bad traffic – so we will apply the first level of blocking, check the effect on the site load, then if there is no change – aspply a borader rule set, and then of still no change we can start rate limiting all traffic – not ideal, but the best choice if nothing else works,
So that’s a high-level overview of how you protect applications – what about applications in Kubernetes?
First, we should look at how you expose apps to the cold-hard world outside the cluster.
LoadBalancer with NodePort is one of the most popular ways to expose apps. A LoadBalancer service accepts external traffic and distributes it across the nodes in the cluster. A NodePort service opens a specific port on every node in the cluster, and forwards any traffic sent to the node on that port to the corresponding app. It’s most often used in a cloud platform and is a good choice for small, static deployments.
However, using LoadBalancer to expose your apps becomes challenging in dynamic environments where app pods need to scale to meet changing levels of demand. Because each pod gets its own IP address, a popular app can have hundreds – or even thousands – of IP addresses to manage. In most cases, the external load balancer connects to the services via NodePort as shown in the following diagram. While this guarantees traffic is distributed evenly across the nodes, load balancing to the services isn’t possible, so you might encounter service overload.
Ingress controllers and gateways provide far more granular control and management of external interactions with pods. You can easily use an Ingress controller not only to control ingress traffic but also to deliver service‑level performance metrics and as part of a security policy. Ingress controllers have many features of traditional external load balancers, like TLS termination, handling multiple domains and namespaces, and of course, load balancing traffic. Ingress controllers can load balance traffic at the per‑request rather than per‑service level, a more useful view of Layer 7 traffic and a far better way to enforce SLAs.
Also, many organizations have use cases that benefit from deploying an external load balancer with multiple Ingress controller instances. This is especially common when organizations need to scale Kubernetes. The tools are typically managed by different teams and used for different purposes:
Load balancer (or ADC):
Owner: A NetOps (or maybe SecOps) team
Use case: Outside Kubernetes as the only public‑facing endpoint for services and apps delivered to users outside the cluster. Used as a more generic appliance designed to facilitate security and deliver higher‑level network management.
Ingress controller:
Owner: A Platform Ops or DevOps team
Use case: Inside Kubernetes for fine‑grained load balancing of north‑south traffic (HTTP2, HTTP/HTTPS, SSL/TLS termination, TCP/UDP, WebSocket, gRPC), API gateway functions, and centralized security and identity.)
This diagram shows the load balancer handling distribution of the traffic across multiple clusters, while the clusters have Ingress controllers to ensure equal distribution to the services.
NGINX App rptoect WAF and app protect Dos are the solutions we will be talking btut todat
NGINX App Protect F can be deployed in several different ways:
1 – It can live on NGINX Plus when NGINX plus is used as a load balancer,
2 – If you’re using NGINX Plus as an API Gateway, you can install NGINX App Protect WAF in that instance to protect API’s
3 – You can use it as an Ingress controller for Kubernetes clusters
4 – You can use it as a Per-Service proxy for microservices or
5 – You can also use it as a Per-Pod Proxy for microservices
Each one covers different scenarios and fulfills specific needs in
terms of north-south or east-west based security. With the full ability to have mix-
and-match kind of deployments that impose virtually no restrictions on how
security can be enforced, it always guarantees full control, auditing, and traceability.
For our conversation today, we will focus on using it with the Ingress Controller.
architecturally as shown in this diagram. At the edge, at the ingress, as a k8s service
and as a sidecar.
Our customers can deploy NGINX App Protect WAF in different environments:
OK, back to my application characteristic matrix – and as the title says, this is a generalization, but I’ve found that it’s generally true.
Monolithic apps tend to be deployed less frequently## – where as modern apps are being updated deployed frequently, ##
Monolithic apps tend to be upgraded in place, or patched live. ## Modern apps tend to go for rolling redeploys, canary deployments and so on##
Why does this matter? Well, I think it gives us an indication of where we should put out app protect services.
For a lot of use cases, it makes sense to put the protection services as part of ingress ## – because we are already using ingress to direct traffic, it’s already aware of the changes going on with services etc, and if we add a new service, or delete one, then we can add and remove the App protect functions as they are bound to the ingress. It makes operations incidents less likely where either old configuration is hanging around externally, or new services go into place with no protection.
So the NGINX ingress controller – it’s a really good ingress controller – that you can also add App-Protect as well. You can use the standard Kubernetes ingress API (or chose to use NGINX custom resources – more on that shorty) – it’s made by us, and supported by us too. Also depicted in this diagram is NGINX Service mesh – we’ll probably do a whole separate webinar on that soon, so stay in touch.
So now you can have an ingess controller, configured via manifests, that brings powerful security tools right into your k8s cluster.
I said a minute ago that you could use either the standard ingress API or custom resources – and, hopefully you then thought – ok, but why choose one or the other?
<talk through the pro’s and con’s>
I really like the CRD’s because of the logical structure
You can do so with NGINX Kubernetes objects (VirtualServer, VirtualServerRoute, Policy objects) enabling distributed configuration architectures, where administrators can easily delegate configuration to other teams.
Teams (Identity, DecSecOps, DevOps) are responsible for policy objects (NGINX App Protect WAF, SSO, ZT, traffic control).
And these policies get referenced in many areas of the ingress load balancing configuration (Both at the hostname and path level).
At the hostname level, policy objects will be referenced in the VirtualServer Object
At the HTTP path level, policy objects will be referenced in the VirtualServerRoute objects
Now we are going to see this distributed configuration in action where we implement the WAF self-service model.
As you can see the configuration is very familiar to anyone working with k8s manifests.
Explain config
OK, we’re just going to demo app protect WAF – because simulating a DDoS attack is difficult and boring to watch!