Kubernetes Security: The Overview
Emerging open-source technologies have always faced the security hurdle. Kubernetes, not being an exception, can also be infected in many ways as well. To list a few possible vulnerable gateways into a cluster, there can be
- A vulnerable container: Vulnerability in the configuration of a pod can let the attacker get into a container and probe for weaknesses in a network, process, controls or file system.
- A vulnerable container to container connection: Attacker can attempt to connect a rogue container to other containers and spread malicious codes and files through them.
- Unauthenticated Access: Kubernetes components are created and managed using APIs. If not authenticated properly, attacked can deploy malicious containers or perform DoS attacks.
- Excess permission: Post authentication, authorization is the second level of guard. It is needed for limiting access to users and groups. If not configured properly, service accounts and third party installations can get root permissions and then there’s no limit to the damage it can make.
- Vulnerable host: When talking about Kubernetes security, a topic which often gets neglected is the security of the servers or in this case, the host. Attack on the node systems which form the cluster is as big a threat as getting root access to the cluster.
- Cloud Loopholes: Cloud platforms (AWS, Azure, GCE, etc.) often expose metadata services locally to instances. By default, these APIs are accessible by pods running on an instance and can contain provisioning data such as kubelet credentials. These credentials can be used to escalate permissions within the cluster or to other cloud services under the same account.
This is just a few of the possible attack grounds that can be exploited in a Kubernetes cluster and attackers will keep on coming with new ways to do so. Microsoft has assembled and covered major ground in listing many such vulnerabilities into a Threat Matrix, which can be seen here
However, these attacks can be minimised with proper precautions and adherence to best practices. To prevent any attack, we first need to first look at the way an attack is carried out by an attacker and then create a counter plan.
From where to start?
The origin of most attacks lies in the vulnerability of the network. Hence, network inspection is the first step while carrying out an attack. So the first step to protection should start at the network level.
The second layer of patroling comes at the container level. Containers should be monitored for processes, system calls, and file system activity for potential rogue behaviour. Any suspicious processes or attempts to escalate privileges and break out of the container can be spotted at this stage.
Apart from setting up defensive walls, it is also necessary to continuously monitor the behaviour of the cluster components to make sure that any dormant attacker or malicious code inside a pod don’t become active.
So let’s move in this order and get a deeper look
Securing a network
There are two parts when we talk about network security, networking between pods and networking of pods with the outside world.
When talking about internal networking, the first line of defence that comes into mind is Network Policy. A network policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints. Network Policy is not needed to allow communication in pods, but to restrict it and hence often gets ignored. A good networking system should have network policies configured so that the pod to pod interaction is limited to only suffice the functional requirements. Always keep in mind the principle of least privilege
Now coming to interaction with the outside world, one possible solution is a container firewall. Container firewall is the name given to the policy-based, declarative network security model utilized by Kubernetes to secure ingress and egress traffic.
There are many ways a container firewall can be used to secure a network
Layer 3 and 4 filterings: Unlike simple network segmentation, this approach includes Kubernetes network policy to update rules in a dynamic manner, protecting deployments as they change and scale.
Web Application Firewall (WAF): This approach uses methods to detect common attacks on web-facing containers.
Layer-7 container firewall: This approach of protection is based on network application protocol whitelisting and is capable of providing a container firewall with Layer 7 filtering and deep packet inspection of inter-pod traffic.
The Kubernetes community has to offer a wide range of third-party solutions for this problem too; one of them being Istio. Istio is a completely open-source service mesh that layers transparently onto existing distributed applications. It is also a platform, including APIs that let it integrate into any logging platform, or telemetry or policy system. However, not designed to be a security tool to detect attacks and threats, Istio creates a service mesh for managing service to service communication, including routing, authentication, and encryption. But a proper implementation of Istio can greatly help in securing network communication.
Container inspection
Containers attacks are focussed on leveraging privilege escalation or vulnerabilities in the Linux kernel, packages, libraries or applications themselves.
Ideally, a Kubernetes cluster should be created by keeping microservice architecture in mind, ergo every container should use only those packages and libraries which are needed to carry out its operations. This highly reduces potential threats that a container might pose.
Apart from that, there are many third-party solutions to tackle these problems and make sure that your containers won't pose any threat in the cluster. A couple of them being Clair and Grafeas.
Clair is an open-source project for the static analysis of vulnerabilities in application containers. In regular intervals, Clair ingests vulnerability metadata from a configured set of sources and stores it in the database. Clients use the Clair API to query the database for vulnerabilities of a particular image.
Grafeas is an open-source artefact metadata API that provides a uniform way to audit and govern the software supply chain. Grafeas defines an API spec for managing metadata about software resources, such as container images, Virtual Machine (VM) images, JAR files, and scripts. It can be used to define and aggregate information about your project’s components.
Furthermore, for runtime monitoring, Falco is a great solution too. Falco is an open-source cloud-native runtime security project originally created by Sysdig and now part of the CNCF. Sysdig Secure extends Falco’s rich detection for easier security policy management across the container lifecycle.
Securing Kubernetes infrastructure and its resources
This section can be divided into three sections: controlling access to the cluster, securing kubelet and securing etcd.
Controlling access to the cluster
Firstly, all communication through API should be encrypted using TLS and the majority of installation methods will allow the necessary certificates to be created and distributed to the cluster components.
All API calls have to and should go through an authentication and authorization process. Choose the authentication method wisely. For smaller clusters, authentication using bearer tokens might suffice but for larger clusters in which multiple developers will require access, opt methods like LDAP. For authentication of API clients and installations, create service account or use x509 client certificates.
Post authentication, every API call passes through an authorization check. This is done through Role-Based Access Control. Each role is a bundle of namespace or cluster scoped permissions which contain a combination of verbs (get, create, delete) with resources (pods, services, nodes). For authorization, each the RBAC component is mapped to the user or group for the incoming calls.
Furthermore, firewall rules can be manually configured to prevent unauthorized access.
Securing Kubelet
The kubelet is the primary node agent that runs on each node. Kubelets expose HTTPS endpoints which grant powerful control over the node and containers and by default Kubelets allow unauthenticated access to this API.
The first step to protecting a Kubelet is to add either X509 client certificate authentication or bearer token authentication. However, if not rejected by authentication methods, Kubelet treats the requests as anonymous which should be disabled and any unauthenticated requests should be reverted with a 401 unauthorised response. To achieve this, anonymous-auth flag should be set to false when starting a kubelet.
Once authenticated, all requests are authorized by default. The second layer of protection can be configured to further limit access to kubelet. This can be done by adding a subdivision to the access of requests
Securing etcd
Write access to etcd can be used to easily escalate overall access if you think about it. Hence, administrators should always use strong credentials from the API servers for their etcd server, such as mutual auth via TLS client certificates, and it is often recommended to isolate the etcd servers behind a firewall that only the API servers may access.
Host Security
Kuberenetes cluster is formed by a combination of multiple nodes. Securing these nodes is as important as securing the cluster. This section can go to great lengths as securing a machine is a vast topic in itself.
To keep the focus on Kuberenetes, the only part I would cover is that when selecting the tools and technologies for securing the host, ensure that they are cluster-aware and implements the security accordingly. Also, review all ports which are externally accessible and block unnecessary ports communication. Every external port should either be authenticated or should get restricted access to a whitelisted source only.
Recommended Pre-Deployment Security Steps
Apart from following all the steps mentioned in this blog for securing the clusters, there are also some best practices which should be adhered to. These steps include and are not only limited to
- Use namespaces to create logical boundaries
- Use taint and tolerations to limit pod reach
- Enable audit logging
- Limit the use of alpha and beta features
- Restrict Linux capabilities
- Enable Security-Enhanced Linux (SELinux)
- Utilize Seccomp for shell features
- Configure Cgroups to limit resource utilization
- Use a minimal Host OS
- Update system patches
- Verify third party installations
- Use ResourceLimits against DoS attacks
Benchmarking
Another best-practice is benchmarking of your cluster. Similar to electronics, benchmarking can also say a lot about your cluster. The Center for Internet Security (CIS) Kubernetes Benchmark is a reference document that can be used by system administrators, security & audit professionals and other IT roles to establish a secure configuration baseline for Kubernetes. CIS Benchmarks are developed by an open community of security practitioners.
CIS document for Docker can be found here. And for Kubernetes, here.
Parting note
Security is an essential and really vast topic. I have tried to cover the very basic of it in this article. For more articles on advanced security implementations and practices in Kubernetes world, stay tuned.
