Guide to Kubernetes Security Monitoring

The importance of security in the multifaceted ecosystem of Kubernetes clusters can’t be overstated. Kubernetes is a powerful orchestration platform that enables scalable deployments across diverse platforms. However, the complexity and dynamic nature of these deployments can pose unique security challenges that require careful consideration.

This article is a comprehensive guide aimed at strengthening your understanding of Kubernetes security monitoring. We will discuss why monitoring is important and outline practical techniques for securing and monitoring a cluster.

What is Kubernetes?

Kubernetes (also known as K8s) is an open-source container orchestration system purpose-built to automate the deployment, scaling, and management of containerized applications. It cohesively bundles together related application containers into pods and then orchestrates the allocation of resources, networking, and storage of these pods across a cluster of nodes.

Kubernetes, being open-source, is adaptable to various environments, such as public clouds, private clouds, hybrid setups, and even on-premise environments. This versatility has made Kubernetes a staple of many modern IT infrastructures.

Architecture

A Kubernetes cluster contains the following core components:

Control plane

The control plane is regarded as the nucleus of a Kubernetes cluster. It’s responsible for orchestrating all of the cluster operations and comprises the following entities:

• API server: The API server (kube-apiserver) acts as the front-end to the cluster’s control plane. It exposes an API that allows users to create, read, update, and delete Kubernetes objects.
• Scheduler: The scheduler assigns workloads to nodes based on various factors, like node capacity, resource requirements, and affinity and anti-affinity rules.
• Controller manager: This is a collection of controllers that maintain the desired (configured) state of resources in a cluster. For example, they ensure the desired pod state, replicate pods during failures or excessive traffic and manage other resources like services and nodes.
• etcd: A distributed key value store that serves as the primary database in a Kubernetes cluster.

Nodes

Nodes are the worker machines that house the containerized apps. Their main responsibilities are to execute the tasks specified by the control plane and provide the resources for running pods. Each node comprises the following:

• kubelet: An agent that runs on all nodes. It communicates with the control plane to receive instructions and manages the lifetime of pods on the node.
• Container runtime: The software that provides the runtime for containerized applications. Common container runtimes are containerd and cri-o.
• kube-proxy: Implements network services on each node, which enables pods to communicate across the cluster.

Why use Kubernetes?

Kubernetes offers several benefits for modern deployments:

Automated rollouts and rollbacks

Kubernetes automates the process of deploying and updating applications. This eradicates downtime and decreases the risk of errors.

Autoscaling

Kubernetes can automatically scale an infrastructure up or down based on instantaneous needs. It constantly monitors resource utilization and traffic patterns to adjust the number of pods in real time. This not only boosts performance but also optimizes resource usage and reduces the overall cost of running applications.

Self-healing

Kubernetes clusters are self-healing, which means that they can automatically detect and recover from failures. The Kubernetes control plane is capable of restarting failed pods, rescheduling them to run on healthy nodes, and replacing failed nodes.

Secure secret storage

Kubernetes is an all-in-one orchestration platform that also provides secure storage for sensitive information, like passwords, OAuth tokens, SSH keys, and API keys. This allows applications to access secrets securely and easily without having to go beyond the cluster.

Extensibility

Kubernetes has a mature ecosystem of plugins and extensions. Moreover, since it’s open-source, developers can also tweak its source code to better align with specific business needs.

Why is it important to monitor Kubernetes for security?

Here are a few reasons why security monitoring of a Kubernetes cluster is crucial:

Detecting anomalies and threats

Continuous monitoring helps in detecting anomalous behavior or suspicious access that may indicate an active or impending attack. By analyzing metrics, logs, traffic, and events, security teams can promptly identify potential threats and take proactive measures to mitigate risks.

Avoiding downtime

Security breaches can lead to significant downtime, service disruptions, and financial losses. By regularly monitoring Kubernetes clusters, you can identify and resolve potential problems before they lead to potential outages. For example, suppose you notice that the number of pod replicas is not increasing despite repeated traffic surges. In that case, you can take corrective action before the cluster goes into an unhealthy state.

Preventing resource abuse

Kubernetes clusters typically run multiple applications. To prevent any single application from abusing resources, you must monitor resource utilization metrics like CPU, memory, and network bandwidth. For example, suppose monitoring reveals that some application pods are hogging all the CPU resources. In response, you may implement a resource quota to ensure fair CPU allocation among all the applications.

Identifying exploitable vulnerabilities

Kubernetes clusters are constantly evolving as new containers get deployed and updated. This dynamic nature increases the risk of vulnerabilities being introduced into a cluster. Security monitoring is pivotal in identifying these vulnerabilities before adversaries can take advantage of them.

Achieving and maintaining compliance

Many industries, like finance and pharmaceuticals, are subject to compliance regulations and standards that mandate stringent security controls. Continuous monitoring is important to maintain and demonstrate compliance with these regulations. For example, you may collect, analyze, and present audit logs as proof of adherence to regulatory requirements.

Protecting sensitive data

Monitoring for unauthorized access or attempted data breaches is crucial to secure the sensitive data stored in a Kubernetes cluster. For example, if a surge of suspicious failed GET requests overwhelms the API server, it indicates a potential attack or intrusion. Regular monitoring can help you detect and avert such attacks promptly.

Ensuring operational continuity

Continuous security monitoring contributes to the overall operational continuity of Kubernetes clusters. It ensures that security measures evolve along with the changing threat landscape, thereby safeguarding business operations.

Kubernetes security monitoring techniques

Now that we know just how important Kubernetes security monitoring is let’s explore some practical techniques to secure and monitor Kubernetes clusters.

Enforcing network security

Robust network security controls can significantly reduce the attack surface of a Kubernetes cluster. Here are a few tips to implement this:

Use network policies

Using network policies, you can create rules that govern how pods and services communicate with each other. Using them, you can restrict traffic based on specific criteria, such as pod labels, namespaces, or ports.

For instance, the following network policy definition ensures that only pods that have the label access: “true” can access pods labeled as app:apache.

apiVersion: networking.k8s.io/v1 
kind: NetworkPolicy 
metadata: 
  name: allow-apache-access 
spec: 
  podSelector: 
	matchLabels: 
  	app: apache 
  ingress: 
  - from: 
	- podSelector: 
    	matchLabels: 
      	access: "true"   

Container Network Interface (CNI) plugins

Leverage Container Network Interface (CNI) plugins to have more control over the cluster’s network. Open-source plugins cover a wide range of networking and monitoring use cases. For example:

• Calico: An open-source plugin that simplifies network policy enforcement and enables granular control over network traffic between pods.
• Flannel: An open-source plugin that offers a simple overlay network for Kubernetes.
• Cilium: A versatile CNI plugin that leverages eBPF (extended Berkeley Packet Filters) technology to enforce network policies and provide deep visibility into network communication.

Incorporate a service mesh

Implementing a service mesh like Istio or Linkerd helps add robust security controls for microservices communication within Kubernetes. These meshes offer an array of essential functionalities, including encryption, access control, traffic management, observability, and fault tolerance.

For instance, the monitoring features of Linkerd make it easy to track many networking metrics, including request volume and latency, TCP bytes, success-to-failure ratio, and per-service metrics.

Encrypted communication

Encrypting network traffic between pods, nodes, and services ensures that sensitive data remains protected even if an attacker manages to infiltrate the network infrastructure. TLS (Transport Layer Security) and IPsec (Internet Protocol Security) are the most commonly used protocols to encrypt network traffic in Kubernetes clusters.

Enforcing pod security

Securing and regularly monitoring pods plays a pivotal role in improving the overall security posture of a Kubernetes cluster. Here are a few recommendations to consider:

Define pod placement rules

Pod placement is the process of assigning pods to nodes. This decision has significant security implications, as it determines which pods share resources and how pods are segregated. You can define the rules for pod placement in the pod specification files. Examples of pod placement methods are:

• Affinity and Anti-Affinity: Affinity rules allow pods to be placed on the same node or different nodes based on specific criteria, such as labels or selectors. Anti-affinity rules guarantee that matching pods are not scheduled to run on the same node.
• Node labels: You may assign labels to nodes to govern the nature of pods that are scheduled to run on them.

It’s recommended to use efficient pod placement strategies that keep your application containers secure. For example, you may assign security labels to nodes to indicate their security posture. Only the pods that meet the security requirements defined by those labels will then be allowed to run on those nodes.

Pod security standards

Kubernetes comes with three different pod security policies that can be configured based on the nature of the pod:

• Privileged: This policy enforces no restrictions and permits pods to perform all possible operations.
• Baseline: A moderately restrictive policy that enforces some constraints to mitigate common privilege escalations.
• Restricted: The most limiting policy that enforces additional security constraints on top of the baseline policy.

It’s important to exercise caution while choosing policies for your pods. For pods that don’t require elevated privileges, the baseline policy is a good choice. For pods that are considered to be high risk (e.g., those running security-critical apps), always use restricted policies to decrease the attack surface. A privileged policy should only be considered as a last resort, as it has the potential to expose the cluster to security vulnerabilities.

Sensitive data in pods

It’s recommended to never store sensitive data directly inside application pods. Instead, it should be safeguarded using Kubernetes Secrets. Additional security controls that you may enforce include configuring role-based access control (RBAC), encrypting data at rest and in transit, and performing regular key rotations.

Monitor pod logs and communication

Continuous monitoring of pod activity and access logs offers valuable insights into pod behavior and helps identify potential security threats. Use the following command to check the logs of any pod:

kubectl logs <pod_name>   

Modern monitoring tools, such as the Site24x7 Kubernetes monitoring tool, offer an end-to-end monitoring solution for application pods in a Kubernetes cluster.

Auditing logs

Kubernetes offers a built-in auditing feature to maintain visibility and control over all activities within a cluster. This includes activities performed by users, applications, and the control plane. Administrators can control which events are recorded using fine-grained audit policies.

An audit policy defines different rules to log or skip events. The audit level of an event is determined based on the first matching rule. The four supported audit levels are:

• None: If this rule is matched, don’t log the event.
• Metadata: If this rule is matched, log only the metadata.
• Request: If this rule is matched, log the request body along with the metadata.
• RequestResponse: If this rule is matched, log the request body, the metadata, and the response.

For example, the following block defines a network policy that logs the metadata and the request body of events that change ConfigMaps in a namespace:

 # Log requests made to change ConfigMaps in the sample namespace 
  - level: Request 
	resources: 
	- group: "" 
  	resources: ["configmaps"] 
	# Only apply this rule to the specified namespace 
	namespaces: ["sample"]  
 

Similarly, the following block ensures that authenticated requests to certain URL paths are not logged at all.

 # Skip authenticated requests to the specified URL paths: 
 - level: None 
	userGroups: ["system:authenticated"] 
	nonResourceURLs: 
	- "/api/example/*" 
	- "/health"  
 

Security measures: Securing secrets

Enforcing strict security measures for secrets like passwords, tokens, and API keys is essential to prevent unauthorized access or sensitive data exposure.

Use encryption at rest and in transit

Kubernetes provides built-in features to encrypt secrets at rest and in transit using cryptographically secure methods. These secrets can be seamlessly referenced by pods using environment variables or mounted files without requiring any custom coding.

Best practices for accessing secrets

• Use Kubernetes RBAC to restrict access to secrets based on the principle of least privilege. Access to any particular secret should be limited to pods and applications that use it.
• Don’t hard code secrets in application code or store them in ConfigMaps. Instead, store them using dedicated Secrets objects.
• Automate the generation, storage, and rotation of secrets to reduce the risk of human error.
• Store secrets only for as long as they are needed. Regularly audit for and remove any unnecessary secrets to minimize your attack surface.
• Keep secret manifests under version control to track changes, enable rollbacks, and maintain a history of secret usage.
• Enable logging for secrets and regularly monitor logs for any unauthorized operations.

Image security best practices

Container images help you design and run application pods in Kubernetes. Follow these best practices to avoid common image security pitfalls:

• Verify the sources of container images before using them for your applications. Moreover, regularly scan all images for vulnerabilities using image scanning tools like Dagda. These tools can identify exploitable vulnerabilities in base images and any installed packages.
• Implement a robust and automated update and patch management system to keep your images up to date at all times.
• Adhere to the principle of least privilege while granting access to image registries.
• Remove unnecessary or outdated images from your registries to decrease your attack surface.
• Keep your images as lightweight as possible. Remove any components, libraries, or modules that are not used.
• Implement runtime security measures to monitor and control the behavior of container images in a live cluster. Use tools that offer runtime protection and behavior analysis, such as Falco, to detect and mitigate threats at runtime.

Admission controllers

Admission controllers process requests after they have been validated, but before they are allowed to make any side effects (create, delete, or modify objects). They are a great tool to reject unauthorized requests and improve the overall security of a cluster.

Kubernetes supports two types of controllers: validating and mutating. Mutating controllers can make changes to requests, whereas validating controllers can’t. Admission control happens in two phases: the mutating controllers are executed in the first phase, and then the validating admission controllers in the second phase. A request is rejected if it fails to pass through any of the controllers.

Kubernetes supports several types of admission controllers to cater to a wide range of security use cases. For example:

• CertificateApproval: Verifies the approving user’s authorization to approve CertificateSigningRequests resources.
• DenyServiceExternalIPs: Prevents users from creating services on external IPs.
• EventRateLimit: Controls the rate at which the API server receives requests, preventing it from being overwhelmed.

To enable an admission controller for your cluster, use this command:

 kube-apiserver --enable-admission-plugins= EventRateLimit 
 

To disable an admission controller:

 kube-apiserver --disable-admission-plugins= EventRateLimit 
 

Conclusion

Implementing robust security controls and performing regular monitoring of a Kubernetes cluster is crucial for protecting sensitive data, detecting and mitigating threats, and ensuring high availability. This article has offered many insights and recommendations to secure and monitor Kubernetes clusters; we hope you found it useful.