Kubernetes Audit Logs

Create a policy file audit-policy.yaml at the /etc/kubernetes/audit-policy.yaml file path. As the API server process the first matching rule defined in the policy file, define the specific rules at the top, followed by the generic one.

Below is the sample policy file from the official Kubernetes documentation.

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"] 

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"] 
  
  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"] 

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version" 

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"] 

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"] 

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included. 

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

You must enable audit logs only on the master node, as the kube-apiserver is part of the cluster's control plane hosted on the master node. Configure the following flags in the /etc/kubernetes/manifests/kube-apiserver.yaml file.

--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log

Update the volumeMounts and volumes section in the configuration file as below:

volumeMounts:
- mountPath: /etc/kubernetes/audit-policy.yaml
name: audit
readOnly: true
- mountPath: /var/log/kubernetes/audit/
name: audit-log
readOnly: false

volumes:
- hostPath:
path: /etc/kubernetes/audit-policy.yaml
type: File
name: audit
- hostPath:
path: /var/log/kubernetes/audit/
type: DirectoryOrCreate
name: audit-log

1. Log in to your Site24x7 account.
2. Download and install the Site24x7 Server monitoring agent (Linux) in Kubernetes cluster master node.
3. Go to Admin > AppLogs > Log Profile > Add Log Profile.
4. Select Kubernetes Audit Logs from the Choose the Log type drop-down.
5. Include /var/log/kubernetes/audit/audit.log to the List of files to search for logs.
   

   That's it. You are set to monitor the audit logs in your on-premises environment using Site24x7 AppLogs.

From the Site24x7 web console, navigate to Admin > AppLogs > Log Profile > Add Log Profile, and enter the following:

1. Enter a name for your Log Profile in the Profile Name field.
2. Choose Kubernetes Audit Logs from the Choose the Log Type drop-down.
3. Choose Azure Functions from the Log Source drop-down.
4. Choose UTC for the Log Timezone.
5. Click Save.
   

Follow the instructions in the Azure diagnostic logs document to configure the Azure resource using an ARM template. When entering the details on the custom deployment blade (steps 7 and 8) in the above document, make sure you edit the following specific to the Kubernetes Audit Logs:

• Resource group: Create a new resource group with a name similar to Site24x7-AKS-Audit-Logs.
• LogTypeConfig: Copy the value from the Log Profile page that you already created within Site24x7 (Admin > AppLogs > Log Profile > <Kubernetes Audit Logs profile name>).
  
  

1. Log in to your Azure portal.
2. Go to Kubernetes services, choose your Kubernetes cluster and click Diagnostic settings.
   

3. Select Kubernetes Audit under Logs, Stream to an event hub under the Destination details and choose the Event hub namespace.
   

   Now, you are set to monitor the audit logs in your Azure Kubernetes Service environment using Site24x7 AppLogs.

Refer to the guidelines in this official documentation to enable or disable Amazon EKS audit logs.

• From the Site24x7 web console, navigate to Admin > AppLogs > Log Profile > Add Log Profile, and then:
  • Enter a name for your Log Profile in the Profile Name field.
  • Choose Kubernetes Audit Logs from the Choose the Log Type drop-down.
  • Choose Amazon Lambda from the Log Source drop-down.
  • Click Save.
• Follow the instructions in the AWS setup section of this document to collect the CloudWatch logs and view them in Site24x7. However, make sure you choose Kubernetes Audit Logs, it's logtypeconfig, and the Log group as /aws/eks/<my-cluster>/cluster/ and continue the instructions defined in the document.

Now, you are set to monitor the audit logs in your Amazon Elastic Kubernetes Service environment using Site24x7 AppLogs.

1. Create and define the audit policy mentioned in the Log backend configurations section in this document.
2. Create a Log Type, as shown in this document.
   • For the Log Type, enter Kubernetes Audit Logs. This will automatically populate the rest of the fields.
   • Ensure that API Upload is enabled.
   • Copy the HTTPS endpoint URL to configure a webhook backend configuration.
   • Click Save.
3. Create a new file in the path /etc/kubernetes/pki/audit-webhook.yaml and copy the content below. Make sure you provide the HTTPS endpoint URL you copied in the above step in the YAML file for the server key.
   

   apiVersion: v1
   kind: Config
   preferences: {}
   clusters:
   - name: kubernetes
   cluster:
   server: "https://logc.site24x7.com/event/receiver/M3NjE4ODc234242342wfwew3zMDhmMTZkYTVmM2I2N2MxOGZiZC9rdWJlY="
   contexts:
   - context:
   cluster: kubernetes
   user: kubernetes-admin
   name: kubernetes-admin@kubernetes
   current-context: kubernetes-admin@kubernetes
   preferences: {}
   users: []

4. To configure a backend webhook audit, set the below flag:
   
   

   --audit-webhook-config-file=/etc/kubernetes/pki/audit-webhook.yaml flag

   in /etc/kubernetes/manifests/kube-apiserver.yaml file.

Now, you are set to monitor the audit logs of your Amazon Elastic Kubernetes Service environment using Site24x7 AppLogs.

For log backend configurations

1. Follow the instructions defined in the Log backend configurations section in this document, and wait a few seconds for your changes to the Kube-apiserver config file to be reflected.
2. Run the below command to check whether the API server is up and running:

   kubectl get pods -n kube-system -w

3. Now, run the below command to open the log file and check whether the audit entry has been created:

   sudo cat /var/log/kubernetes/audit/audit.log

4. If you see a log similar to the sample log mentioned above, it is confirmed that the audit entry is being created.

For webhook backend configurations

1. Follow the instructions defined in the webhook backend configurations section in this document, and wait a few seconds for your changes to the Kube-apiserver config file to be reflected.
2. Run the below command to check whether the API server is up and running:

   kubectl get pods -n kube-system -w

3. As the webhook backend sends events to an external HTTP API, wait for a few seconds (up to one minute), log in to our web client, and confirm the audit entries.

Each Kubernetes API call comprises stages such as RequestReceived, ResponseStarted, ResponseComplete, and Panic. Logs are recorded based on the rules and audit levels such as None, Metadata, Request, and RequestResponse defined in the policy.

1. Request and RequestResponse levels provide complete insight into your API calls, but it will increase the memory consumption and, thereby, the audit logs storage.
2. Use Metadata to exclude sensitive information like authentication tokens.