Penguin-KarChunTKarChunT

Audit Logging

Understand how to enable and use audit logging in Kubernetes.

Concept of Audit Logging

Reference

https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/

Before I explain what is audit logging? It is important to answer the following questions:

  • How do we audit and monitor what is happening in our Kubernetes cluster?
    • For example, if a user creates a new pod, how do we know
      • who created it?
      • when it was created?
      • which namespace it was created in?
      • where was the request made from?
      • what was the response?
      • what happened?

Audit logging is a way to record the actions taken by users and system components in a Kubernetes cluster. It is important to have audit logging enabled to track changes made to the cluster and to identify potential security issues.

Important

Audit logging is handled by the kube-apiserver component. It is important to note that audit logging is not enabled by default in Kubernetes. You need to enable it explicitly.

Each request on each stage will generate an event. The event will contain information about the request. The event will be stored in a file.

Four Stages of Audit Logging

Each request can be recorded with an associated stage and there are four stages in the audit logging process.

  1. RequestReceived - This stage occurs when the API server receives a request. At this point, the server has not yet processed the request. It is useful for tracking when a request was initiated and by whom. For example, if a user creates a new pod, this stage logs the initial receipt of that request.
  2. ResponseStarted - This stage is logged when the API server starts sending a response (only send response headers without response body) back to the client. It indicates that the server has begun processing the request and has enough information to start generating a response. For example, this could occur after validating the request and determining that it can proceed.
  3. ResponseComplete - This stage is logged when the API server has fully processed the request and completed sending the response (response body) to the client. It includes details about the outcome of the request, such as whether it succeeded or failed, and any associated metadata.
  4. Panic - This stage is logged if the API server encounters an unexpected error or crash while processing request or got invalid request, the request goes through a panic stage. It is used to capture critical failures that may require further investigation or debugging.

Usage of Audit Logging

You can use Log or Webhook backend (eg, Falco) to enable audit logging. In this example, we will use Log backend.

Step 1: Create an Audit Policy File

Important

Remember, we only want to record the important events (eg, delete pods in specific namespace). We do not want to record all events, as if we record all events for every stage within the cluster, it will generate a lot of data.

There are four audit levels in Kubernetes:

LevelDescription
NoneDon't log events that match this rule
MetadataLog events with metadata (requesting user, timestamp, resources, verb, etc.) but not request or response body
RequestLog events with request metadata and body but not response body. This does not apply for non-resource requests
RequestResponseLog events with request metadata, request body and response body. This does not apply for non-resource requests
/etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages: # optional
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
 
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]
 
  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]
 
  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]
 
  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
      - "/api*" # Wildcard matching.
      - "/version"
 
  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
      - group: "" # core API group
        resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]
 
  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
      - group: "" # core API group
        resources: ["secrets", "configmaps"]
 
  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
      - group: "" # core API group
      - group: "extensions" # Version of group should NOT be included.
 
  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"
 
  # Log all requests at the Metadata level.
  - level: Metadata
  • omitStages is optional, but when it is defined, the events of this stage will not be recorded.
  • level is mandatory
  • verbs is optional, when it is not defined, it will be applied to all verbs, like get, list, watch, create, update, patch, delete, etc.
  • pods/log - This subresource is used to access the logs of a specific pod. kubectl logs <pod-name>
  • pods/status - This subresource is used to retrieve the status of a specific pod. kubectl get pod <pod-name> -o jsonpath='{.status}'

Step 2: Enable Audit Logging in kube-apiserver

Reference

https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/#log-backend

You can enable audit logging in kube-apiserver by passing the --audit-policy-file and --audit-log-path flags to the kube-apiserver command and then you need to mount with volumes and volumeMounts.

  • --audit-policy-file flag specifies the path to the audit policy file

You can also add additional fields like

  • --audit-log-path - specifies the log file path that log backend uses to write audit events.
  • --audit-log-maxage - defines the maximum number of days to retain old audit log files.
  • --audit-log-maxbackup - defines the maximum number of audit log files to retain.
  • --audit-log-maxsize - defines the maximum size in megabytes of the audit log file before it gets rotated.
kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver-kind-cluster-control-plane
  namespace: kube-system
spec:
  containers:
    - command:
      - kube-apiserver
      - --advertise-address=10.89.0.2
      - --allow-privileged=true
      - --authorization-mode=Node,RBAC
      - --client-ca-file=/etc/kubernetes/pki/ca.crt
      - --enable-admission-plugins=NodeRestriction,PodSecurity
      - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
      - --audit-log-path=/var/log/kubernetes/audit/audit.log
      - --audit-log-maxage=10
      - --audit-log-maxbackup=5
      - --audit-log-maxsize=100
      ...
      image: registry.k8s.io/kube-apiserver:v1.30.0
      volumeMounts: 
        - mountPath: /etc/kubernetes/audit-policy.yaml
          name: audit
          readOnly: true
        - mountPath: /var/log/kubernetes/audit/
          name: audit-log
          readOnly: false
  volumes: 
    - name: audit
      hostPath: 
        path: /etc/kubernetes/audit-policy.yaml
        type: File
    - name: audit-log
      hostPath: 
        path: /var/log/kubernetes/audit/
        type: DirectoryOrCreate
Edit on GitHub

Monitoring

Understand how to setup Monitoring in Kubernetes.

OS Upgrades

Understand how to perform OS Upgrades in Kubernetes.

On this page