Encrypting Secret Data at REST

Steps of Encrypting Secret Data at REST

Step 1: Check encryption at REST is already enabled on the cluster

# method 1
ps -aux | grep kube-apiserver | grep "encryption-provider-config"
# if it returns empty, that means is no encryption at REST enabled
 
# method 2
cat /etc/kubernetes/manifests/kube-apiserver.yaml
# check got option --encryption-provider-config

Step 2: Understand the encryption configuration

sample-encryption-config.yaml

---
#
# CAUTION: this is an example configuration.
#          Do not use this for your own cluster!
#
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: # You can pick which resources you want to encrypt
      - secrets
      - configmaps
      - pandas.awesome.bears.example # a custom resource API
 
    # Here got a lot of providers, this order matters, as the first provider which is identity will encrypt the data first, then it could use any of these (aesgcm, aescbc, secretbox, etc) to decrypt.
    # So since the identity it empty, meaning no encryption at all, so if you want to encrypt your data, then you should choose and place either one (aesgcm, aescbc, secretbox) at the first place.
    providers:
      # This configuration does not provide data confidentiality. The first
      # configured provider is specifying the "identity" mechanism, which
      # stores resources as plain text.
      #
      - identity: {} # plain text, in other words NO encryption
      - aesgcm:
          keys:
            - name: key1
              secret: c2VjcmV0IGlzIHNlY3VyZQ== # this secret will be used for the encryption by the encryption algorithm
            - name: key2
              secret: dGhpcyBpcyBwYXNzd29yZA==
      - aescbc:
          keys:
            - name: key1
              secret: c2VjcmV0IGlzIHNlY3VyZQ==
            - name: key2
              secret: dGhpcyBpcyBwYXNzd29yZA==
      - secretbox:
          keys:
            - name: key1
              secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
  - resources:
      - events
    providers:
      - identity: {} # do not encrypt Events even though *.* is specified below
  - resources:
      - '*.apps' # wildcard match requires Kubernetes 1.27 or later
    providers:
      - aescbc:
          keys:
          - name: key2
            secret: c2VjcmV0IGlzIHNlY3VyZSwgb3IgaXMgaXQ/Cg==
  - resources:
      - '*.*' # wildcard match requires Kubernetes 1.27 or later
    providers:
      - aescbc:
          keys:
          - name: key3
            secret: c2VjcmV0IGlzIHNlY3VyZSwgSSB0aGluaw==

Here got a lot of providers, this order matters, as the first provider which is identity will encrypt the data first, then it could use any of these (aesgcm, aescbc, secretbox, etc) to decrypt.
So since the identity it empty, meaning no encryption at all, so if you want to encrypt your data, then you should choose and place either one (aesgcm, aescbc, secretbox) at the first place.

Step 3: Generate a new encryption key

# Linux
head -c 32 /dev/urandom | base64
 
# Windows
# Do not run this in a session where you have set a random number
# generator seed.
[Convert]::ToBase64String((1..32|%{[byte](Get-Random -Max 256)}))

Step 4: Create a new encryption configuration file

Create a new file called enc.yaml and replace the secret value with the newly generated encryption key.

enc.yaml

---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
      - pandas.awesome.bears.example
    providers:
      - aescbc:
          keys:
            - name: key1
              # See the following text for more details about the secret value
              secret: <BASE 64 ENCODED SECRET> # Replace your newly generated encryption key here
      - identity: {} # this fallback allows reading unencrypted secrets;
                     # for example, during initial migration

Step 5: Apply this config file to `kube-apiserver` static pod

# save the new encryption configuration file to /etc/kubernetes/enc
mkdir /etc/kubernetes/enc
mv enc.yaml /etc/kubernetes/enc
 
# edit the kube-apiserver manifest file
vim /etc/kubernetes/manifests/kube-apiserver.yaml

/etc/kubernetes/manifests/kube-apiserver.yaml

---
#
# This is a fragment of a manifest for a static Pod.
# Check whether this is correct for your cluster and for your API server.
#
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.20.30.40:443
  creationTimestamp: null
  labels:
    app.kubernetes.io/component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml  # add this line #
    volumeMounts:
    ...
    - name: enc                           # add this line #
      mountPath: /etc/kubernetes/enc      # add this line #
      readOnly: true                      # add this line #
    ...
  volumes:
  ...
  - name: enc                             # add this line #
    hostPath:                             # add this line #
      path: /etc/kubernetes/enc           # add this line #
      type: DirectoryOrCreate             # add this line #
  ...

Step 6: Restart the `kube-apiserver`

If not mistaken, it will auto restart once you save the file, if not, you can restart it manually.

# method 1
docker ps | grep kube-apiserver
dokcer restart <kube-apiserver-container-id>
 
# method 2
sudo systemctl restart kube-apiserver
 
# method 3
kubectl delete pod -n kube-system -l component=kube-apiserver

Step 7: Test the encryption

7.1 Check whether the etcd cmd exists

etcdctl
# if not exists then install it
sudo apt-get install etcd-client

7.2 Create a new secret

kubectl create secret generic secret1 -n default --from-literal=mykey=mydata

7.3 Read the secret out of etcd

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key  \
  get /registry/secrets/default/<your-secret> | hexdump -C
 
# example
ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key  \
  get /registry/secrets/default/secret1  | hexdump -C

The output is similar to this (abbreviated):

00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 73 65 63 72 65 74  |s/default/secret|
00000020  31 0a 6b 38 73 3a 65 6e  63 3a 61 65 73 63 62 63  |1.k8s:enc:aescbc|
00000030  3a 76 31 3a 6b 65 79 31  3a c7 6c e7 d3 09 bc 06  |:v1:key1:.l.....|
00000040  25 51 91 e4 e0 6c e5 b1  4d 7a 8b 3d b9 c2 7c 6e  |%Q...l..Mz.=..|n|
00000050  b4 79 df 05 28 ae 0d 8e  5f 35 13 2c c0 18 99 3e  |.y..(..._5.,...>|
[...]
00000110  23 3a 0d fc 28 ca 48 2d  6b 2d 46 cc 72 0b 70 4c  |#:..(.H-k-F.r.pL|
00000120  a5 fc 35 43 12 4e 60 ef  bf 6f fe cf df 0b ad 1f  |..5C.N`..o......|
00000130  82 c4 88 53 02 da 3e 66  ff 0a                    |...S..>f..|
0000013a

Now your data is encrypted at REST. Congratulations!

Step 8 (Optional): Ensure all previous secrets are encrypted

# Run this as an administrator that can read and write all Secrets
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Encrypting Secret Data at REST

On this page