TLS

Transport Layer Security (TLS) Basics

The concept is very similar to public and private keys, where the public key in used to encrypt the data and the private key is used to decrypt the data.

Transport Layer Security (TLS) certificate also known as Secure Sockets Layer (SSL) certificate, mainly to secure internet connections by encrypting data sent between browser and server.

Generate a private key

# Generate a private key
openssl genrsa -out sample.key 1024
cat sample.key
 
# sample.key file content
-----BEGIN PRIVATE KEY-----
MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBALZgjDfEXndMv3eg
+cQqLoL+0jcqb2dWt63cvBH38rqD2XCxgpzsXT5KN0Q6kiT72tnMp2ij5gO1i7DG
w00t5767AX0kolI0QqvLgW3BMhTMhlZTUtEsNCl3/09P1z7Z6gJhgOfWHBTlkPiv
tJlmXGqgUnT5yFioemEd0WIMqYERAgMBAAECgYEAqhom7iTDjzYQJz4Hd4V4WEng
O//iZFone4w0BfZjeL9dYZTPJBn5Zg3GumZ+xOv7VcViJ2EnOjJsFfdtVWEJkfKt
xDYZPRDah88EiOMSScuAsxLC+gk7N9QbywLPPP4PphrFVMzoAoIKXNtJ/+CRKTTc
psxCYtEn61hsrKTNJ4ECQQDbsOi5OgdFFM3gvAaRSU7iDfjkYeXmZpfRhrvAqlYs
unjAdAUkT7NnCglVITdYmxhZvPa2Wby2mWSGixZuHls5AkEA1ITktryAZhjQmdS9
fPklE/88NRysMw9xXxdTIKJrKbgvbtzkXEIARayw3GK0UmKJtXwSs5mzLH30FnMP
tyDcmQJAECoGFYrVehm7xCtReGMicLOCeptRZRBrXzlAmz5tNMFLnS1sK23Jz20H
nsHg5p4SvpeOt4AGyJCkutmE/vkw0QJAHMhS4ZzuFuXNesghUce8idTLprXXbKLu
y94w1/6nXOMhKXapRwV5W/ZjjbWgjnfiBr5jGgf7CE+Wu2OiAT9mIQJAFe0cO7BV
8xYNqIty34/iSfRghJ96TtbozgHhXQ5CmbYzt+b5VK1MNx8g5mRr/CkUldBTei8L
eDn7hXBb1eqedg==
-----END PRIVATE KEY-----

Generate a public key based on private key

#------------------------------------------------------------#
# Generate a public key based on private key
openssl rsa -in sample.key -pubout > sample.pem
cat sample.pem
 
# sample.pem file content
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2YIw3xF53TL93oPnEKi6C/tI3
Km9nVret3LwR9/K6g9lwsYKc7F0+SjdEOpIk+9rZzKdoo+YDtYuwxsNNLee+uwF9
JKJSNEKry4FtwTIUzIZWU1LRLDQpd/9PT9c+2eoCYYDn1hwU5ZD4r7SZZlxqoFJ0
+chYqHphHdFiDKmBEQIDAQAB
-----END PUBLIC KEY-----

Now, the above commands will generate a self-signed certificate. This certificate is not trusted by browsers, so you need to get a certificate from a trusted Certificate Authority (CA). CA are companies that are trusted to sign, validate, and issue certificates. For example, GlobalSign, DigiCert, GoDaddy, etc.

In this case, we will generate a Certificate Signing Request (CSR) using the private key and with the domain name.

openssl req -new \
  -key sample.key \
  -out sample.csr \
  -subj "/C=US/ST=California/L=San Francisco/O=Example/OU=IT Department/CN=domain.com"

more details on this command can be found here
generate a certifate signing request using openssl here

So, this sample.csr file will be sent to the CA to verify, validate, and sign the certificate. The CA will then send back the signed certificate that the browser will trust. Remember, CAs themselves have a set of public and private keys to sign the certificate.

private key = sign the certificate
public key = verify and validate the certificate. These public keys from the CA are already installed in the browser.

Public key	Private Key
`.crt`, `.pem`	`.key`, `-key.pem`

Glossary:

CA: Certificate Authority
CSR: Certificate Signing Request
CRT: Certificate
PEM: Privacy Enhanced Mail, it can contain various types of data, including certificates, private keys, and other related information. PEM files are encoded in Base64 and typically include headers and footers to indicate the type of data they contain. The most common types of data found in PEM files are
- Certificates: These can be in the form of X.509 certificates, which are used in SSL/TLS for securing communications.
- Private Keys: These are the private keys associated with the certificates, used for decryption and signing.
- Certificate Chains: These include the certificate along with intermediate and root certificates to establish a chain of trust.
- Public Keys: These are the public keys associated with the private keys, used for encryption and verifying signatures.
KEY: Private key

TLS in Kubernetes

In Kubernetes, the communication between all the Kubernetes components must be secure and encrypted. This is where TLS comes into play. For example, no matter the admin is using kubectl or Kubernetes API directly, the communication must be secure.

So, we have server and client certificates. The server certificate is used secure communication between the Kubernetes components, for example, kube-apiserver, kubelet, etcd, where the client certificate is used to authenticate users and applications that are trying to interact with the Kubernetes cluster and to verify who they are.

Remember, when we create these certificates, we need to sign them with a CA. In fact, Kubernetes requires you to have at least one certificate authority to sign all the certificates, but you can have multiple CAs configured in your cluster.

Certificates

Server certificates: Deployed on servers.
Client certificates: Used by clients to authenticate themselves to the server.
Root certificates: Held by the Certificate Authority (CA) to sign server certificates.

Server certificates

We actually got 3 parties here:

kube-apiserver
- This is the main component in Kubernetes, so it is a server. Therefore it needs a server certificate to secure the communication betweeh the client and the server (authenticate their clients).
- For example, you can create kube-apiserver.crt and kube-apiserver.key files. Remember, you can put any file name you want.
etcd
- This is the database that will store all the Kubernetes cluster data. So it is a server.
- For example, you can create etcd.crt and etcd.key files.
kubelet
- This is the agent that runs on each node and they will expose an HTTPS API endpoint that the kube-apiserver will talk to in the Kubernetes cluster. So it is a server.
- For example, you can create kubelet.crt and kubelet.key files.

Client certificates

For the clients parties to access the services, we have:

The clients who access the kube-apiserver. The client needs a client certificate to authenticate themselves to the kube-apiserver.
- kube-scheduler
  - For example, kube-scheduler.crt and kube-scheduler.key.
- kube-controller-manager
  - For example, kube-controller-manager.crt and kube-controller-manager.key.
- kube-proxy
  - For example, kube-proxy.crt and kube-proxy.key.
- Admin who uses kubectl or Kubernetes API directly
  - For example, admin.crt and admin.key.
The clients who access the etcd or kubelet. The client needs a client certificate to authenticate themselves to the etcd or kubelet.
- kube-apiserver
  - You can use back the kube-apiserver.crt and kube-apiserver.key files, or you want to create a new one also can. The same goes for the kubelet.

Manually create certificates

Reference

You can use any tools like OpenSSL, Easy-RSA, etc to create the certificates.

Step 1: Generate CA certificate

# Generate a private key
openssl genrsa -out ca.key 2038
 
# Generate a csr using the private key
# CN = Common Name
openssl req -new -key ca.key -subj "/CN=Kubernetes-CA" -out ca.csr
 
# Sign the certificate
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt

With all the above commands, the CA has its own private key and root** certificate file**.

Step 2: Generate clients certificate

Let's say we want to create a client certificate for the admin user. You just need to repeat the same steps for other clients that need certificates to access the kube-apiserver.

# Generate a private key
openssl genrsa -out admin.key 2038
 
# Generate a csr using the private key
# CN = Common Name
openssl req -new -key admin.key -subj "/CN=Kubernetes-admin" -out admin.csr
 
# Sign the certificate using CA cert and key
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt

Remember to sign the certificate using the CA certificate and key, so the cluster will trust the certificate.
The admin.crt will be used by the admin to authenticate themselves to the server.

Now assuming you create a new user called john, how do you differentiate betweeh the admin and john users, as admin user is mainly do the administrators job while john user is a normal user. Well, in this case, you have to mention the group details in the processing of generating CSR.

openssl req -new -key admin.key \
  -subj "/CN=Kubernetes-admin/O=system:masters" -out admin.csr

The system:masters group name already exists in the Kubernetes cluster with admin privileges.
The system:masters is the group name that the admin user belongs to. So, the admin user will have the admin privileges.

kubelet

For kubelet client certificates, actually got a bit different from the other clients. The certificates naming and formats starts with the system:node:<node-name>. This is because kubelet is running on each node, and nodes are system components like kube-scheduler. With this format, the kube-apiserver knows which node is authenticating and give the right permissions based on this format.

You might have a question? How the kube-apiserver give the right permissions based on the format. Well, actually nodes must be added to a group called system:nodes in the Kubernetes cluster. This group is predefined in the Kubernetes cluster and has the right permissions to access the kubelet.

After the certificates are generated, remember to configure them in kubeconfig files.

Test the certificate

Actually, there are multiple ways to test the connections.

One way is to use the curl command to test the connection.

curl https://kube-apiserver:6443/api/v1/pods \
  --key admin.key --cert admin.crt --cacert ca.crt

The other way is to move all these configurations to the kube-config.yaml file and use the kubectl command to test the connection. This method is more common and easier to manage.

kube-config.yaml

apiVersion: v1
kind: Config
clusters:
  - cluster:
      certificate-authority: ca.crt
      server: https://kube-apiserver:6443
    name: kubernetes
users:
  - name: admin
    user:
      client-certificate: admin.crt
      client-key: admin.key

Now, remember, all the clients and servers need to have the CA's root certificate to trust the certificates. So, you need to distribute, copy, and specify the CA's root certificate to all the Kubernetes components, clients, and servers when you configure them.

Step 3: Generate servers certificate

The steps of creating a server certificate are similar to creating a client certificate. The only difference is that we want to ensure those servers like kube-apiserver, etcd, and kubelet are high available and secure by deploying them in multiple servers. So, to secure the communication between the servers, we will need to generate additional peer certificates.

each peer will have its own certificate and private key.

etcd

etcd.yaml

- etcd
  - --advertise-client-urls=https://10.89.0.2:2379
  - --cert-file=/etc/kubernetes/pki/etcd/server.crt # The path to the server's SSL certificate file
  - --client-cert-auth=true # Enables client certificate authentication for client connections
  - --data-dir=/var/lib/etcd
  - --experimental-initial-corrupt-check=true
  - --experimental-watch-progress-notify-interval=5s
  - --initial-advertise-peer-urls=https://10.89.0.2:2380
  - --initial-cluster=kind-cluster-control-plane=https://10.89.0.2:2380
  - --key-file=/etc/kubernetes/pki/etcd/server.key # The path to the server's SSL key file
  - --listen-client-urls=https://127.0.0.1:2379,https://10.89.0.2:2379
  - --listen-metrics-urls=http://127.0.0.1:2381
  - --listen-peer-urls=https://10.89.0.2:2380
  - --name=kind-cluster-control-plane
  - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt # The path to the peer's SSL certificate file
  - --peer-client-cert-auth=true # Enables client certificate authentication for peer connections
  - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key # The path to the peer's SSL key file
  - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt # The path to the trusted CA certificate file for peer connections
  - --snapshot-count=10000
  - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt # The path to the trusted CA certificate file for client connections

Remember to specify the CA root certificate as well.

kube-apiserver

We know that every operation in the Kubernetes cluster is done through the kube-apiserver. So this component may have many names and aliases within the cluster. Here is the list of names and aliases that the kube-apiserver may have:

kubernetes
kubernetes.default
kubernetes.default.svc
kubernetes.default.svc.cluster.local
ip address of the server like 10.89.0.2

When we have these names available, we have configure these names and aliases in the certificate, so that any clients or users can use these names and aliases to access the kube-apiserver.

In this case, we have to create a configuration file to specify the aliases and names.

openssl.cnf

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[ v3_req ]
basicContrainsts = CA:FALSE
keyUsage = nonRepudiation,
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
IP.1 = 10.89.0.2

# Pass the configuration file while generating a CSR.
openssl req -new -key kubeapiserver.key \
  -subj "/CN=kube-apiserver" -out kubeapiserver.csr \
  -config openssl.cnf
 
# Sign the certificate using CA cert and key
openssl x509 -req -in kubeapiserver.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out kubeapiserver.crt -extensions v3_req -extfile openssl.cnf -days 1000

kube-apiserver.yaml

- kube-apiserver
  - --advertise-address=10.89.0.2
  - --allow-privileged=true
  - --authorization-mode=Node,RBAC
  - --client-ca-file=/etc/kubernetes/pki/ca.crt
  - --enable-admission-plugins=NodeRestriction
  - --enable-bootstrap-token-auth=true
  - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
  - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
  - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
  - --etcd-servers=https://127.0.0.1:2379
  - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
  - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
  - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
  - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
  - --requestheader-allowed-names=front-proxy-client
  - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
  - --requestheader-extra-headers-prefix=X-Remote-Extra-
  - --requestheader-group-headers=X-Remote-Group
  - --requestheader-username-headers=X-Remote-User
  - --runtime-config=
  - --secure-port=6443
  - --service-account-issuer=https://kubernetes.default.svc.cluster.local
  - --service-account-key-file=/etc/kubernetes/pki/sa.pub
  - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
  - --service-cluster-ip-range=10.96.0.0/16
  - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
  - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

kubelet

Assuming you have multiple worker nodes, you will need to generate a certificate for each worker node, but how do you name the certificates? Well, actually they will be named after the node name. Once the certificates are generated, configure them kubelet YAML configuration file.

kubelet-config.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  x509:
    clientCAFile: "/etc/kubernetes/pki/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.32.0.10"
resolvConf: "/etc/resolv.conf"
runtimeRequestTimeout: "15m"
tlsCertFile: "/etc/kubernetes/kubelet/node-1.crt"
tlsPrivateKeyFile: "/etc/kubernetes/kubelet/node-1-key.key"

Certificate details

Certificate paths and details

Before we understand the certificate details, we need to understand how the Kubernetes cluster was setup. There are multiple ways to setup the Kubernetes cluster, each way has its own method to generate and manage the certificates. In this case we are looking at the common practice that people will do: Deploy Kubernetes components as pods.

Let's take a look at the kube-apiserver pod.

kube-apiserver.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver-kind-cluster-control-plane
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.89.0.2
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --runtime-config=
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/16
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.k8s.io/kube-apiserver:v1.30.0
    ...

Take a look at /etc/kubernetes/pki/apiserver.crt.

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout
 
# output
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1579479310736570071 (0x15eb70108e8836d7)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: Dec 28 06:55:09 2024 GMT
            Not After : Jan  5 06:16:55 2026 GMT
        Subject: CN = kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c4:ac:ac:9c:d8:2c:cb:22:95:cd:a5:16:05:62:
                    83:2c:3d:34:bb:a3:76:a9:9d:ca:68:a3:cc:89:3d:
                    90:a8:46:33:5c:2c:eb:5d:d4:66:55:1d:80:16:39:
                    f2:26:a0:23:74:fd:23:00:cc:49:2e:6f:70:82:c0:
                    ec:ca:74:8f:c0:9a:d5:c1:ef:f5:80:dd:7e:58:b6:
                    a9:66:82:5a:10:49:7e:90:f2:6a:e9:18:35:7a:c1:
                    6c:8a:f9:1d:b9:cc:5f:b2:63:18:7a:d6:3b:48:ec:
                    95:c4:e9:2f:43:89:cf:4b:5a:5b:08:c6:6d:10:89:
                    ea:c4:32:1f:48:85:17:ed:da:19:ae:f5:80:76:e6:
                    7d:93:bb:f2:7a:39:3b:45:a9:c9:c1:99:13:f7:94:
                    38:e4:97:9b:9e:c5:f2:4a:59:f6:78:0f:bd:47:18:
                    44:62:6f:15:aa:fe:7b:fc:00:11:ed:c8:f0:24:4b:
                    16:8f:1b:34:05:34:2a:6c:60:c3:b7:3e:d7:0b:1b:
                    c3:f5:f3:fe:63:5e:d5:46:47:59:7a:3c:19:de:7b:
                    57:43:e5:62:6e:a3:fc:98:d5:88:e0:8a:4c:b7:81:
                    04:5f:b8:3c:9e:ad:d5:f4:80:d9:c1:80:39:18:ff:
                    60:1e:87:0b:08:0d:be:89:ac:e7:cd:df:f2:f4:3e:
                    ac:ab
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
                4A:FB:80:18:F0:10:D6:2B:18:D9:35:6C:28:35:4C:8E:16:C7:84:C2
            X509v3 Subject Alternative Name:
                DNS:kind-cluster-control-plane, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, IP Address:10.96.0.1, IP Address:10.89.0.2, IP Address:127.0.0.1
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        55:45:84:5c:24:44:d0:e0:be:30:3b:91:01:84:1e:b8:ec:11:
        b6:cb:d9:d1:9d:ea:70:0c:45:0c:ff:b9:ce:6b:20:68:40:d7:
        de:86:b5:2d:0b:1e:8c:ec:79:14:c1:fc:ef:99:41:bf:a9:36:
        34:42:61:d8:9e:2a:4e:82:c2:1c:74:bd:0f:ff:6d:d2:f3:aa:
        4f:ab:2d:21:b3:e2:3c:47:f5:04:0a:b5:a4:4b:fb:53:b5:1c:
        fb:bb:e2:ff:dc:82:1c:e6:14:b7:8c:ab:fc:4b:30:b3:28:38:
        13:9b:d5:79:0d:b6:57:94:4f:a8:92:86:3c:0e:0f:78:dc:70:
        9a:dc:22:dd:10:c4:9d:56:eb:d8:13:ca:69:81:52:af:ff:9d:
        4d:65:14:39:42:2e:7f:4a:65:d1:0e:88:4d:eb:0c:e9:18:df:
        15:18:06:89:cb:49:0c:ee:59:9c:11:51:c4:f7:f6:b7:6e:f9:
        73:3a:ab:e4:39:ed:ed:6c:31:1c:cb:e6:12:bf:ed:4d:ae:7c:
        b6:75:1b:50:ef:e0:79:cd:d2:6f:48:cf:8d:33:6a:d0:72:1a:
        81:2d:c3:ac:71:40:71:9c:2c:22:a2:66:f7:33:82:37:48:6e:
        39:ce:4b:93:ef:7f:85:d6:df:e7:42:41:18:64:9b:f6:1a:6b:
        64:db:eb:d1

Issuer: CN = kubernetes = Issuer of the certificate
Not After : Jan 5 06:16:55 2026 GMT = Expiry date of the certificate
Subject: CN = kube-apiserver = Common name of the certificate
X509v3 Subject Alternative Name = Alias names of the certificate

If you want to debug the certificate, then you can use kubectl logs command to see the logs of the pod, if the kubectl unable to work properly, then you can use the docker logs command to see the logs of the container and troubleshoot.

On this page