# Manual Rotation of CA Certificates

LLMS index: [llms.txt](/llms.txt)

---

<!-- overview -->

This page shows how to manually rotate the certificate authority (CA) certificates.

## Before you begin

<p>You need to have a Kubernetes cluster, and the kubectl command-line tool must
be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a
cluster, you can create one by using
<a href="https://minikube.sigs.k8s.io/docs/tutorials/multi_node/">minikube</a>
or you can use one of these Kubernetes playgrounds:</p>
<ul>
<li><a href="https://labs.iximiuz.com/playgrounds?category=kubernetes&filter=all">iximiuz Labs</a></li>
<li><a href="https://killercoda.com/playgrounds/scenario/kubernetes">Killercoda</a></li>
<li><a href="https://kodekloud.com/public-playgrounds">KodeKloud</a></li>
</ul>



- For more information about authentication in Kubernetes, see
  [Authenticating](/docs/reference/access-authn-authz/authentication).
- For more information about best practices for CA certificates, see
  [Single root CA](/docs/setup/best-practices/certificates/#single-root-ca).

<!-- steps -->

## Rotate the CA certificates manually

<div class="alert alert-caution" role="note"><h4 class="alert-heading">Caution:</h4><p>Make sure to back up your certificate directory along with configuration files and any other necessary files.</p>
<p>This approach assumes operation of the Kubernetes control plane in an HA configuration with multiple API servers.
Graceful termination of the API server is also assumed, so clients can cleanly disconnect from one API server and
reconnect to another.</p>
<p>Configurations with a single API server will experience unavailability while the API server is being restarted.</p>
</div>


1. Distribute the new CA certificates and private keys (for example: `ca.crt`, `ca.key`, `front-proxy-ca.crt`,
   and `front-proxy-ca.key`) to all your control plane nodes in the Kubernetes certificates directory.

1. Update the `--root-ca-file` flag for the <a class='glossary-tooltip' title='Control Plane component that runs controller processes.' data-bs-toggle='tooltip' data-bs-placement='top' href='/docs/reference/command-line-tools-reference/kube-controller-manager/' target='_blank' aria-label='kube-controller-manager'>kube-controller-manager</a> to include
   both old and new CA, then restart the kube-controller-manager.

   Any <a class='glossary-tooltip' title='Provides an identity for processes that run in a Pod.' data-bs-toggle='tooltip' data-bs-placement='top' href='/docs/tasks/configure-pod-container/configure-service-account/' target='_blank' aria-label='ServiceAccount'>ServiceAccount</a> created after this point will get
   Secrets that include both old and new CAs.

   
<div class="alert alert-info" role="note"><h4 class="alert-heading">Note:</h4><p>The files specified by the kube-controller-manager flags <code>--client-ca-file</code> and <code>--cluster-signing-cert-file</code>
cannot be CA bundles. If these flags and <code>--root-ca-file</code> point to the same <code>ca.crt</code> file, which is now a
bundle (includes both old and new CA), you will face an error. To workaround this problem, you can copy the new CA
to a separate file and make the flags <code>--client-ca-file</code> and <code>--cluster-signing-cert-file</code> point to the copy.
Once <code>ca.crt</code> is no longer a bundle, you can restore the problem flags to point to <code>ca.crt</code> and delete the copy.</p>
<p><a href="https://github.com/kubernetes/kubeadm/issues/1350">Issue 1350</a> for kubeadm tracks an bug with the
kube-controller-manager being unable to accept a CA bundle.</p>
</div>


1. Wait for the controller manager to update `ca.crt` in the service account Secrets to include both old and new CA certificates.

    If any Pods are started before the new CA is used by API servers, the new Pods get this update and will trust both
    old and new CAs.

1. Restart all pods using in-cluster configurations (for example: kube-proxy, CoreDNS, etc) so they can use the
   updated certificate authority data from Secrets that link to ServiceAccounts.

   * Make sure CoreDNS, kube-proxy, and other Pods using in-cluster configurations are working as expected.

1. Append both old and new CA to the file against `--client-ca-file` and `--kubelet-certificate-authority`
   flag in the `kube-apiserver` configuration.

1. Append both old and new CA to the file against the `--client-ca-file` flag in the `kube-scheduler` configuration.

1. Update certificates for user accounts by replacing the content of `client-certificate-data` and `client-key-data`
   respectively.

   For information about creating certificates for individual user accounts, see
   [Configure certificates for user accounts](/docs/setup/best-practices/certificates/#configure-certificates-for-user-accounts).

   Additionally, update the `certificate-authority-data` section in the kubeconfig files,
   respectively with Base64-encoded old and new certificate authority data

1. Update the `--root-ca-file` flag for the <a class='glossary-tooltip' title='Control plane component that integrates Kubernetes with third-party cloud providers.' data-bs-toggle='tooltip' data-bs-placement='top' href='/docs/concepts/architecture/cloud-controller/' target='_blank' aria-label='Cloud Controller Manager'>Cloud Controller Manager</a> to include
   both old and new CA, then restart the cloud-controller-manager.

   
<div class="alert alert-info" role="note"><h4 class="alert-heading">Note:</h4>If your cluster does not have a cloud-controller-manager, you can skip this step.</div>


1. Follow the steps below in a rolling fashion.

   1. Restart any other
      [aggregated API servers](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) or
      webhook handlers to trust the new CA certificates.

   1. Restart the kubelet by updating the file against `clientCAFile` in the kubelet configuration and
      `certificate-authority-data` in `kubelet.conf` to use both the old and new CA on all nodes.

      If your kubelet is not using client certificate rotation, update `client-certificate-data` and
      `client-key-data` in `kubelet.conf` on all nodes, along with the kubelet client certificate file
      usually found in `/var/lib/kubelet/pki`.

   1. Restart API servers with the certificates (`apiserver.crt`, `apiserver-kubelet-client.crt`, and
      `front-proxy-client.crt`), signed by the new CA.
      You can use the existing private keys or new private keys.
      If you changed the private keys, then update these in the Kubernetes certificates directory as well.

      Since the Pods in your cluster trust both old and new CAs, there will be a momentary disconnection
      after which, pods' Kubernetes clients reconnect to the new API server.
      The new API server uses a certificate signed by the new CA.

      * Restart the <a class='glossary-tooltip' title='Control plane component that watches for newly created pods with no assigned node, and selects a node for them to run on.' data-bs-toggle='tooltip' data-bs-placement='top' href='/docs/reference/command-line-tools-reference/kube-scheduler/' target='_blank' aria-label='kube-scheduler'>kube-scheduler</a> to use and
        trust the new CAs.
      * Make sure control plane components logs no TLS errors.

      
<div class="alert alert-info" role="note"><h4 class="alert-heading">Note:</h4><pre><code>  To generate certificates and private keys for your cluster using the `openssl` command line tool,
  see [Certificates (`openssl`)](/docs/tasks/administer-cluster/certificates/#openssl).
  You can also use [`cfssl`](/docs/tasks/administer-cluster/certificates/#cfssl).
</code></pre>
</div>


    1. Annotate any DaemonSets and Deployments to trigger pod replacement in a safer rolling fashion.

      ```shell
      for namespace in $(kubectl get namespace -o jsonpath='{.items[*].metadata.name}'); do
          for name in $(kubectl get deployments -n $namespace -o jsonpath='{.items[*].metadata.name}'); do
              kubectl patch deployment -n ${namespace} ${name} -p '{"spec":{"template":{"metadata":{"annotations":{"ca-rotation": "1"}}}}}';
          done
          for name in $(kubectl get daemonset -n $namespace -o jsonpath='{.items[*].metadata.name}'); do
              kubectl patch daemonset -n ${namespace} ${name} -p '{"spec":{"template":{"metadata":{"annotations":{"ca-rotation": "1"}}}}}';
          done
      done
      ```

      
<div class="alert alert-info" role="note"><h4 class="alert-heading">Note:</h4><pre><code>  To limit the number of concurrent disruptions that your application experiences,
  see [configure pod disruption budget](/docs/tasks/run-application/configure-pdb/).
</code></pre>
</div>


        Depending on how you use StatefulSets, you may also need to perform a similar rolling replacement.

1. If your cluster is using bootstrap tokens to join nodes, update the ConfigMap `cluster-info` in the `kube-public`
   namespace with new CA.

   ```shell
   base64_encoded_ca="$(base64 -w0 /etc/kubernetes/pki/ca.crt)"

   kubectl get cm/cluster-info --namespace kube-public -o yaml | \
       /bin/sed "s/\(certificate-authority-data:\).*/\1 ${base64_encoded_ca}/" | \
       kubectl apply -f -
   ```

1. Verify the cluster functionality.

    1. Check the logs from control plane components, along with the kubelet and the kube-proxy.
       Ensure those components are not reporting any TLS errors; see
       [looking at the logs](/docs/tasks/debug/debug-cluster/#looking-at-logs) for more details.

    1. Validate logs from any aggregated api servers and pods using in-cluster config.

1. Once the cluster functionality is successfully verified:

   1. Update all service account tokens to include new CA certificate only.

      * All pods using an in-cluster kubeconfig will eventually need to be restarted to pick up the new Secret,
        so that no Pods are relying on the old cluster CA.

   1. Restart the control plane components by removing the old CA from the kubeconfig files and the files against it
      `--client-ca-file`, `--root-ca-file` flags resp.

   1. On each node, restart the kubelet by removing the old CA from the file against the `clientCAFile` flag
      and from the kubelet kubeconfig file. You should carry this out as a rolling update.

      If your cluster lets you make this change, you can also roll it out by replacing nodes rather than
      reconfiguring them.