KubeOne takes care of upgrading kubeadm and kubelet binaries, running
kubeadm upgrade on all control plane nodes, upgrading components and addons
deploy by KubeOne, and optionally upgrading all MachineDeployments objects to
the desired Kubernetes version. Upgrades are done in-place, i.e. KubeOne
connects to nodes over SSH and runs commands needed to upgrade the node.
Worker nodes managed by Kubermatic machine-controller are upgraded using the rolling-upgrade strategy, i.e. the old nodes are replaced with the new ones. KubeOne Static Workers are upgraded in-place, similar to the control plane nodes.
KubeOne is doing a set of preflight checks to ensure all prerequisites are satisfied. The following checks are done by KubeOne:
Once the upgrade process starts for a node, KubeOne applies the
kubeone.io/upgrade-in-progress label on the Node object. This label is used
as a lock mechanism, so if upgrade fails or it’s already in progress, you can’t
start it again.
It’s recommended to backup your cluster before running the upgrade process, which can be done using the Backups Addons.
Before running upgrade, please ensure that your KubeOne version supports
upgrading to the desired Kubernetes version. Check the
Compatibility page for more details on supported Kubernetes
versions for each KubeOne release. You can check what KubeOne version you’re
running using the kubeone version command.
You need to update the KubeOne configuration manifest to use the desired
Kubernetes version by changing the versions.Kubernetes field. It is possible
to upgrade only to the next minor release, or to any patch release as long as
the minor version is same or the next one.
After modifying the configuration manifest, you can use the apply command to
run upgrade. The kubeone.yaml file is the configuration manifest
and the tf.json file is the Terraform state file (can be omitted if
the Terraform Integration is not used).
kubeone apply --manifest kubeone.yaml -t tf.json --upgrade-machine-deployments
By default KubeOne does not update the MachineDeployment objects. If you
want to update them run the apply command with the
--upgrade-machine-deployments flag. This updates all MachineDeployment in the
cluster regardless of what’s specified in the KubeOne configuration manifest
or Terraform state file.
If you encounter any issue with the apply command or you want to force the
upgrade process, you can run the upgrade command manually:
kubeone upgrade --manifest kubeone.yaml -t tf.json.
It’s recommended to use the apply command whenever it’s possible.
The apply command analyzes the given instances, verifies that there is
Kubernetes running on those instances, runs the preflight checks, and offers
you to upgrade the cluster if needed. You’ll be asked to confirm your intention
to upgrade the cluster by typing yes.
INFO[13:59:27 CEST] Determine hostname…
INFO[13:59:31 CEST] Determine operating system…
INFO[13:59:32 CEST] Running host probes…
INFO[13:59:33 CEST] Electing cluster leader…
INFO[13:59:33 CEST] Elected leader "ip-172-31-220-51.eu-west-3.compute.internal"…
INFO[13:59:36 CEST] Building Kubernetes clientset…
INFO[13:59:36 CEST] Running cluster probes…
The following actions will be taken:
Run with --verbose flag for more information.
~ upgrade control plane node "ip-172-31-220-51.eu-west-3.compute.internal" (172.31.220.51): 1.18.5 -> 1.18.6
~ upgrade control plane node "ip-172-31-221-177.eu-west-3.compute.internal" (172.31.221.177): 1.18.5 -> 1.18.6
~ upgrade control plane node "ip-172-31-222-48.eu-west-3.compute.internal" (172.31.222.48): 1.18.5 -> 1.18.6
~ ensure nodelocaldns
~ ensure CNI
~ ensure credential
~ ensure machine-controller
~ upgrade MachineDeployments
Do you want to proceed (yes/no):
After confirming your intention to upgrade the cluster, the process will start. It usually takes 5-10 minutes for cluster to be upgraded. At the end, you should see output such as the following one:
INFO[13:59:55 CEST] Determine hostname…
INFO[13:59:55 CEST] Determine operating system…
INFO[13:59:55 CEST] Generating kubeadm config file…
INFO[13:59:56 CEST] Uploading config files… node=172.31.222.48
INFO[13:59:56 CEST] Uploading config files… node=172.31.220.51
INFO[13:59:56 CEST] Uploading config files… node=172.31.221.177
INFO[13:59:57 CEST] Building Kubernetes clientset…
INFO[13:59:58 CEST] Running preflight checks…
INFO[13:59:58 CEST] Verifying that Docker, Kubelet and Kubeadm are installed…
INFO[13:59:58 CEST] Verifying that nodes in the cluster match nodes defined in the manifest…
INFO[13:59:58 CEST] Verifying that all nodes in the cluster are ready…
INFO[13:59:58 CEST] Verifying that there is no upgrade in the progress…
INFO[13:59:58 CEST] Verifying is it possible to upgrade to the desired version…
INFO[13:59:58 CEST] Labeling leader control plane… node=172.31.220.51
INFO[13:59:58 CEST] Draining leader control plane… node=172.31.220.51
INFO[14:00:07 CEST] Upgrading kubeadm binary on the leader control plane… node=172.31.220.51
INFO[14:00:21 CEST] Running 'kubeadm upgrade' on leader control plane node… node=172.31.220.51
INFO[14:00:44 CEST] Upgrading kubernetes system binaries on the leader control plane… node=172.31.220.51
INFO[14:00:59 CEST] Uncordoning leader control plane… node=172.31.220.51
INFO[14:01:00 CEST] Waiting 30s to ensure all components are up… node=172.31.220.51
INFO[14:01:30 CEST] Unlabeling leader control plane… node=172.31.220.51
INFO[14:01:30 CEST] Labeling follower control plane… node=172.31.221.177
INFO[14:01:30 CEST] Draining follower control plane… node=172.31.221.177
INFO[14:01:30 CEST] Upgrading Kubernetes binaries on follower control plane… node=172.31.221.177
INFO[14:01:44 CEST] Running 'kubeadm upgrade' on the follower control plane node… node=172.31.221.177
INFO[14:01:55 CEST] Upgrading kubernetes system binaries on the follower control plane… node=172.31.221.177
INFO[14:02:14 CEST] Uncordoning follower control plane… node=172.31.221.177
INFO[14:02:14 CEST] Waiting 30s to ensure all components are up… node=172.31.221.177
INFO[14:02:44 CEST] Unlabeling follower control plane… node=172.31.221.177
INFO[14:02:44 CEST] Labeling follower control plane… node=172.31.222.48
INFO[14:02:44 CEST] Draining follower control plane… node=172.31.222.48
INFO[14:02:53 CEST] Upgrading Kubernetes binaries on follower control plane… node=172.31.222.48
INFO[14:03:10 CEST] Running 'kubeadm upgrade' on the follower control plane node… node=172.31.222.48
INFO[14:03:24 CEST] Upgrading kubernetes system binaries on the follower control plane… node=172.31.222.48
INFO[14:03:48 CEST] Uncordoning follower control plane… node=172.31.222.48
INFO[14:03:48 CEST] Waiting 30s to ensure all components are up… node=172.31.222.48
INFO[14:04:18 CEST] Unlabeling follower control plane… node=172.31.222.48
INFO[14:04:18 CEST] Downloading PKI…
INFO[14:04:19 CEST] Downloading PKI files… node=172.31.220.51
INFO[14:04:20 CEST] Creating local backup… node=172.31.220.51
INFO[14:04:20 CEST] Ensure node local DNS cache…
INFO[14:04:21 CEST] Activating additional features…
INFO[14:04:22 CEST] Applying canal CNI plugin…
INFO[14:04:34 CEST] Creating credentials secret…
INFO[14:04:34 CEST] Installing machine-controller…
INFO[14:04:37 CEST] Installing machine-controller webhooks…
INFO[14:04:37 CEST] Waiting for machine-controller to come up…
INFO[14:05:03 CEST] Upgrade MachineDeployments…
If the upgrade process fails, it’s recommended to continue manually and resolve
errors. In this case the kubeone.io/upgrade-in-progress label will prevent
you from running KubeOne again but you can ignore it using the --force flag.
kubeone upgradeIn case you want to change some of the cluster properties (e.g. enable a new
feature), you can use the upgrade command to reconcile the changes.
Modify your manifest to include the desired changes, but don’t change the
Kubernetes version (unless you want to upgrade the cluster), and then run the
upgrade command with the --force flag:
kubeone upgrade --manifest kubeone.yaml -t tf.json --force
Alternatively, the kubeone apply command can be used as well:
kubeone apply --manifest kubeone.yaml -t tf.json --force-upgrade
The --force flag instructs KubeOne to ignore the preflight errors, including
the error saying that you’re trying to upgrade to the already running version.
At the upgrade time, KubeOne ensures that the actual cluster configuration
matches the expected configuration, and therefore the upgrade
command can be used to modify cluster properties.