The cluster reconciliation process determines the actual state of the cluster and takes actions based on the difference between actual and expected states. The reconciliation is capable of automatically installing, upgrading, and repairing the cluster. On top of that, reconciliation can change some cluster properties, such as apply addons and create new worker nodes.
More information about the technical implementation can be found in the KubeOne Reconciliation Process (kubeone apply) proposal.
Before getting started, make sure you have up-to-date Terraform output file if you’re using the Terraform integration:
terraform output -json > tf.json
The cluster reconciliation is implemented under the
kubeone apply command
which has similar semantics as other KubeOne commands.
You can use the following command:
kubeone apply --manifest config.yaml -t .
For more details about available flags and properties, run the command with
kubeone apply -h
By default, the
apply command expects the user confirmation before taking
any action. If you are running
kubeone apply in a script and/or without
terminal and ability to confirm actions, you can use the
kubeone apply --manifest config.yaml -t tf.json --auto-approve
The reconciliation process is explained in details in the technical proposal. This section includes important details about the reconciliation process that you should be aware of.
The actual state of the cluster is determined by running a set of probes (Bash scripts and Kubernetes API requests) on the cluster instances.
The expected state is defined with the KubeOne configuration manifest, and optionally with the Terraform state.
The general reconciliation workflow is based on:
The apply command has ability to detect is cluster in an unhealthy state. The cluster is considered unhealthy if there’s at least one node that’s unhealthy, which can happen if:
In such a case, the operator needs to manually delete a broken instance, update the KubeOne configuration file to remove the old instance and add a new one, and then run the apply command again.
If Terraform is used for managing the infrastructure, the
can be used to mark instance for recreation. Running
terraform apply the
next time would recreate the instance. Make sure to update the Terraform output
file by running
terraform apply again. For example:
terraform taint 'aws_instance.control_plane[<index-of-instance>]' terraform apply terraform output -json > tf.json
If there are multiple unhealthy instances, it might be required to replace and repair instance by instance in order to maintain the etcd quorum. KubeOne recommends which instances are safe to be deleted without affecting the quorum. It’s strongly advised to follow the order or otherwise you’re risking losing the quorum and all cluster data. If it’s not possible to repair the cluster without affecting the quorum, KubeOne will fail to repair the cluster. In that case, disaster recovery might be required.
The apply command doesn’t modify or delete existing MachineDeployments. The MachineDeployments are created by the apply command only if there’s another action to be taken, such as install or upgrade. Managing MachineDeployments should be done by the operator either by using kubectl or the Kubernetes API directly.
To make managing MachineDeployments easier, the operator can generate the
manifest containing all MachineDeployments defined in the KubeOne
configuration by using the
kubeone config machinedeployments command:
kubeone config machinedeployments --manifest config.yaml -t tf.json
The apply command doesn’t remove or unprovision the static worker nodes. That can be done by removing the appropriate instance manually. If there is CCM (cloud-controller-manager) running in the cluster, the Node for the removed worker should be deleted automatically. If there’s no CCM, you can remove the Node object manually using kubectl.
Currently, the apply command doesn’t reconcile features. If you enable/disable any feature on already provisioned cluster, you have to run the upgrade process for changes to be in the effect. The upgrade process can be run using the following upgrade command:
kubeone upgrade --manifest config.yaml -t . --force