This manual explains how to migrate to using external Cloud Controller Managers for supporting providers.
Cloud Controller Manager (CCM)
The CCM (Cloud Controller Manager) is a Kubernetes
control plane component that embeds cloud-specific control logic. There are two different kinds of Cloud controller managers:
in-tree and out-of-tree. According to the Kubernetes design proposal,
the in-tree code is “code that lives in the core Kubernetes repository k8s.io/kubernetes”,
while the out-of-tree code is “code that lives in an external repository outside of k8s.io/kubernetes”.
The first cloud-specific interaction logic was completely in-tree, bringing in a not-negligible amount of problems,
among which the dependency of the CCM release cycle from the Kubernetes core release cycle and the difficulty to add new providers
to the Kubernetes core code. Then, the Kubernetes community moved toward the out-of-tree implementation by introducing
a plugin mechanism that allows different cloud providers to integrate their platforms with Kubernetes.
Out-of-tree CCM Migration
Since the Kubernetes community has planned to deprecate and then remove all the code related to in-tree cloud
controller managers and the Kubernetes documentation explain how to migrate from in-tree to out-of-tree CCM, KKP itself
needed a mechanism to allow users to migrate their clusters to the out-of-tree implementation, as detailed below.
Support and Prerequisites
The CCM/CSI migration is supported for the following providers:
- vSphere: vSphere 7.0u1 is required for CCM/CSI migration
- Microsoft Azure
Enabling the External Cloud Provider
The migration is specific per user cluster, meaning that it is activated by the
externalCloudProvider feature in the
When this feature gets enabled in a cluster belonging to a supported cloud provider, a mutating webhook patches the cluster
by adding two different annotations, producing the following cluster:
The addition of the
externalCloudProvider feature triggers the following operations:
- Deployment in the user cluster of the components being part of both the Cloud Controller Manager and the CSI controller
- Patch of The Machine controller deployment to configure the external cloud provider for the new machines.
- Addition of a condition related to the ccm migration to the cluster status.
Finalize the CCM Migration
The last step to complete the CCM migration is the rolling restart of all the machineDeployments in the user cluster.
To do so via cli, simply follow the guide in the machine-controller documentation.
Performing the rolling update of all the machineDeployments implies the deletion of all the machines (hence all the nodes) and
their recreation. Since the MachineController has been patched to configure the external cloud provider for the new machines,
all the recreated machines will be configured to use the out-of-tree CCM. You can check this condition by verifying that
the new machines have the
Once all the machineDeployments are rolled out, and the new machines have the aforementioned annotation, the cluster
CSIKubeletMigrationCompleted will be set to true, and the migration is considered completed.
Disabling the External CCM
Since the Kubernetes community is on the way to deprecating in-tree CCM, once the
externalCloudProvider feature gets
enabled, it cannot be disabled.