On the KubeVirt provider, KubeOne deploys the KubeVirt CSI driver with the csi-kubevirt
addon. The driver provisions the volumes of the cluster as DataVolumes in the infra cluster, and hot-plugs them into
the VirtualMachines that run the nodes.
As of v1.15, the addon deploys the driver through kubevirt-csi-driver-operator. You
configure the StorageClasses and VolumeSnapshotClasses of the cluster in the KubeOneCluster manifest, and KubeOne keeps
them in sync with it on every kubeone apply.
KubeOne supports the following operations:
kubeone apply, including changes to
fields that are immutable on a StorageClassThe first kubeone apply with KubeOne v1.15 on an existing KubeVirt cluster replaces the CSI node plugin. See
Upgrading Existing Clusters before you upgrade.
cloudProvider.kubevirt), and cloudProvider.disableBundledCSIDrivers is
not set.KUBEVIRT_KUBECONFIG credential) needs create, get
and delete permissions on volumesnapshots.snapshot.storage.k8s.io in the infraNamespace. This is only needed if
you configure volumeSnapshotClasses.Add storageClasses and volumeSnapshotClasses to cloudProvider.kubevirt in the KubeOneCluster manifest:
apiVersion: kubeone.k8c.io/v1beta3
kind: KubeOneCluster
versions:
kubernetes: "1.37.2"
cloudProvider:
kubevirt:
infraNamespace: tenant-a
storageClasses:
- infraStorageClassName: standard
isDefaultClass: true
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
bus: scsi
- infraStorageClassName: fast-ssd
allowVolumeExpansion: true
zones:
- zone-a
labels:
tier: fast
volumeSnapshotClasses:
- infraVolumeSnapshotClass: csi-snapclass
isDefaultClass: true
deletionPolicy: Delete
The operator names each class after the infra class it’s backed by, with a kubevirt- prefix. The example above
creates the StorageClasses kubevirt-standard and kubevirt-fast-ssd, and the VolumeSnapshotClass
kubevirt-csi-snapclass.
Both lists are optional. If neither is set, KubeOne deploys the driver but no StorageClass or VolumeSnapshotClass,
and you have to create your own csi.kubevirt.io StorageClasses.
| Field | Description | Default |
|---|---|---|
infraStorageClassName | Name of the StorageClass in the infra cluster. Required. | |
isDefaultClass | Marks the StorageClass as the default StorageClass of the cluster. | false |
volumeBindingMode | Immediate or WaitForFirstConsumer. | Immediate |
allowVolumeExpansion | Allows PersistentVolumeClaims of the StorageClass to be expanded. | false |
reclaimPolicy | Delete or Retain. | Delete |
bus | Disk bus used to attach the volumes to the VirtualMachines: virtio, scsi or sata. | |
zones | Restricts the provisioned volumes to the given zones (topology.kubernetes.io/zone). | |
regions | Restricts the provisioned volumes to the given regions (topology.kubernetes.io/region). | |
labels | Labels to set on the StorageClass. |
| Field | Description | Default |
|---|---|---|
infraVolumeSnapshotClass | Name of the VolumeSnapshotClass in the infra cluster. Required. | |
isDefaultClass | Marks the VolumeSnapshotClass as the default VolumeSnapshotClass of the cluster. | false |
deletionPolicy | Delete or Retain. | Delete |
See the API reference for the full specification.
KubeOne validates the configuration before it changes anything in the cluster. The validation fails if:
infraStorageClassName or infraVolumeSnapshotClass is emptykubevirt- prefix added, isn’t a valid DNS subdomain nameisDefaultClass: truebus, volumeBindingMode, reclaimPolicy or deletionPolicy has an unsupported valuestorageClasses or volumeSnapshotClasses is set together with cloudProvider.disableBundledCSIDrivers: true.
Without the bundled CSI driver, nothing would create the classesRun kubeone apply after you change storageClasses or volumeSnapshotClasses. KubeOne waits up to 5 minutes for
the StorageClasses, and then up to 5 more minutes for the VolumeSnapshotClasses, to match the manifest.
The operator can’t change most fields of an existing StorageClass, because they’re immutable. KubeOne handles this for you:
bus, volumeBindingMode, reclaimPolicy, allowVolumeExpansion, zones, regions or
labels changed is deleted, and the operator recreates it with the new values.Recreating a StorageClass doesn’t affect existing PersistentVolumes, because Kubernetes only reads a StorageClass when
a volume is provisioned or expanded. A new PersistentVolumeClaim of a StorageClass that is being recreated stays
Pending until the StorageClass exists again.
KubeOne only manages classes created by the operator. StorageClasses and VolumeSnapshotClasses that you created
yourself aren’t touched, unless one of them is named kubevirt-<name> for a configured entry. The operator adopts such
a class, and KubeOne may delete and recreate it from the manifest.
Previous KubeOne versions deployed the CSI node plugin as the kube-system/kubevirt-csi-node DaemonSet. The operator
runs its own node plugin, the kubevirt-csi-driver/kubevirt-csi-node DaemonSet. Both use the same host paths, so they
can’t run on a node at the same time.
The first kubeone apply with KubeOne v1.15 on an existing KubeVirt cluster:
kube-system/kubevirt-csi-node DaemonSet and waits until its pods are gone. Then it deletes the
kube-system/kubevirt-csi-node-sa ServiceAccount, the kubevirt-csi-node-cr ClusterRole, the
kubevirt-csi-node-binding ClusterRoleBinding and the kube-system/driver-config ConfigMap.csi-kubevirt addon: the operator, the updated CSI controller and the Tenant object.Between steps 1 and 2, nodes run no CSI node plugin. Volumes that are already mounted keep working, but new attach, mount and unmount operations wait until the new node plugin pod is ready on the node. The node plugin images don’t change, so the new pods usually start without pulling images.
Existing PersistentVolumes keep working, because the driver name csi.kubevirt.io doesn’t change. StorageClasses that
you created yourself keep working as well.
The csi-kubevirt addon deploys:
tenants.csiprovisioner.kubevirt.io CustomResourceDefinition.kubevirt-csi-driver/kubevirt-csi-driver-operator Deployment. If
registryConfiguration.overwriteRegistry is set, the operator pulls the node plugin images from that registry.kube-system/kubevirt-csi-controller Deployment. It talks to the infra cluster with the
credentials from the kube-system/kubeone-ccm-credentials Secret, and includes the csi-snapshotter and
csi-resizer sidecars for snapshots and volume expansion.Tenant object named tenant, rendered from cloudProvider.kubevirt.storageClasses and
cloudProvider.kubevirt.volumeSnapshotClasses.From the Tenant, the operator creates the csi.kubevirt.io CSIDriver, the node plugin DaemonSet in the
kubevirt-csi-driver namespace, and the StorageClasses and VolumeSnapshotClasses.
On KubeVirt, KubeOne applies the csi-kubevirt addon after the csi-external-snapshotter addon, which provides the
snapshot CRDs that the VolumeSnapshotClasses need.
If kubeone apply fails while waiting for the StorageClasses or VolumeSnapshotClasses, check the operator logs:
kubectl logs -n kubevirt-csi-driver deployment/kubevirt-csi-driver-operator
If the operator pod isn’t running, check that its image can be pulled. When you use a mirror registry, run
kubeone config images to list all images that the registry must have.