Helm applies a chart’s crds/ directory only on helm install. It skips that directory on every helm upgrade, and it never deletes CRDs. This is documented Helm behavior, not a KubeLB limitation, and there is no flag that changes it.
The consequence that catches people out: this rule applies to addon subcharts as well. The KubeLB manager chart bundles Gateway API, Envoy Gateway and External DNS CRDs inside its addon subcharts. When you run helm upgrade, the addon workloads move to their new versions while their CRDs stay at whatever version was current when the cluster was first installed.
So the CRD apply is a mandatory step of every upgrade, not a one-time install step.
helm upgrade reports success whether or not the CRDs are current. Nothing in the Helm output tells you that the CRDs were skipped.
Upgrade in this order:
Upgrade the manager before the CCMs. The manager owns the shared API surface, and a newer CCM talking to an older manager can send resources the manager does not yet understand. A newer manager with older CCMs is the supported direction and is what you pass through during a rollout.
Do not leave a fleet running mixed versions longer than the rollout needs. Manager and CCM are released and tested as a matched pair; skew is a transitional state, not a supported configuration.
Pull the chart and apply every CRD it ships, then upgrade the release:
helm pull oci://quay.io/kubermatic/helm-charts/kubelb-manager-ee --version=<version> --untardir "." --untar
kubectl apply --server-side --force-conflicts -R \
-f kubelb-manager-ee/crds/ \
-f kubelb-manager-ee/charts/kubelb-addons/charts/external-dns/crds/ \
-f kubelb-manager-ee/charts/kubelb-addons/charts/gateway-helm/charts/crds/crds/
helm upgrade --install kubelb-manager kubelb-manager-ee --namespace kubelb -f values.yaml
helm pull oci://quay.io/kubermatic/helm-charts/kubelb-manager --version=<version> --untardir "." --untar
kubectl apply --server-side --force-conflicts -R \
-f kubelb-manager/crds/ \
-f kubelb-manager/charts/kubelb-addons/charts/external-dns/crds/ \
-f kubelb-manager/charts/kubelb-addons/charts/gateway-helm/charts/crds/crds/
helm upgrade --install kubelb-manager kubelb-manager --namespace kubelb -f values.yaml
Three directories, because the chart keeps CRDs in three places:
| Directory | Contents |
|---|---|
crds/ | KubeLB’s own CRDs |
charts/kubelb-addons/charts/external-dns/crds/ | The External DNS DNSEndpoint CRD |
charts/kubelb-addons/charts/gateway-helm/charts/crds/crds/ | Gateway API and Envoy Gateway CRDs |
-R is required: the Gateway API directory has a generated/ subdirectory holding the Envoy Gateway CRDs, and kubectl apply -f <dir> does not descend into subdirectories on its own.
Apply all three even for addons you have not enabled. The chart ships the CRDs regardless of the enabled flags, and applying a CRD for an addon you do not run is inert.
The chart also vendors subcharts that are themselves named crds, under MetalLB and under gateway-helm. Those keep their manifests in templates/, so Helm upgrades them normally and they must stay out of the command above. Some of them are Helm templates rather than plain manifests, which kubectl apply cannot parse. A blanket find . -type d -name crds picks them up and fails; use the three explicit paths.
The CCM chart bundles no addon subcharts, so its own crds/ directory is the complete set:
helm pull oci://quay.io/kubermatic/helm-charts/kubelb-ccm-ee --version=<version> --untardir "." --untar
kubectl apply --server-side --force-conflicts -f kubelb-ccm-ee/crds/
helm upgrade --install kubelb-ccm kubelb-ccm-ee --namespace kubelb -f values.yaml
helm pull oci://quay.io/kubermatic/helm-charts/kubelb-ccm --version=<version> --untardir "." --untar
kubectl apply --server-side --force-conflicts -f kubelb-ccm/crds/
helm upgrade --install kubelb-ccm kubelb-ccm --namespace kubelb -f values.yaml
KubeLB v1.5.0 moves the bundled Envoy Gateway from 1.7.2 to 1.8.3. Envoy Gateway 1.8.3 requires the ListenerSet CRD (listenersets.gateway.networking.k8s.io), which arrived in Gateway API v1.5. The chart ships it, but as described above helm upgrade will not apply it.
Run the management cluster CRD apply above before helm upgrade. Fresh installations are unaffected, because helm install applies the CRDs itself.
The failure is delayed, which makes it hard to connect back to the upgrade:
helm upgrade reports success.envoy-gateway pod keeps running and traffic keeps flowing.The envoy-gateway pod crashloops with:
failed to create gatewayapi controller: error watching resources:
no matches for kind "ListenerSet" in version "gateway.networking.k8s.io/v1"
Check for it with:
kubectl get crd listenersets.gateway.networking.k8s.io
Recovery is the CRD apply itself. Run the management cluster command above; the pod recovers on its own once the CRD exists. No rollback or reinstall is needed.
v1.5.0 installs a safe-upgrades.gateway.networking.k8s.io ValidatingAdmissionPolicy that rejects any Gateway API CRD older than v1.5.0. If you downgrade to a v1.4.x bundle while it is in place, the apply is denied with:
Installing CRDs with version before v1.5.0 is prohibited by default.
Delete the binding and the policy first:
kubectl delete validatingadmissionpolicybinding safe-upgrades.gateway.networking.k8s.io
kubectl delete validatingadmissionpolicy safe-upgrades.gateway.networking.k8s.io
The bundled Gateway API CRDs are channel=experimental, bundle-version=v1.5.1. Clusters running KubeLB v1.4.x are on channel=experimental, bundle-version=v1.4.1, so this is an in-channel upgrade: no experimental fields are stripped and the safe-upgrades policy rules pass.
Do not apply the upstream standard-install.yaml bundle to an existing KubeLB cluster. KubeLB ships the experimental channel, and applying the standard channel over it is a channel downgrade that strips experimental fields from your Gateway API resources.