Release Notes

Kubermatic 2.29

v2.29.9

GitHub release: v2.29.9

Supported Kubernetes Versions

  • Add support for k8s patch releases 1.34.9/1.33.13 (#15994)

Bugfixes

  • Fix a kkp-master-operator issue where the /nvidia-gpu-operator ApplicationDefinition would fail with a context deadline exceeded error during KKP upgrades (#15960)
  • Fix seed Prometheus scraping envoy-agent directly via worker private IPs for tunneling user clusters (#16024)
  • KKP now configures Cilium to exclude the reserved (KKP) NodeLocalDNS address from local address detection when NodeLocalDNS is enabled. This fixes DNS access to NodeLocalDNS for Cilium clusters with restrictive egress NetworkPolicies, for example Web Terminal sessions with internet access disabled. Existing clusters require a restart of the Cilium DaemonSet for the new startup configuration to take effect if needed. Admins can either restart it manually or set Cilium’s rollOutCiliumPods=true Helm value, this will roll the agents automatically on configmap changes (#15996)
  • Fix the user cluster event count and last occurrence date (#8102)
  • Fix the web terminal failing with a TLS “unknown authority” error (#8135)
  • Vim is now pre-installed in the web-terminal image (#8134)

Updates

  • Update the default Cilium CNI version to 1.18.10 and added Cilium 1.17.16 and 1.18.10 as supported CNI versions (#15969)

Cleanup

  • The dashboard now reports its real version and edition at startup instead of always logging “Development” (#8131)

v2.29.8

GitHub release: v2.29.8

Bugfixes

  • Fix recovery for Helm-based ApplicationInstallations whose Helm release is stuck in a pending state or whose retry state no longer matches the deployed Helm release (#15915)
  • SSH keys from machine deployment providerSpec are no longer removed from worker nodes by the user-ssh-key-agent (#15863)
  • Cluster/machine metrics endpoints return an empty result for unavailable BYO CNI user clusters to avoid triggering KubermaticAPITooManyErrors alerts (#8060)
  • Fix background-repeat for multi-line ‘menu-item’ icons on low zoom (#8054)
  • Fix VSphere provider ignoring project-level allowed operating system restrictions during cluster creation (#8010)
  • Fix the Azure availability zone selector data population within the MachineDeployment edit dialog (#8028)
  • Fix the default OS image selection to correctly use the enabled OS when Ubuntu is disabled globally or per project (#7927)
  • Fix project creation dialog not applying admin-configured allowed operating systems to new projects (#7956)

Cleanups

Updates

  • Update vSphere CSI driver to v3.6.0 to pick up upstream session and ListView handling improvements that address vSphere volume attach failures after vCenter session expiry (#15766)
  • Add support of k8s patch releases v1.34.8/v1.33.12 (#15871)
  • Update kubeone package to v1.12.3 (#8047)
  • Update Operating System Manager to v1.8.1 (#15921)
  • Update Machine Controller to v1.64.2 (#15928)

v2.29.7

GitHub release: v2.29.7

Supported Kubernetes Versions

  • Add support for k8s patch releases v1.34.7/v1.33.11 (#15749)

New Features

  • Add cluster-level resource configuration for Kyverno. Users can now configure resource requests and limits for the Kyverno admission, background, cleanup, and reports controllers (#15770)

Bugfixes

  • Kubermatic-operator now reconciles Gateway API resources before Deployments, preventing missing ConfigMaps from blocking Gateway creation (#15712)

Updates

  • Update gpu-operator application to v26.3.0 (#15760)

v2.29.6

GitHub release: v2.29.6

Supported Kubernetes Versions

  • Add support for k8s patch releases v1.34.6/v1.33.10 (#15680)

API Changes

  • Add spec.ingress.gateway.infrastructureAnnotations to KubermaticConfiguration to configure Gateway.spec.infrastructure.annotations on the operator managed Gateway (#15725)

New Features

  • Add new alerts providing insights into health of cortex used by user-cluster MLA (#15630)
  • Dex HTTPRoute path and pathType are now configurable via httpRoute.path and httpRoute.pathType values, allowing users to deploy Dex on a separate subdomain with root path instead of being limited to path-based routing (#15627)
  • Envoy-gateway-controller: The envoyProxy image configuration now supports separate repository and tag fields for easier image mirroring. The legacy single-string format continues to work for backward compatibility (#15595)
  • Seed Grafana now has 12 new grafana dashboards under MLA Stack folder (#15603)

Bugfixes

  • Add missing condition to skip MLA Secrets deployment (#15659)
  • Fix Gateway API listener churn where kubermatic-operator would cyclically remove and re-add dynamic listeners during reconciliation. Dynamic listeners added by httproute-gateway-sync controller are now preserved (#15677)
  • Fix ineffective anti-affinity for the seed nodeport-proxy-envoy Deployment by aligning its anti-affinity selector with the pod labels actually used by the Deployment (#15601)
  • Gateway and HTTPRoute resources are now properly owned by KubermaticConfiguration and will be garbage collected on deletion. User-added labels and annotations on these resources are no longer overwritten during reconciliation (#15688)
  • The label key used for network policies for kubevirt virtual machines changed from cluster.x-k8s.io/cluster-name to kubermatic.k8c.io/cluster-id (#15606)
  • Fixed Kubeconfig download with non nginx ingresses (#7800)
  • Fix missing OIDC group scope for kubelogin kubeconfig to fix group mapping for KKP user clusters (#7990)
  • Respect datacenter selectors for default/enforced apps.Prevent duplicate app additions when switching datacenters.Fix loading enforced apps in the edit/customize cluster template (#7936)

v2.29.5

GitHub release: v2.29.5

Supported Kubernetes Versions

  • Add support for k8s patch release v1.34.5/v1.33.9/v1.32.13 (#15546)

New Features

  • Add HTTPRoute-Gateway sync controller to enable automatic certificate provisioning via cert-manager for KKP components (#15522)

Bugfixes

  • Add missing envoy-gateway-controller chart in release artifacts (#15491)
  • Fix alertmanager service port name reference after upstream chart migration (#15512)
  • Set hostname on Gateway while using cert-manager (#15496)
  • EE: Upgraded Kyverno to v1.15.3 to address CVE-2026-22039 and regenerated user-cluster Kyverno CRDs. (#15540)

Cleanup

  • Update nodeport-proxy Envoy to v1.35.8 (#15521)

v2.29.4

GitHub release: v2.29.4

Supported Kubernetes Versions

  • Add support of new Kubernetes patch releases v1.34.4/v1.33.8/v1.32.12 (#15472)

New Features

  • Add Gateway API support to the IAP chart, allowing IAP deployments to use HTTPRoute resources instead of Ingress when migrating to Gateway API (#15365)
  • Kubermatic now supports Gateway API for external traffic routing as an alternative to NGINX Ingress. It can be enabled via the --enable-gateway-api operator flag (set via migrateGatewayAPI: true in Helm values and --migrate-gateway-api in kubermatic-installer) (#15402)
  • The installer now uses a consistent scheme setup for both deploy and local kind commands (#15288)

Bugfixes

  • Add optional Seed setting spec.nodeportProxy.envoy.replicas to configure the nodeport-proxy-envoy replica count. If unset, existing default behavior remains (3 replicas) (#15464)

Updates

  • Update nginx-ingress-controller version to 1.14.3 (#15364)

v2.29.3

GitHub release: v2.29.3

New Features

  • Add status conditions for policy binding resources (#15209)

Bugfixes

  • Fix issue where OIDC kubeconfig downloads would fail with RBAC “Forbidden” errors when the identity provider returns uppercase email addresses (#7740)

Updates

v2.29.2

GitHub release: v2.29.2

Breaking Changes

  • Fix cluster-autoscaler RBAC permissions.cluster-autoscaler application needs to be re-installed to force recreating ApplicationInstallation resource, in order to get the new updated default values.yaml (#15152)
  • Update oauth2-proxy to appversion v7.13.0. If your configuration relies on matching query parameters in skip_auth_routes patterns, you must update your regex patterns to match paths only. Review all skip_auth_routes entries for potential impact. For detailed information, migration guidance, and security implications, see the upstream security advisory (#15174)

New Features

  • Add new env vars for KubeVirt provider in machine controller (#15253)
  • Set Tolerations overrides for control plane components (#15252)
  • Users can now configure additional arguments to oauth2-proxy pods. (useful for seed and user-mla) (#15241)

Bugfixes

  • Cortex upgrade to 1.16.1 fixing issue of cortex-ingester taking up a lot of storage space (#15242)
  • Delete orphaned UserProjectBinding resources on User or Project deletion (#15181)
  • Velero backup hook annotations have been corrected to use proper JSON format and ASCII quotes, fixing backup failures caused by invalid exec commands (#15217)
  • Fix Operating System Manager args, for flags like containerd-registry-mirrors (#15154)
  • Add omitempty to component settings fields to allow partial configuration (#15182)
  • Fix encryption at rest feature failing in environments with separate master and seed clusters (#7718)

Updates

  • Update machine-controller to v1.64.1 (#15267)
  • Add support of the latest k8s patch releases v1.34.3/v1.33.7 (#15239)

v2.29.1

GitHub release: v2.29.1

Supported Kubernetes Versions

  • Add support for k8s patch releases v1.34.2/v1.33.6/v1.32.10/v1.31.14 (#15170)

New Features

  • The image for KubeLB CCM can be overridden using .spec.userCluster.kubelb in the KubermaticConfiguration (#15159)
  • Update to KubeLB v1.2.1 (#15165)

Bugfixes

  • Fix azurefile-csi with kubernetes 1.31 and 1.32 (#15162)
  • Fix policy template selector targeting with empty target selectors (#15145)
  • A regression bug was fixed which introduced errors when a user tried to login with a user email containing uppercase letters and the one with only lowercase was already stored (#7671)
  • Fix a bug where the user cluster logging/monitoring checkboxes were shown even though user cluster MLA was disabled in the seed settings (#7681)

Updates

  • Update azuredisk-csi-driver to 1.32.11 for 1.32 kubernetes version and to 1.31.12 for 1.31 kubernetes version (#15147)

v2.29.0

GitHub release: v2.29.0

Breaking Changes

  • Bump cert-manager to v1.17.4 (#14853)

    • Cert-manager now hashes large RSA keys (3072 & 4096bit) with SHA-384 or SHA-512 respectively. If you are using these key sizes in your certificates, make sure your environment can handle the aforementioned hashing algorithms
    • Log messages that were not structured have now been replaced with structured logs. If you were matching on specific log strings, this could break your setup.

  • Update default Cilium version to 1.18.2 (#15095)

    • Cilium 1.18 will fail to start on Ubuntu 22.04 nodes using kernel 5.15.0-47-generic due to missing BPF verifier fixes. Upgrading to a newer kernel (either enabling “Upgrade system on first boot” from KKP UI, or using a newer kernel like 5.15.0-160), or using Ubuntu 24.04 will resolve the issue.

Supported Kubernetes Versions

  • Add support for Kubernetes version 1.34 (#14940)
  • Remove support for Kubernetes version 1.30 (#14828)
  • Add support for k8s patch releases 1.33.5/1.33.4/1.33.3/1.33.2/1.32.9/1.32.8/1.32.7/1.32.6/1.31.13/1.31.12/1.31.11/1.31.10 (#14998, #14910, #14830, #14783)

Supported Versions

  • 1.34.1
  • 1.33.5
  • 1.33.3
  • 1.33.2
  • 1.32.9
  • 1.32.7
  • 1.32.6
  • 1.31.13
  • 1.31.11
  • 1.31.10

Cloud Providers

KubeVirt

  • A bug was fixed where evicted KubeVirt VMs configured with evictionStrategy LiveMigrate were treated like VMs with External evictionStrategy by deleting the related machine object (#14736)
  • A bug regarding network policy cleanup in KubeVirt infra clusters when the removal of the finalizer failed after deleting the network policy was fixed (#14802)
  • Support KubeVirt vCPUs validation in the resource quota controller (#14728)

OpenStack

  • Add Load Balancer Class support for OpenStack cloud provider on cluster level (#15046)
  • Support IPv4 and IPv6 custom subnet for Openstack provider (#15080)
  • Add the ability to skip router reconciliation in the OpenStack provider (#14771)
  • Fix adding router-link OpenStack finalizer in the wrong place (#15086)

GCP

  • Fix Load Balancer assignment in Kubernetes 1.33 and 1.34 GCP clusters.(#15123)

New Features

  • Improve PolicyBinding resources cleanup(#15110)
  • The newly introduced external application catalog manager was added to kubermatic-installer mirror-images command to be respected in offline environments and fetching catalog apps from an OCI image when the external manager is enabled was introduced for that purpose. (#14995)
  • Add Kueue to the default applications catalog (#15004)
  • Non root device usage on worker nodes can now be enabled for containerd runtime by setting seed datacenter value spec.datacenter.node.enableNonRootDeviceOwnership to true (#14891)
  • The KubeLB tenant spec can now be defaulted at project level under .spec.defaultTenantSpec for KKP user cluster. For further details regarding this configuration, please take a look at KubeLB tenant docs (#14819)
  • Add the ability to configure kube-state-metrics in the KKP user clusters (#14829)
  • Promtail is replaced by Grafana Alloy as the log shipper in the KKP seed clusters (#14767)
  • Add an option to restrict project modification to the admins (#14843)
  • Overwrite system application images when overwriteRegistry is defined (#14773)
  • KubeLB: KKP defaulting will now enable KubeLB for a cluster if it’s enforced at the datacenter level (#14732)
  • Allow setting registry settings of container-runtime deployed user cluster through Cluster CR (#14745)
  • Enable DynamicResourceAllocation (DRA) for user clusters (#14872)
  • You can now use annotations and labels on user clusters to enable templating during application installations. This allows for dynamic configuration using expressions like {{- if eq (index .Cluster.Annotations “env”) “dev” }}custom1{{ else }}custom2{{ end }}. This feature is useful for more flexible multi-environment setups, for example (#14877)

Bugfixes

  • Fix invalid PolicyTemplate resources that set both spec.enforced and spec.namespacedPolicy (#15110)
  • Fix the default policy catalog --deploy-default-policy-template-catalog flag timing out in the kubermatic-installer (#15099)
  • [User Cluster MLA] Minor upgrade of Cortex to fix repeating errors in the logs (#14944)
  • The daemonset “node-local-dns” in the KKP user clusters now correctly defines port 9253 as the metrics port (#14926)
  • A caching functionality for used http.Transports when initializing MinIO clients in the seed-controller-manager is added to avoid TCP connection leaks (#14927, #14848)
  • Fix issue with CBSL credentials and status not syncing to seed clusters (#14703)
  • Add RBAC rules for Velero Backup resources to allow get, list, and watch operations (#14822)
  • Fix log spam on deleted ResourceQuota objects (#14714)
  • Fix a regression bug regarding node-exporter pod labeling which didn’t exclude node-exporter pods from pod discovery (#14740)
  • Add Velero post-backup hook to clean up /backup/* files after Prometheus backup completion to prevent disk space accumulation on the node where Prometheus is running (#14708)
  • A bug which lead to missing kube state metrics scraping was fixed (#14759)
  • Add the ETCDCTL_ENDPOINTS environment variable with name-based endpoints in all etcd pods. This enables successful execution of the etcdctl endpoint health command without the need for the --cluster flag which pulls IP based endpoints from the etcd ring (#14724)
  • Mirror the WebTerminal image (#15108)

Updates

  • Update OpenStack CSI version to 1.34.0 (#15115)
  • Bump KubeVirt CSI Driver Operator to v0.4.5 (#15096)
  • Add Cilium 1.17.7 and 1.18.2 as supported CNI version, deprecate cilium version 1.14.16 as it’s impacted by CVEs (#15095, #15065, #15048)
  • Update default Canal version to v3.30.3 and deprecate v3.27 (#15078)
  • Update machine-controller to v1.64.0 (#15131)
  • Update operating-system-manager to v1.8.0 (#15130)
  • Update nginx-ingress-controller version to 1.13.2 (#15036)
  • Update Dex chart to appversion 2.44.0 (#15041)
  • KubeLB CCM has been upgraded to v1.2.0 (#14961)
  • Update Prometheus federation configuration to include machine deployment metrics from user clusters in the seed MLA Prometheus (#14817)
  • Update helm to v3.17.4 (#14831)
  • Update the user cluster and metering Prometheus instances in the KKP Seed cluster to scrape kubelet_volume_stats_capacity_bytes and kubelet_volume_stats_used_bytes metrics from the KKP user clusters (#14769)
  • Update kubermatic-installer local kind Dex static client configurations (#14735)
  • Update Go version to 1.25.1 (#14940)
  • Replace Bitnami charts and images with kubermatic-mirror charts and images to address issues identified in bitnami/containers#83267 (#14873)

Cleanups

  • Gateway API CRDs installation and management have been delegated to KubeLB, that natively manages these CRDs using “-install-gateway-api-crds” and “-gateway-api-crds-channel” flags (#14919)
  • Remove support for Equinix Metal (Packet) provider (#14827)
  • By default the oauth2-proxy disables Dex’s approval screen now. To return to the old behaviour, set approval_prompt = "force" for each IAP deployment in your Helm values.yaml (#14751)
  • Early deprecation of unsupported Falco versions 0.35.1 and 0.37.0 from the default application catalog, since they are not compatible with modern Linux Kernel versions present in machine templates (#14861)
  • The deprecated field defaultComponentSettings in the Seed Resource has been removed (#15102)

Dashboard and API

Cloud Providers

GCP
  • Fix disk types and machine types values are not loaded in cluster template for Google Cloud Provider (#7639)
OpenStack
  • Add OpenStack LoadBalancer Class configuration support at the Cluster level (#7646)
  • Add a new option to enable the config drive on the OpenStack provider for machine deployments, along with a datacenter level option to enforce it for all machine deployments (#7516)
  • Add new option in the user-cluster to skip router reconciliation option for OpenStack provider (#7483)
  • Fix network selection to display network ID when name is missing in OpenStack (#7513)
KubeVirt
  • Skip setting custom CPUs field in machine deployment for KubeVirt user clusters (#7493)

New Features

  • Add strict validation for cluster encryption keys requiring proper 32-byte base64 format (#7653)
  • Add the overwrite-registry flag in the api server (#7651)
  • Add support to toggle Encryption at Rest for Edit Cluster dialog (#7599)
  • Enables Encryption at rest feature for secrets (#7525)
  • Display node labels in the nodes list table (#7588)
  • Add dialog to choose authentication type (KKP API or OIDC-kubelogin) when downloading or sharing kubeconfig via OIDC (#7522)
  • Use cabundle as key for caching http.Transport (#7591)
  • Make NVIDIA GPU Operator labels accessible through Dashboard API (#7520)
  • Web-terminal: k9s, krew, and krew plugins ns, ctx, and oidc-login are available to use in the web-terminal image (#7509)
  • Add a new button in the initial node step of the user cluster wizard to configure the Cluster Autoscaler application (#7500)
  • Add support to delete presets from the admin interface with detailed linkage information (#7479)
  • Add an option to restrict project modification to admins (#7504)
  • Move web-terminal cleanup job to seed to fix cleanup not working when the token is expired (#7451)

Bugfixes

  • Kyverno policy bindings disappear when the template selector no longer matches the cluster. Enforcing Kyverno Policy disables the Namespaced option. (#7654)
  • Fix the Kyverno Policybinding in a multi-seed setup (#7631)
  • Fix encryption configuration handling during cluster editing (#7620)
  • Fix cluster template editing when autoscaler application is not present (#7619)
  • Fix a possible null pointer exception for isGlobalViewer (#7610)
  • Disable min/max options if cluster autoscaler is not available (#7559)
  • Fix web terminal token expiration by refreshing expired tokens automatically (#7508)
  • Project viewers can now only view cluster templates. Create, update, and delete actions are restricted except deletion by the owner (#7446)
  • Fix validation error when switching expose strategy from Tunneling to LoadBalancer by clearing tunnelingAgentIP automatically (#7422)
  • Fix KubeLB checkbox state management and UI flickering issues in cluster creation wizard/edit cluster dialog (#7458)
  • KubeLB: Fix a bug where enforcement on a datacenter was not enabling KubeLB for the user clusters in the dashboard (#7453)

Updates

  • Update KKP SDK to include subnetAllocationPool and subnetCIDR (#7626)
  • Update Go version to v1.25.1 (#7554)
  • Update Node version to 22 (#7539)
  • Update web-terminal image to v0.11.0 (#7509)

Cleanups

  • Remove deprecated Azure Basic Load Balancer SKU option, defaulting to Standard SKU (#7590)
  • Remove Equinix (Packet) provider support from cluster creation, KubeOne, and presets (#7533)