Release Notes


Before upgrading, make sure to read the general upgrade guidelines. Consider tweaking seedControllerManager.maximumParallelReconciles to ensure user cluster reconciliations will not cause resource exhaustion on seed clusters. A full upgrade guide is available from the official documentation.

Read Before Upgrading

  • ACTION REQUIRED: legacy backup controller has been removed. Before upgrading, please change to the backup and restore feature that uses backup destinations if the legacy controller is still in use (#12473)
  • s3-storeuploader has been removed (#12473)
  • OpenVPN for control plane to node connectivity has been deprecated. It will be removed in future releases of KKP. Upgrading all user cluster to Konnectivity is strongly recommended (#12691)
  • User clusters require upgrading to Kubernetes 1.26 prior to upgrading to KKP 2.24 (#12740)

Action Required

  • ACTION REQUIRED (EE ONLY): Update metering component to v1.1.1, fixing highly inaccurate data in cluster reports. Reports generated in KKP v2.23.2+ or v2.22.5+ do not represent actual consumption. Ad-hoc reports for time frames that need correct consumption data can be generated by following our documentation (#12822)

API Changes

  • The field vmNetName in Cluster and Preset resources for vSphere clusters is deprecated and networks should be used instead (#12444)
  • The field konnectivityEnabled in Cluster resources is deprecated. Clusters should set this to true to migrate off OpenVPN as Konnectivity being enabled will be assumed in future KKP releases (#12691)
  • Set Cilium as default CNI for user clusters (#12752)

Supported Kubernetes Versions

  • Add support for Kubernetes 1.28 (#12593)
  • Add support for Kubernetes 1.26.9, 1.27.6 and 1.28.2 (#12638)
  • Set default Kubernetes version to 1.27.6 (#12638)
  • Remove support for Kubernetes 1.24 (#12570)
  • Remove support for Kubernetes 1.25 (#12740)

Supported Versions

  • v1.26.1
  • v1.26.4
  • v1.26.6
  • v1.26.9
  • v1.27.3
  • v1.27.6 (default)
  • v1.28.2

KubeLB (Enterprise Edition only)

This release adds support for KubeLB, a cloud native multi-tenant load balancing solution by Kubermatic.

  • Add KubeLB integration with KKP; introduce KubeLB as a first-class citizen in KKP (#12667)
  • Extend cluster health status with KubeLB health check (#12685)
  • Support for enforcing KubeLB at the datacenter level (#12685)
  • Support to configure node address type for KubeLB at the datacenter level (#12715)
  • Update KubeLB CCM image to v0.4.0 (#12786)

Metering (Enterprise Edition only)

  • Following fields are removed from metering reports (#12545)
    • Cluster reports
      • Removal of total-used-cpu-seconds, use average-used-cpu-millicores instead
      • Removal of average-available-cpu-cores, use average-available-cpu-millicores instead
    • Namespace reports
      • Removal of total-used-cpu-seconds, use average-used-cpu-millicores instead
  • Add monthly parameter for metering monthly report generation (#12544)
  • Update metering component to v1.1.1, fixing highly inaccurate data in cluster reports (see Action Required for more details) (#12822)

Cloud Providers


  • Remove Azure NSG rules that only duplicated rules always present in NSGs (#12565)
  • The icmp_allow_all rule of the Azure NSG created for each cluster now only allows ICMP and takes precedence over the TCP and UDP catch-all rules that were guarding it (#12559)


  • Support for configuring multiple networks for vSphere (#12444)
  • Support for propagating vSphere cluster tags to folders created by KKP (#12581)
  • Update vSphere CCM to 1.27.2 for Kubernetes 1.27 user clusters (#12599)
  • If a vSphere cluster uses a custom datastore, the Seed’s default datastore should not be validated (#12655)
  • Add basePath optional configuration for vSphere clusters that will be used to construct a cluster-specific folder path (<root path>/<base path>/<cluster ID> or <base path>/<cluster ID>) (#12668)
  • Fix a bug where datastore cluster value was not being propagated to the CSI driver (#12474)
  • Migrate CSIDriver to no longer advertise inline ephemeral volumes as supported (#12813)



  • Hetzner CSI: recreate CSIDriver to allow upgrade from 1.6.0 to 2.2.0 (#12432)
  • EE: Correctly validate Hetzner API response for server type while calculating resource requirements and for networks while validating cloud spec (#12716)



  • Set Cilium as default CNI for user clusters (#12752)
  • Add support for Cilium 1.14.3 and 1.13.8 and deprecate previous patch releases, mitigating CVE-2023-44487, CVE-2023-39347, CVE-2023-41333, CVE-2023-41332 (#12761)
  • Update Cilium v1.11 and v1.12 patch releases to v1.11.20 and v1.12.13 (#12561)
  • Remove and replace deprecated clusterPoolIPv4PodCIDR and clusterPoolIPv6PodCIDR Helm value with clusterPoolIPv4PodCIDRList and clusterPoolIPv6PodCIDRList for Cilium 1.13+ (#12561)


  • Add support for Canal v3.26.1 (#12561)
  • Deprecate Canal v3.23 (#12561)
  • Mark all Canal CRDs with preserveUnknownFields: false (#12538)


  • Mark MLA Grafana dashboards as non-editable as they are managed by KKP (#12626)
  • Fix configuration live reload for monitoring-agent and logging-agent (#12507)
  • Grafana Kubernetes dashboard will not repeatedly ask to be saved (#12614)
  • Replace irate with rate for node cpu usage graphs (#12427)
  • The kube_service_labels metric was not scraped with all expected labels, due to a change in labels on the kube-state-metrics service. The related scraping config was adapted accordingly (#12551)
  • Fix default url configuration of Blacbox exporter (#12412)
  • Fix several Prometheus record and alert rules (#12533)
  • Made Prometheus Helm chart extensible so that external metric storage solutions like Thanos can be easily integrated for seed long-term monitoring (#12425)
  • Fixes for the Kubernetes overview dashboard in Grafana (#12520)
  • Fix CPU Utilization graph showing no data for User Cluster MLA dashboard “Nodes Overview” (#12814)
  • Fix empty panels in Grafana dashboard “Resource Usage per Namespace” for Master/Seed MLA (#12816)

New Features

  • EE: Default ApplicationCatalog can be deployed via --deploy-default-app-catalog flag (#12623)
  • Add disableCsiDriver as optional field on Cluster and Seed resources to disable CSI driver deployment. This can be configured at a user cluster and datacenter level. If the admin disables CSI drivers at a datacenter level then the user is prohibited from enabling them at the user cluster level (#12515)
  • Introduce DisableAdminKubeconfig flag in KubermaticSettings to disable the admin kubeconfig feature from dashboard (#12679)
  • Disabled CSI addon on user clusters where it was enabled & then disabled using DisableCSIDriver option. The CSI addon is removed only if the CSI drivers created by it are not in use (#12621)
  • Extend kubermatic-installer mirror-images command with an option to export a tarball instead of syncing to a remote repository. This can be helpful in airgapped scenarios (#12613)
  • Extend MinIO configuration options to allow enabling MinIO console access and exposing MinIO API and console via Ingress (#12683)
  • New configuration option for Dex (oauth chart): Allow modification of web frontend issuer (#12608)
  • Support for configuring IPFamilies and IPFamilyPolicy for nodeport-proxy (#12472)
  • Support for configuring OIDC username and group prefix for user clusters (#12648)
  • Support for configuring the Dex theme via values file (#12560)
  • Switch backup containers to use etcd-launcher snapshot for creating etcd database snapshots (#12462)
  • Use OCI VM images as preconfigured default for local KubeVirt setup (#12534)
  • Allow to modify allocation range in IPAM Pools (#12423)


  • Add missing cluster-autoscaler release for user clusters using Kubernetes 1.27 (#12597)
  • Add missing images from envoy-agent DaemonSet in Tunneling expose strategy when running kubermatic-installer mirror-images (#12537)
  • Fix always defaulting allowed node port IP ranges for user clusters to and ::/0, even when a more specific IP range was given (#12589)
  • Fix an issue in Applications, which resulted in “empty git-upload-pack given” errors for git sources (#12487)
  • Fix an issue in the kubermatic-installer mirror-images command, which led to failure on the mla-consul chart (#12513)
  • Fix an issue where IPv6 IPs were being ignored when determining the address of a user cluster (#12505)
  • Fix node-labeller controller not applying the label to RHEL nodes (#12751)
  • Fix reconcile loop for seed-proxy-token Secret on Kubernetes 1.27 (#12557)
  • Increase memory limit of kube-state-metrics addon to 600Mi (#12692)
  • kubermatic-installer will now validate the existing MinIO filesystem before attempting a kubermatic-seed stack installation (#12477)
  • Increase default CPU limits for KKP API/seed/master-controller-managers to prevent general slowness (#12764)
  • Extend project-synchronizer controller in kubermatic-master-controller-manager to propagate labels from Projects in the master cluster to Projects in the seed cluster. This fixes an issue where the metering report doesn’t contain project-labels in separate master/seed setups (#12791)


  • Update Vertical Pod Autoscaler to 0.14.0 (#12604)
  • Update d3fk/s3cmd to version (latest “arch-stable”) with fb4c4dcf hash (#12640)
  • Update cert-manager to 1.12.2 (#12443)
  • Update curl in kubermatic/util image and mla/grafana chart to 8.4.0 (CVE-2023-38545 and CVE-2023-38546 do not affect KKP) (#12694)
  • Update (helper image) to 2.3.1 (includes curl version patched against CVE-2023-38545 and CVE-2023-38546) (#12726)
  • Update etcd for user clusters to 3.5.9 (#12453)
  • Update KubeVirt chart for the installer local command to 1.0.0 (#12470)
  • Update metering Prometheus to next LTS version 2.45.0 (#12532)
  • Update metrics-server for all deployments to 0.6.4 (#12516)
  • Update nginx-ingress-controller to 1.9.3 (fixes CVE-2023-44487, HTTP/2 rapid reset attack) (#12712)
  • Update supported Kubernetes releases for EKS/AKS (#12579)
  • Update telemetry-agent to 0.4.1 (#12572)
  • Update controller-runtime to 0.16.1 and Kubernetes libraries to 1.28 (#12609)
  • Update Go to 1.21.3 (#12697)
  • Update KubeVirt CDI for local installer to 1.57.0 (#12605)
  • Add Kubernetes 1.28 to EKS versions, remove Kubernetes 1.23 (#12789)
  • Update machine-controller to v1.58.0 (#12825)
  • Update operating-system-manager to v1.4.0 (#12826)


  • Use etcd-launcher to check if etcd is running before starting kube-apiserver and to defragment etcd clusters (#12450)
  • Create a NetworkPolicy for user cluster kube-apiserver to access the Seed Kubernetes API (#12569)
  • Improve http-prober performance in user clusters with a lot of CRDs (#12634)
  • Update Velero helm chart’s apiVersion to v2; Helm 3 & above would be required to install it (#12765)

Dashboard & API


  • Remove unused v1 endpoints for KKP API (#6116)


  • Add operating system profile to the machine deployment patch object (#6264)
  • Add vertical scroll to the install Addon dialog (#6123)
  • Allow expansion of sidenav on small screen sizes (#6218)
  • Fix a bug where available version upgrades for CNI plugins were not being properly deduced (#6317)
  • Fix a bug where network and IPv6 subnet pool options were not loading during Openstack cluster creation (#6120)
  • Fix a bug where project scope endpoints for GCP were working only with the presets instead of one of presets or credentials (#6078)
  • CE: Fix a bug where the values configured for vSphere, Hetzner, and Nutanix nodes were not being persisted (#6171)
  • Fix an issue where a custom OSP value was not selected when editing/customizing cluster template (#6325)
  • Fix docs link about OIDC groups on user settings page (#6208)
  • Fix listing events for external clusters (#6337)
  • Fix support for keycloak OIDC logout. New field oidc_provider was introduced to support OIDC provider specific configurations. Configuring oidc_provider as keycloak will properly configure the logout workflow (#6144)
  • Fix the default value for CNI plugin version (#6258)
  • Fix the empty id_token_hint value when logout from Keycloak (#6248)
  • Fix: vSphere tags for initial machine deployments (#6179)
  • OpenStack: Fix project and projectID header propagation for project scoped endpoints (#6082)
  • Openstack: take TenantID into account while listing networks, security groups and subnet pools (#6156)
  • VMware Cloud Director: fix an issue where the API Token from preset was not being sourced to the cluster (#6196)
  • Fix Enable Share Cluster button in Admin Settings (#6340)
  • Fix an issue where clusterDefaultNodeSelector label was being added back on opening of edit cluster dialog (#6362)
  • Fix issue with managing clusters if some seeds are down (#6374)
  • Fix a bug where API call to list projects was failing due to slowness (#6385)

New Features

  • Support for enabling/disabling operating systems for machines of user clusters (#6070)
  • Add functionality to configure basePath in preset and cluster for vSphere (#6281)
  • Add support for encrypted root volumes in AWS (#6125)
  • Add VM anti-affinity setting for vSphere machine deployments (#6068)
  • EE: Support for configuring KubeLB for user clusters (#6256)
  • Support for configuring multiple networks for vSphere (#6069)
  • Support for disabling admin kubeconfig endpoint (#6246)
  • Support multiple NodePort allowed IP ranges (#6188)
  • Update default CNI plugin to Cilium (#6328)
  • VMware Cloud Director: Support for configuring placement and sizing policy for machines (#6094)
  • Enforce Konnectivity value because OpenVPN support is now deprecated (#6361)


  • Update to Go 1.21.3 (#6268)
  • Update web-terminal image to kubectl 1.27, Helm 3.12.3 and curl 8.4.0 (#6283)