Before upgrading, make sure to read the general upgrade guidelines. Consider tweaking seedControllerManager.maximumParallelReconciles
to ensure user cluster reconciliations will not cause resource exhaustion on seed clusters. A full upgrade guide is available from the official documentation.
Read Before Upgrading
- ACTION REQUIRED: legacy backup controller has been removed. Before upgrading, please change to the backup and restore feature that uses backup destinations if the legacy controller is still in use (#12473)
s3-storeuploader
has been removed (#12473)- OpenVPN for control plane to node connectivity has been deprecated. It will be removed in future releases of KKP. Upgrading all user cluster to Konnectivity is strongly recommended (#12691)
- User clusters require upgrading to Kubernetes 1.26 prior to upgrading to KKP 2.24 (#12740)
Action Required
- ACTION REQUIRED (EE ONLY): Update metering component to v1.1.1, fixing highly inaccurate data in cluster reports. Reports generated in KKP v2.23.2+ or v2.22.5+ do not represent actual consumption. Ad-hoc reports for time frames that need correct consumption data can be generated by following our documentation (#12822)
API Changes
- The field
vmNetName
in Cluster
and Preset
resources for vSphere clusters is deprecated and networks
should be used instead (#12444) - The field
konnectivityEnabled
in Cluster
resources is deprecated. Clusters should set this to true
to migrate off OpenVPN as Konnectivity being enabled will be assumed in future KKP releases (#12691) - Set Cilium as default CNI for user clusters (#12752)
Supported Kubernetes Versions
- Add support for Kubernetes 1.28 (#12593)
- Add support for Kubernetes 1.26.9, 1.27.6 and 1.28.2 (#12638)
- Set default Kubernetes version to 1.27.6 (#12638)
- Remove support for Kubernetes 1.24 (#12570)
- Remove support for Kubernetes 1.25 (#12740)
Supported Versions
- v1.26.1
- v1.26.4
- v1.26.6
- v1.26.9
- v1.27.3
- v1.27.6 (default)
- v1.28.2
KubeLB (Enterprise Edition only)
This release adds support for KubeLB, a cloud native multi-tenant load balancing solution by Kubermatic.
- Add KubeLB integration with KKP; introduce KubeLB as a first-class citizen in KKP (#12667)
- Extend cluster health status with KubeLB health check (#12685)
- Support for enforcing KubeLB at the datacenter level (#12685)
- Support to configure node address type for KubeLB at the datacenter level (#12715)
- Update KubeLB CCM image to v0.4.0 (#12786)
Metering (Enterprise Edition only)
- Following fields are removed from metering reports (#12545)
- Cluster reports
- Removal of
total-used-cpu-seconds
, use average-used-cpu-millicores
instead - Removal of
average-available-cpu-cores
, use average-available-cpu-millicores
instead
- Namespace reports
- Removal of
total-used-cpu-seconds
, use average-used-cpu-millicores
instead
- Add
monthly
parameter for metering monthly report generation (#12544) - Update metering component to v1.1.1, fixing highly inaccurate data in cluster reports (see Action Required for more details) (#12822)
Cloud Providers
Azure
- Remove Azure NSG rules that only duplicated rules always present in NSGs (#12565)
- The
icmp_allow_all
rule of the Azure NSG created for each cluster now only allows ICMP and takes precedence over the TCP and UDP catch-all rules that were guarding it (#12559)
vSphere
- Support for configuring multiple networks for vSphere (#12444)
- Support for propagating vSphere cluster tags to folders created by KKP (#12581)
- Update vSphere CCM to 1.27.2 for Kubernetes 1.27 user clusters (#12599)
- If a vSphere cluster uses a custom datastore, the Seed’s default datastore should not be validated (#12655)
- Add
basePath
optional configuration for vSphere clusters that will be used to construct a cluster-specific folder path (<root path>/<base path>/<cluster ID>
or <base path>/<cluster ID>
) (#12668) - Fix a bug where datastore cluster value was not being propagated to the CSI driver (#12474)
- Migrate
CSIDriver
csi.vsphere.vmware.com
to no longer advertise inline ephemeral volumes as supported (#12813)
DigitalOcean
Hetzner
- Hetzner CSI: recreate CSIDriver to allow upgrade from 1.6.0 to 2.2.0 (#12432)
- EE: Correctly validate Hetzner API response for server type while calculating resource requirements and for networks while validating cloud spec (#12716)
CNIs
Cilium
- Set Cilium as default CNI for user clusters (#12752)
- Add support for Cilium 1.14.3 and 1.13.8 and deprecate previous patch releases, mitigating CVE-2023-44487, CVE-2023-39347, CVE-2023-41333, CVE-2023-41332 (#12761)
- Update Cilium v1.11 and v1.12 patch releases to v1.11.20 and v1.12.13 (#12561)
- Remove and replace deprecated
clusterPoolIPv4PodCIDR
and clusterPoolIPv6PodCIDR
Helm value with clusterPoolIPv4PodCIDRList
and clusterPoolIPv6PodCIDRList
for Cilium 1.13+ (#12561)
Canal
- Add support for Canal v3.26.1 (#12561)
- Deprecate Canal v3.23 (#12561)
- Mark all Canal CRDs with preserveUnknownFields: false (#12538)
MLA
- Mark MLA Grafana dashboards as non-editable as they are managed by KKP (#12626)
- Fix configuration live reload for monitoring-agent and logging-agent (#12507)
- Grafana Kubernetes dashboard will not repeatedly ask to be saved (#12614)
- Replace
irate
with rate
for node cpu usage graphs (#12427) - The
kube_service_labels
metric was not scraped with all expected labels, due to a change in labels on the kube-state-metrics service. The related scraping config was adapted accordingly (#12551) - Fix default url configuration of Blacbox exporter (#12412)
- Fix several Prometheus record and alert rules (#12533)
- Made Prometheus Helm chart extensible so that external metric storage solutions like Thanos can be easily integrated for seed long-term monitoring (#12425)
- Fixes for the Kubernetes overview dashboard in Grafana (#12520)
- Fix CPU Utilization graph showing no data for User Cluster MLA dashboard “Nodes Overview” (#12814)
- Fix empty panels in Grafana dashboard “Resource Usage per Namespace” for Master/Seed MLA (#12816)
New Features
- EE: Default ApplicationCatalog can be deployed via
--deploy-default-app-catalog
flag (#12623) - Add
disableCsiDriver
as optional field on Cluster
and Seed
resources to disable CSI driver deployment. This can be configured at a user cluster and datacenter level. If the admin disables CSI drivers at a datacenter level then the user is prohibited from enabling them at the user cluster level (#12515) - Introduce
DisableAdminKubeconfig
flag in KubermaticSettings
to disable the admin kubeconfig feature from dashboard (#12679) - Disabled CSI addon on user clusters where it was enabled & then disabled using
DisableCSIDriver
option. The CSI addon is removed only if the CSI drivers created by it are not in use (#12621) - Extend
kubermatic-installer mirror-images
command with an option to export a tarball instead of syncing to a remote repository. This can be helpful in airgapped scenarios (#12613) - Extend MinIO configuration options to allow enabling MinIO console access and exposing MinIO API and console via Ingress (#12683)
- New configuration option for Dex (
oauth
chart): Allow modification of web frontend issuer (#12608) - Support for configuring IPFamilies and IPFamilyPolicy for nodeport-proxy (#12472)
- Support for configuring OIDC username and group prefix for user clusters (#12648)
- Support for configuring the Dex theme via values file (#12560)
- Switch backup containers to use
etcd-launcher snapshot
for creating etcd database snapshots (#12462) - Use OCI VM images as preconfigured default for local KubeVirt setup (#12534)
- Allow to modify allocation range in IPAM Pools (#12423)
Bugfixes
- Add missing cluster-autoscaler release for user clusters using Kubernetes 1.27 (#12597)
- Add missing images from envoy-agent
DaemonSet
in Tunneling expose strategy when running kubermatic-installer mirror-images
(#12537) - Fix always defaulting allowed node port IP ranges for user clusters to 0.0.0.0/0 and ::/0, even when a more specific IP range was given (#12589)
- Fix an issue in Applications, which resulted in “empty git-upload-pack given” errors for git sources (#12487)
- Fix an issue in the
kubermatic-installer mirror-images
command, which led to failure on the mla-consul chart (#12513) - Fix an issue where IPv6 IPs were being ignored when determining the address of a user cluster (#12505)
- Fix node-labeller controller not applying the
x-kubernetes.io/distribution
label to RHEL nodes (#12751) - Fix reconcile loop for
seed-proxy-token
Secret on Kubernetes 1.27 (#12557) - Increase memory limit of kube-state-metrics addon to 600Mi (#12692)
kubermatic-installer
will now validate the existing MinIO filesystem before attempting a kubermatic-seed
stack installation (#12477)- Increase default CPU limits for KKP API/seed/master-controller-managers to prevent general slowness (#12764)
- Extend project-synchronizer controller in kubermatic-master-controller-manager to propagate labels from Projects in the master cluster to Projects in the seed cluster. This fixes an issue where the metering report doesn’t contain project-labels in separate master/seed setups (#12791)
Updates
- Update Vertical Pod Autoscaler to 0.14.0 (#12604)
- Update
d3fk/s3cmd
to version (latest “arch-stable”) with fb4c4dcf
hash (#12640) - Update cert-manager to 1.12.2 (#12443)
- Update curl in
kubermatic/util
image and mla/grafana
chart to 8.4.0 (CVE-2023-38545 and CVE-2023-38546 do not affect KKP) (#12694) - Update
quay.io/kubermatic/util
(helper image) to 2.3.1 (includes curl version patched against CVE-2023-38545 and CVE-2023-38546) (#12726) - Update etcd for user clusters to 3.5.9 (#12453)
- Update KubeVirt chart for the installer local command to 1.0.0 (#12470)
- Update metering Prometheus to next LTS version 2.45.0 (#12532)
- Update metrics-server for all deployments to 0.6.4 (#12516)
- Update nginx-ingress-controller to 1.9.3 (fixes CVE-2023-44487, HTTP/2 rapid reset attack) (#12712)
- Update supported Kubernetes releases for EKS/AKS (#12579)
- Update telemetry-agent to 0.4.1 (#12572)
- Update controller-runtime to 0.16.1 and Kubernetes libraries to 1.28 (#12609)
- Update Go to 1.21.3 (#12697)
- Update KubeVirt CDI for local installer to 1.57.0 (#12605)
- Add Kubernetes 1.28 to EKS versions, remove Kubernetes 1.23 (#12789)
- Update machine-controller to v1.58.0 (#12825)
- Update operating-system-manager to v1.4.0 (#12826)
Miscellaneous
- Use
etcd-launcher
to check if etcd is running before starting kube-apiserver and to defragment etcd clusters (#12450) - Create a
NetworkPolicy
for user cluster kube-apiserver to access the Seed Kubernetes API (#12569) - Improve
http-prober
performance in user clusters with a lot of CRDs (#12634) - Update Velero helm chart’s apiVersion to v2; Helm 3 & above would be required to install it (#12765)
Dashboard & API
Cleanup
- Remove unused v1 endpoints for KKP API (#6116)
Bugfixes
- Add operating system profile to the machine deployment patch object (#6264)
- Add vertical scroll to the install Addon dialog (#6123)
- Allow expansion of sidenav on small screen sizes (#6218)
- Fix a bug where available version upgrades for CNI plugins were not being properly deduced (#6317)
- Fix a bug where network and IPv6 subnet pool options were not loading during Openstack cluster creation (#6120)
- Fix a bug where project scope endpoints for GCP were working only with the presets instead of one of presets or credentials (#6078)
- CE: Fix a bug where the values configured for vSphere, Hetzner, and Nutanix nodes were not being persisted (#6171)
- Fix an issue where a custom OSP value was not selected when editing/customizing cluster template (#6325)
- Fix docs link about OIDC groups on user settings page (#6208)
- Fix listing events for external clusters (#6337)
- Fix support for keycloak OIDC logout. New field
oidc_provider
was introduced to support OIDC provider specific configurations. Configuring oidc_provider
as keycloak
will properly configure the logout workflow (#6144) - Fix the default value for CNI plugin version (#6258)
- Fix the empty
id_token_hint
value when logout from Keycloak (#6248) - Fix: vSphere tags for initial machine deployments (#6179)
- OpenStack: Fix project and projectID header propagation for project scoped endpoints (#6082)
- Openstack: take
TenantID
into account while listing networks, security groups and subnet pools (#6156) - VMware Cloud Director: fix an issue where the API Token from preset was not being sourced to the cluster (#6196)
- Fix
Enable Share Cluster
button in Admin Settings (#6340) - Fix an issue where
clusterDefaultNodeSelector
label was being added back on opening of edit cluster dialog (#6362) - Fix issue with managing clusters if some seeds are down (#6374)
- Fix a bug where API call to list projects was failing due to slowness (#6385)
New Features
- Support for enabling/disabling operating systems for machines of user clusters (#6070)
- Add functionality to configure
basePath
in preset and cluster for vSphere (#6281) - Add support for encrypted root volumes in AWS (#6125)
- Add VM anti-affinity setting for vSphere machine deployments (#6068)
- EE: Support for configuring KubeLB for user clusters (#6256)
- Support for configuring multiple networks for vSphere (#6069)
- Support for disabling admin kubeconfig endpoint (#6246)
- Support multiple NodePort allowed IP ranges (#6188)
- Update default CNI plugin to
Cilium
(#6328) - VMware Cloud Director: Support for configuring placement and sizing policy for machines (#6094)
- Enforce Konnectivity value because OpenVPN support is now deprecated (#6361)
Updates
- Update to Go 1.21.3 (#6268)
- Update web-terminal image to kubectl 1.27, Helm 3.12.3 and curl 8.4.0 (#6283)