GitHub release: v1.5.0
KubeLB v1.5 ships KubeLB Insights — a deterministic advisor for the platform operator. The management cluster already knows every tenant’s effective configuration, every route and every WAF policy; the insights engine periodically evaluates that state against a registry of 15 checks and records what it finds as Insight resources.
Config.spec.insights.disabledChecks.kubelb_manager_posture_score metric is scored per tenant and category, with a bundled Grafana dashboard and Prometheus alert rules (findings, degraded posture, checks failing to evaluate, stalled sweep loop).kubelb.enableInsights: false.Management-to-tenant backend traffic can now be encrypted end to end. Enabled from the management cluster with Config.spec.backendTransport.mode: MTLS, traffic is routed through a KubeLB-managed Envoy proxy in the tenant cluster instead of straight to workload NodePorts. TCP backends use raw mTLS/SNI; UDPRoute uses a CONNECT-UDP tunnel over the same port.
Config.spec.backendTransport.udp.mode switches UDP between Tunnel (default, encrypted CONNECT-UDP) and Direct (classic per-service NodePort, unencrypted) for workloads sensitive to the tunnel’s MTU overhead. TCP stays on mTLS either way.TenantState, so tenant-side controllers cannot drift from what the manager has configured.Direct <-> MTLS re-plumbs the tenant dataplane, so it is gated behind an explicit kubelb.k8c.io/confirm-backend-transport-change annotation on the Config.kubelb.tenantProxy.serviceType and kubelb.tenantProxy.staticAddresses/staticPort on the kubelb-ccm chart cover proxies behind an appliance, NAT, or user-managed DNS.Tenants can now define namespaced TenantWAFPolicy resources against their own routes, instead of every rule going through the platform operator. The feature is opt-in per installation (Config.spec.waf.enableTenantPolicies) and per tenant (Tenant.spec.waf); tenant directives are validated against a strict allowlist and each policy is isolated to the tenant’s own namespace.
WAF also gained dataplane observability this release — Grafana dashboards and Prometheus alerts for blocked requests, filter failures, Coraza VM reloads and xDS NACKs (alerts off by default via prometheusRule.enabled).
The KubeLB CLI — until now released from kubermatic/kubelb-cli on its own version line — has been merged into this repository under cli/. It is developed, tested and released on the same tag and the same cycle as the manager and CCM, with full supply-chain parity: keyless cosign signatures and provenance attestations on every artifact.
v0.2.0 → v1.5.0 and tracks the KubeLB release from here on.kubelb-cli_<version>_<os>_<arch> from the GitHub release; the binary inside the archive is named kubelb.expose, loadbalancer, ingress (including convert and preview for the Ingress → Gateway API migration), status, serve and tunnel. tunnel requires Enterprise Edition.VERSION=1.5.0
curl -LO https://github.com/kubermatic/kubelb/releases/download/v${VERSION}/kubelb-cli_${VERSION}_linux_amd64.tar.gz
tar -xzf kubelb-cli_${VERSION}_linux_amd64.tar.gz kubelb
sudo install kubelb /usr/local/bin/
Verify the download with keyless cosign:
gh attestation verify kubelb-cli_${VERSION}_linux_amd64.tar.gz --repo kubermatic/kubelb
Existing kubelb-cli users have two behaviour changes to pick up — see Urgent Upgrade Notes.
Stock Envoy timeouts were cutting off streaming and large-file workloads — object-storage downloads, model artifacts, websockets, SSE, long-running gRPC streams — because the KubeLB Envoy layer was double-bounding requests that the edge proxy already bounds.
| Timeout | Was | Now |
|---|---|---|
Per-request (RouteAction.Timeout) | 15s (Envoy default) | disabled |
| HTTP idle connection | 60s | 1h |
| HCM stream idle | 5m (Envoy default) | 1h |
Enterprise Edition additionally makes six timeouts configurable at Config, Tenant, Route and LoadBalancer level, with kubelb.k8c.io/timeout-* annotations on Ingress/HTTPRoute/GRPCRoute/TCPRoute/TLSRoute: request, streamIdle, requestHeaders, idleConnection, tcpIdle and connect.
kubelb.k8c.io/managed-by=kubelb, and CRDs that disappear upstream between versions are now pruned instead of being left pinned at a stale bundle version.GatewayClassNotAccepted warning event instead of being ignored.backendRefs and Gateway TLS certificateRefs can be restricted to references permitted by a ReferenceGrant, via spec.gatewayAPI.enforceReferenceGrants on Config/Tenant (default off).Tenant.spec.gatewayAPI.classMappings and Config.spec.gatewayAPI.classMappings map tenant-cluster GatewayClasses to distinct management-cluster GatewayClasses.kubelb.k8c.io/backend-pool annotation.A focused pass on the Envoy control plane, driven by what showed up at scale:
kubelb_envoy_control_plane_xds_nacks_total exports configs rejected by Envoy.KubeLB Dashboard v1.1.0 is the companion release for KubeLB v1.5. Install via oci://quay.io/kubermatic/helm-charts/kubelb-dashboard; source at kubermatic/kubelb-dashboard.
Community Edition (CE)
deniedAnnotations on Config and Tenant, and shell-style globs (nginx.ingress.kubernetes.io/*) in both propagatedAnnotations and deniedAnnotations.Config.spec.envoyProxy.headerLimits defaults to Envoy’s maximum, so large client headers no longer produce 431 Request Header Fields Too Large.LoadBalancer.spec.persistence.type: SourceIP; tenant Services using sessionAffinity: ClientIP are propagated with observed-source persistence.STRICT_DNS clusters instead of EDS, fixing “malformed IP address” rejections. EndpointAddress.ip is now optional.Enterprise Edition (EE)
kubelb.k8c.io/health-check-* annotations on tenant Services, Ingresses and Routes.LoadBalancerStatus carries an Accepted condition surfaced by the CCM as a Warning event on the tenant Service, route rejections appear on the tenant cluster object’s status, and a Route rejected for a hostname outside allowedDomains gets a warning event on the tenant’s Ingress/Gateway/Route.spec.persistence.type: SourceIP, taking precedence over loadBalancerPolicy when both are set.Beyond the control plane work above, v1.5 lands a broad batch of correctness fixes:
kubelb.k8c.io/managed-annotations so third-party annotations are not clobbered.PortAllocator reads is fixed.(No, really, you MUST read this before you upgrade)
kubermatic/kubelb and shares the KubeLB version number (kubelb-cli v0.2.0 → v1.5.0). Download kubelb-cli_<version>_<os>_<arch> archives from the kubermatic/kubelb release assets; the binary inside is named kubelb.kubelb-cli static cosign key no longer applies to new releases.kubelb serve now listens on 127.0.0.1:8080 by default instead of on all interfaces. (#497)envoyProxy.resources in Config/Tenant still overrides. (#570)Config.spec.timeouts.idleConnection: 60s and Config.spec.timeouts.request: 15s. (#434)XListenerSet CRD — removed upstream in Gateway API v1.5 and still pinned at bundle v1.4.1 — is dropped from the bundle. CRDs that disappear upstream are now pruned on download instead of lingering in tenant clusters. (#541, #565, #566)deniedAnnotations to Config and Tenant for excluding annotation keys from propagation. Keys in propagatedAnnotations and deniedAnnotations now support shell-style glob patterns (e.g. nginx.ingress.kubernetes.io/*). (#454)ip field on EndpointAddress is now optional; at least one of ip or hostname must be set. (#438)cli/ and released on the same tag and cycle as the manager and CCM, with keyless cosign signatures and provenance attestations. (#497)Config.spec.envoyProxy.headerLimits, defaulting to Envoy’s maximum so large client headers no longer cause 431 Request Header Fields Too Large. (#519)sessionAffinity: ClientIP are propagated with observed-source persistence in KubeLB. (#435)kubelb.k8c.io/managed-by=kubelb. (#487)envoyProxy.resources in Config/Tenant overrides this. (#570)kubelb_envoy_control_plane_xds_nacks_total for configs rejected by Envoy. (#532)STRICT_DNS clusters instead of EDS clusters, fixing “malformed IP address” rejections for tenants that use hostname-based endpoints. (#438)no matches for kind "PodMonitor" on clusters without the Prometheus Operator CRDs installed. (#439)global.imageRegistry rewriting for the cert-manager addon after the subchart bump to 1.21.0. (#524)kubelb-addons to v0.5.0 and pin it in the kubelb-manager chart, with dependency bumps:XListenerSet CRD, removed upstream in Gateway API v1.5, and prune Gateway API CRDs that disappear upstream on download. (#565, #566)Full Changelog: https://github.com/kubermatic/kubelb/compare/v1.4.0...v1.5.0
Enterprise Edition includes everything from Community Edition and more. The release notes below are for changes specific to just the Enterprise Edition.
(No, really, you MUST read this before you upgrade)
kubelb.enableInsights: false to turn it off. (#518)WAFPolicy and TenantWAFPolicy no longer accept targetRef.kind: Ingress. WAF applies to Gateway API routes (HTTPRoute, GRPCRoute); it was never enforced on Ingress traffic. Existing policies targeting Ingress must be retargeted. (#489)WAFPolicy directives are now capped at 64 items / 1024 characters, with Config.spec.waf.maxDirectivesPerPolicy and maxDirectiveLength defaulting to those values. Set either to 0 for unlimited. (#486)kubelb.envoyProxy.resources). (#553, #554)Config.spec.timeouts.idleConnection: 60s and Config.spec.timeouts.request: 15s. (#386)Config.spec.backendTransport.mode requires explicit confirmation. A Direct <-> MTLS flip re-plumbs the tenant dataplane, so it is not applied until the Config carries kubelb.k8c.io/confirm-backend-transport-change set to the target mode. Until then each TenantState reports BackendTransportChangePending. (#430)Config.spec.prometheus replaces Config.spec.ai.prometheus. The setting was generalized out of the AI tree and is now owned by Insights. (#502)--enable-insights) that evaluates the management cluster against a registry of checks and records findings as Insight resources, with triage (acknowledge, snooze, dismiss), per-check suppression via Config.spec.insights.disabledChecks, and Prometheus metrics. (#507)kubelb.k8c.io/timeout-* annotations on Ingress/HTTPRoute/GRPCRoute/TCPRoute/TLSRoute propagate to Route.spec.timeouts. Six fields are configurable: request, streamIdle, requestHeaders, idleConnection, tcpIdle, connect. (#386)kubelb.k8c.io/health-check-* annotations on tenant Services, Ingresses and Routes. (#471)backendRefs and Gateway TLS certificateRefs can now be restricted to references permitted by a ReferenceGrant, via spec.gatewayAPI.enforceReferenceGrants on Config/Tenant (default off). Also fixed CCM RBAC so events on tenant objects are no longer rejected. (#435)Tenant.spec.gatewayAPI.classMappings and Config.spec.gatewayAPI.classMappings. (#423)Accepted condition, surfaced by the CCM as a Warning event on the tenant Service. (#556)Config.spec.waf.maxDirectivesPerPolicy and maxDirectiveLength to 64/1024. Set either to 0 for unlimited. (#486)Config.spec.backendTransport.mode: MTLS. TCP backends use raw mTLS/SNI; UDPRoute uses a CONNECT-UDP tunnel over the same port. Config.spec.backendTransport.udp.mode switches UDP between Tunnel (default) and Direct. Ships as Beta / Technical Preview, with per-tenant name-constrained intermediates, two-phase certificate rotation and a mode-flip confirmation gate. (#387, #428, #429, #430, #433, #434, #436, #437, #557, #562)TenantWAFPolicy resources to apply WAF rules to their own routes. Opt-in per installation (Config.spec.waf.enableTenantPolicies) and per tenant (Tenant.spec.waf); tenant directives are validated against a strict allowlist and each policy is isolated to the tenant’s own namespace. (#444)kubelb_manager_posture_score metric scored per tenant and category. (#511)prometheusRule.enabled). (#484)kubelb.k8c.io/backend-pool annotation. (#440)spec.persistence.type: SourceIP. Services using sessionAffinity: ClientIP are propagated with observed-source persistence; persistence takes precedence over loadBalancerPolicy when both are set. (#550)kubelb.tenantProxy.serviceType on the kubelb-ccm chart overrides the Service type, and kubelb.tenantProxy.staticAddresses/staticPort publish static IPs or DNS names as the dial target for proxies behind an appliance, NAT, or user-managed DNS. (#528)tenant.spec.allowedDomains. (#405)kubelb in more than one tenant cluster namespace resolved to a single object in the management cluster and silently overwrote each other. The first namespace to claim the name now keeps it and later claimants get an error on the Route. Gateways with any other name were already namespace-qualified and are unaffected. Routes whose sub-resources fail to apply are no longer marked Accepted; the failure is reported as a condition and event, and retried. (#529)TenantProxyConfigured reports the rendered backend count.TenantProxy* conditions reset when a tenant leaves MTLS mode.kubelb.envoyProxy.resources. (#554)TenantWAFPolicy Accepted status is now shown in kubectl output. (#560)allowedDomains now have their generated resources removed, instead of continuing to serve the previously accepted configuration. Disabling a resource type on a Tenant or Config also removes the mirrored resource, which was previously left behind. (#540)GatewayClassNotAccepted warning event instead of being ignored silently. (#538)failureMode now governs runtime WASM failures, invalid fail-closed policies now block traffic instead of serving it unprotected, per-route policy resolution no longer leaks between routes sharing an origin name, Ingress routes can now be protected, and --enable-waf=false fully disables WAF filter injection. (#443)SecRemoteRules, filesystem Include, and audit/log/tmp/data path directives) and correctly validates multi-line directive entries. Such policies are now marked invalid instead of being accepted and failing at runtime. (#442)Ready condition (phase as reason) instead of accumulating one condition per phase; stale phase-typed conditions are removed automatically. (#498)-v=2 and requests rejected for missing headers now log which headers were absent. (#525)kubelb.k8c.io/backend-tls-ca-secret) are now carried as SyncSecrets instead of native Secrets. Under the scoped tenant ServiceAccount that write was forbidden and hung rather than erroring, wedging every subsequent Service reconcile behind the first Verify-policy Service. Bundles are also keyed on the origin UID, so two Services in different tenant-cluster namespaces referencing the same Secret name no longer overwrite each other. (#544)sessionAffinityConfig.clientIP.timeoutSeconds is set, since source-IP session affinity uses Maglev hashing and has no stickiness timeout to honour. (#561)nginx.ingress.kubernetes.io/limit-rps and limit-rpm annotation values above 4294967295 wrapping to a very low rate limit. (#519)no matches for kind "PodMonitor" on clusters without the Prometheus Operator CRDs installed. (#391)global.imageRegistry rewriting for it in air-gapped setups. (#476)kubelb-manager-ee chart now ships airgapped/mirror-images.sh with the executable bit set; customers no longer need a manual chmod +x step. (#395)kubelb-addons chart to v0.5.0 (cert-manager 1.21.1, agentgateway 1.4.1) and drop the XListenerSet CRD removed upstream in Gateway API v1.5. (#551)For Community Edition, the release artifacts are available on GitHub Releases.
The kubelb-cli_<version>_<os>_<arch> archives on that page ship the KubeLB CLI and are used by both editions.
# Login to registry
docker login quay.io -u <username> -p <password>
# kubelb manager
docker pull quay.io/kubermatic/kubelb-manager-ee:v1.5.0
# ccm
docker pull quay.io/kubermatic/kubelb-ccm-ee:v1.5.0
# connection-manager
docker pull quay.io/kubermatic/kubelb-connection-manager-ee:v1.5.0
# kubelb-manager
helm pull oci://quay.io/kubermatic/helm-charts/kubelb-manager-ee --version v1.5.0
# kubelb-ccm
helm pull oci://quay.io/kubermatic/helm-charts/kubelb-ccm-ee --version v1.5.0
# kubelb-addons
helm pull oci://quay.io/kubermatic/helm-charts/kubelb-addons --version v0.5.0
# kubelb-dashboard
helm pull oci://quay.io/kubermatic/helm-charts/kubelb-dashboard --version v1.1.0
Container image SBOMs are attached as OCI artifacts and attested with cosign.
Pull SBOM:
# Login to registry
oras login quay.io -u <username> -p <password>
## kubelb-manager
SBOM_DIGEST=$(oras discover --format json --artifact-type application/spdx+json \
quay.io/kubermatic/kubelb-manager-ee:v1.5.0 | jq -r '.referrers[0].digest')
oras pull quay.io/kubermatic/kubelb-manager-ee@${SBOM_DIGEST} --output sbom/
## kubelb-ccm
SBOM_DIGEST=$(oras discover --format json --artifact-type application/spdx+json \
quay.io/kubermatic/kubelb-ccm-ee:v1.5.0 | jq -r '.referrers[0].digest')
oras pull quay.io/kubermatic/kubelb-ccm-ee@${SBOM_DIGEST} --output sbom/
## kubelb-connection-manager
SBOM_DIGEST=$(oras discover --format json --artifact-type application/spdx+json \
quay.io/kubermatic/kubelb-connection-manager-ee:v1.5.0 | jq -r '.referrers[0].digest')
oras pull quay.io/kubermatic/kubelb-connection-manager-ee@${SBOM_DIGEST} --output sbom/
Verify SBOM attestation:
cosign verify-attestation quay.io/kubermatic/kubelb-manager-ee:v1.5.0 \
--type spdxjson \
--certificate-identity-regexp="^https://github.com/kubermatic/kubelb-ee/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
cosign verify-attestation quay.io/kubermatic/kubelb-ccm-ee:v1.5.0 \
--type spdxjson \
--certificate-identity-regexp="^https://github.com/kubermatic/kubelb-ee/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
cosign verify-attestation quay.io/kubermatic/kubelb-connection-manager-ee:v1.5.0 \
--type spdxjson \
--certificate-identity-regexp="^https://github.com/kubermatic/kubelb-ee/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
Docker images:
cosign verify quay.io/kubermatic/kubelb-manager-ee:v1.5.0 \
--certificate-identity-regexp="^https://github.com/kubermatic/kubelb-ee/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
cosign verify quay.io/kubermatic/kubelb-ccm-ee:v1.5.0 \
--certificate-identity-regexp="^https://github.com/kubermatic/kubelb-ee/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
cosign verify quay.io/kubermatic/kubelb-connection-manager-ee:v1.5.0 \
--certificate-identity-regexp="^https://github.com/kubermatic/kubelb-ee/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
Helm charts:
cosign verify quay.io/kubermatic/helm-charts/kubelb-manager-ee:v1.5.0 \
--certificate-identity-regexp="^https://github.com/kubermatic/kubelb-ee/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
cosign verify quay.io/kubermatic/helm-charts/kubelb-ccm-ee:v1.5.0 \
--certificate-identity-regexp="^https://github.com/kubermatic/kubelb-ee/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
cosign verify quay.io/kubermatic/helm-charts/kubelb-addons:v0.5.0 \
--certificate-identity-regexp="^https://github.com/kubermatic/kubelb/.github/workflows/release.yml@refs/tags/addons-v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
Release checksums (requires repository access):
cosign verify-blob --bundle checksums.txt.sigstore.json checksums.txt \
--certificate-identity-regexp="^https://github.com/kubermatic/kubelb-ee/.github/workflows/release.yml@refs/tags/v.*" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com