Air-Gap Installation

Overview

Air-gap mode lets you install KubeLB Enterprise Edition in clusters that have no network access to public container or chart registries. All container images and Helm charts are mirrored to a private OCI registry inside your network, and the KubeLB charts are configured to pull every image from that mirror.

What is supported:

• Registry rewrite for charts and images across the entire stack: manager, CCM, connection-manager, Envoy Proxy data plane, and all addons, etc.
• A single imagePullSecret propagated to manager, CCM, connection-manager and all addon workloads, including manager-created Envoy Proxy pods.
• A self-contained mirror bundle shipped inside the kubelb-manager-ee Helm chart at airgapped/. Everything you need to mirror is in there, including a script that uses crane to copy all images and charts to your mirror in one shot.

What is not supported:

• Mixing public and mirror registries. Air-gap mode is fail-closed; every image reference must resolve through the mirror.

Prerequisites

• A private OCI registry where all the artifacts will be mirrored. It must be reachable from the air-gapped clusters and from a staging host with internet access (can be the same host).
• Credentials with push access to your mirror, pull access from inside the air-gapped clusters, and pull credentials for quay.io/kubermatic/* (the EE manager chart and EE images are private).
• On a staging host that can reach both the public registries and your mirror the following tools:
  1. helm 3.8+
  2. kubectl 1.28+
  3. Crane

Step 1: Fetch the air-gap bundle

The kubelb-manager-ee Helm chart ships a self-contained mirror bundle under airgapped/. Pull the chart from a host with internet access and extract it:

# Pick the version you want to install.
VERSION=v1.5.0

helm pull oci://quay.io/kubermatic/helm-charts/kubelb-manager-ee \
  --version ${VERSION} --untar
cd kubelb-manager-ee/airgapped

You will find:

All image references are pinned with sha256 digests at release time, so a subsequent retag of an upstream image cannot silently change what you mirror.

Step 2: Mirror images and charts to your registry

Authenticate crane to both the source registries (Quay credentials are required because EE images and the EE manager chart are private) and your target mirror, then run the bundled script:

MIRROR=mirror.internal

crane auth login quay.io -u <quay-user> -p <quay-token>
crane auth login ${MIRROR} -u <mirror-user> -p <mirror-pass>

# Mirror everything: images + charts in one shot.
./mirror-images.sh ${MIRROR}

The script preserves the path after the registry host and skips artifacts already present at the same digest, so re-runs are cheap. To mirror a subset, pass a different list:

# Only the core (no addons).
./mirror-images.sh ${MIRROR} images-core.txt

# Only one addon.
./mirror-images.sh ${MIRROR} images-metallb.txt

The mirror layout looks like this:

If you would rather drive crane yourself or feed the lists to a different tool (Harbor replication, Artifactory remote, hauler), the *.txt files are stable, deterministic inputs. There is nothing mirror-images.sh specific about them.

Step 3: Create the image pull secret

Create a docker-registry Secret named mirror-creds in the kubelb namespace on the management cluster and on every tenant cluster:

kubectl create namespace kubelb --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret docker-registry mirror-creds \
  --namespace kubelb \
  --docker-server=mirror.internal \
  --docker-username=<user> \
  --docker-password=<password>

The same secret name is referenced by every chart in the next steps via global.imagePullSecrets. Use whatever name fits your convention; just keep it consistent across the install commands below.

Step 4: Install the KubeLB Manager

Install the manager chart on the management cluster, pointing it at the mirror:

helm install kubelb-manager \
  oci://mirror.internal/kubermatic/helm-charts/kubelb-manager-ee \
  --version v1.5.0 \
  --namespace kubelb --create-namespace \
  --set global.imageRegistry=mirror.internal \
  --set global.imagePullSecrets[0].name=mirror-creds

global.imageRegistry rewrites every image referenced by the chart, including the manager binary, kube-rbac-proxy, the connection-manager, the Envoy Proxy data plane and the Envoy Gateway shutdown manager. global.imagePullSecrets is propagated to every pod spec the chart and the manager controllers create.

How the runtime rewrite works

The chart passes global.imageRegistry to the manager binary as the --image-registry flag. At runtime the manager rewrites the Envoy Proxy, shutdown-manager, and WAF WASM init-container images before creating pods: the first path segment is stripped when it contains a dot or a colon (i.e. when it is a registry host), and the mirror registry is prepended. For example, docker.io/envoyproxy/envoy:distroless-v1.36.4 becomes mirror.internal/envoyproxy/envoy:distroless-v1.36.4, but cr.agentgateway.dev/controller:v1.3.1 becomes mirror.internal/controller:v1.3.1 because controller is the whole repository path. Your mirror layout must account for this: images always end up at <mirror>/<path-without-registry-host>. The rewrite applies to explicitly overridden image values too, so overrides only need to exist in the mirror at the matching path.

The WAF WASM init container defaults to the detected manager pod image, which is already mirrored. If you override it via kubelb.waf.wasmInitContainerImage, the override is still passed through the same runtime rewrite, so the image must exist in your mirror at the rewritten path.

If you need to install CRDs separately (for example, to manage them with a GitOps tool), pull and untar the chart first and apply crds/ before helm install:

helm pull oci://mirror.internal/kubermatic/helm-charts/kubelb-manager-ee \
  --version v1.5.0 --untar
kubectl apply -f kubelb-manager-ee/crds/

Helm applies crds/ directories only on install, never on upgrade. Re-apply the CRDs on every upgrade — see Upgrading KubeLB.

Step 5: Install the KubeLB CCM

Run the CCM install on every tenant cluster. The kubelb namespace must already contain the cluster Secret produced by tenant registration (see Setup Tenant Cluster) and the mirror-creds pull secret created in Step 3.

helm install kubelb-ccm \
  oci://mirror.internal/kubermatic/helm-charts/kubelb-ccm-ee \
  --version v1.5.0 \
  --namespace kubelb --create-namespace \
  --set global.imageRegistry=mirror.internal \
  --set global.imagePullSecrets[0].name=mirror-creds \
  --set kubelb.clusterSecretName=kubelb-cluster \
  --set kubelb.tenantName=<unique-identifier-for-tenant>

When the mTLS backend transport mode is enabled, the CCM also creates the tenant proxy DaemonSet on the tenant cluster. Its images come from kubelb.tenantProxy.envoy.image and kubelb.tenantProxy.shutdownManager.image in the kubelb-ccm chart (defaults: docker.io/envoyproxy/envoy and docker.io/envoyproxy/gateway). Both are covered by images-core.txt and are rewritten through global.imageRegistry, so they must be present in your mirror at envoyproxy/envoy and envoyproxy/gateway. If you override either value, the override must resolve through the mirror as well.

Step 6: Install the KubeLB Addons

The kubelb-addons chart bundles upstream community charts (ingress-nginx, envoy-gateway, cert-manager, external-dns, metallb, agentgateway). The published OCI chart ships with air-gap patches already applied to each addon’s templates, so the same global.imageRegistry value flows through to every addon image:

helm install kubelb-addons \
  oci://mirror.internal/kubermatic/helm-charts/kubelb-addons \
  --version v0.4.0 \
  --namespace kubelb \
  --set global.imageRegistry=mirror.internal \
  --set global.imagePullSecrets[0].name=mirror-creds \
  --set ingress-nginx.enabled=true \
  --set envoy-gateway.enabled=true \
  --set cert-manager.enabled=true \
  --set cert-manager.crds.enabled=true

Only enable the addons you actually use. Each addon condition is gated by <addon>.enabled in charts/kubelb-addons/values.yaml.

Step 7: Verify the install

Check that every running pod references your mirror and not a public registry:

kubectl get pods --all-namespaces \
  -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' \
  | sort -u

Every line in the output should start with mirror.internal/. If any line starts with quay.io/, docker.io/, registry.k8s.io/, gcr.io/, ghcr.io/ or cr.agentgateway.dev/, that container will fail to pull once the cluster is fully air-gapped. Fix the override before disconnecting.

To verify the mirror itself, the kubermatic/kubelb-ee repository ships a make verify-image-list-remote target that regenerates the image lists and checks with crane that every listed artifact exists in the registries.

You can perform the same check against rendered output before applying:

helm template kubelb-addons oci://mirror.internal/kubermatic/helm-charts/kubelb-addons \
  --version v0.4.0 \
  -f addons-airgap-values.yaml \
  | grep -E '^\s*image:' | sort -u

Troubleshooting

ImagePullBackOff with unauthorized on a tenant cluster

The mirror-creds pull secret is missing from a namespace where KubeLB schedules workloads. The CCM install only puts it in kubelb. If you have set kubelb.namespace or the manager creates Envoy Proxy pods in a different namespace, copy the secret there too, or re-run Step 3 with the additional namespace.

An addon image still references a public registry

Re-render with helm template ... --set global.imageRegistry=mirror.internal | grep image: to find the offender. If a specific image is not getting rewritten, file an issue against kubermatic/kubelb-ee; the air-gap patch for that addon chart needs an update.

Manager-created Envoy Proxy pods pull from docker.io

The Envoy data plane and the Envoy Gateway shutdown manager are exposed on the manager chart as kubelb.envoyProxy.image and kubelb.envoyProxy.gracefulShutdown.shutdownManagerImage. They are rewritten automatically at runtime when global.imageRegistry is set on the manager chart, including explicitly overridden values. If a rewritten image fails to pull, the mirror is missing the image at the rewritten path (see “How the runtime rewrite works” in Step 4). The same applies to the CCM chart values kubelb.tenantProxy.envoy.image and kubelb.tenantProxy.shutdownManager.image used by the mTLS tenant proxy.