Bring Your Own KDP

In this mode, KubeV joins a KDP you already run, configured with kdp.external. Two commands set it up:

1. kubermatic-virtualization apply installs KubeV and points its dashboard and API server at your kcp. It installs nothing on your KDP.
2. kubermatic-virtualization kdp connect registers KubeV in your KDP, with the kubeconfig of a kcp administrator, and connects it to the KubeV cluster.

Your identity provider is the only thing left to change by hand: kdp connect prints the client KubeV needs there.

The api-syncagent runs on the KubeV cluster and talks to your kcp at two addresses: the front-proxy, with the kubeconfig KDP issues for the KubeV service, and kcp’s virtual workspace, through which it syncs every object.

How kdp connect helps

Registering KubeV in KDP takes dozens of objects in two places, in kcp and on the KubeV cluster, and some of them must be applied in a set order. kdp connect does all of it in one run:

kdp connect is safe to run again at any time: a second run changes only what differs.

The flow

Before you start

To see the virtual workspace address your kcp advertises, list the endpoints of KDP’s own APIExports in root:

kubectl --kubeconfig kcp-admin.kubeconfig get apiexportendpointslices \
  -o jsonpath='{.items[*].status.endpoints[*].url}'

Each address must be one the KubeV cluster reaches, such as https://kcp.example.com:8443/services/apiexport/... on the front-proxy, not a name inside the KDP cluster such as https://kcp:6443/....

Get a kcp administrator’s kubeconfig

Any administrator of root can run the first kdp connect, through OIDC or a client certificate: your own OIDC user, if your KDP binds it to cluster-admin in root, or a client certificate in the group system:kcp:admin, which kcp lets administer every workspace. kcp records such a user as the creator of the provider workspace: it skips only system:masters.

On a KDP installed with the kcp Helm chart, cert-manager issues such a certificate from the front-proxy’s client issuer, in the namespace kcp runs in:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kubev-connect
  namespace: <kcp namespace>
spec:
  secretName: kubev-connect
  commonName: kubev-connect
  subject:
    organizations: ["system:kcp:admin"]
  usages: ["client auth"]
  duration: 24h
  issuerRef:
    name: kcp-front-proxy-client-issuer
    kind: Issuer

Build the kubeconfig from its Secret, with the front-proxy’s address and the CA that signs the front-proxy’s certificate, which the chart keeps in the Secret kcp-ca, as tls.crt and ca.crt:

apiVersion: v1
kind: Config
clusters:
  - name: kcp
    cluster:
      server: https://kcp.example.com:8443/clusters/root
      certificate-authority-data: <base64 of kcp-ca's tls.crt followed by its ca.crt>
users:
  - name: kcp-admin
    user:
      client-certificate-data: <base64 of kubev-connect's tls.crt>
      client-key-data: <base64 of kubev-connect's tls.key>
contexts:
  - name: kcp-admin
    context: {cluster: kcp, user: kcp-admin}
current-context: kcp-admin

Delete the Certificate and its Secret once kdp connect is done, and issue a new one for the next run. The kubeconfig in the chart’s Secret kcp-external-admin-kubeconfig doesn’t do: its user can only read root, and kdp connect lists the rights it lacks.

Step 1: Configure KubeV for your KDP

Add the dashboard’s sign-in and a kdp section with an external block to your KubeV configuration:

dashboard:
  enabled: true
  dashboardURL: "https://kubev.example.com"      # where users open KubeV; KDP links here
  auth:
    oidc:
      issuerURL: "https://dex.example.com"        # the issuer your kcp trusts
      clientID: "kdp-kubelogin"                   # the client ID kcp accepts
      clientSecret: "<that client's secret>"
      redirectURL: "https://kubev.example.com/auth/callback"
      scopes: ["openid", "email", "profile", "groups"]

kdp:
  enabled: true
  providerWorkspace: "root:kubev"                 # created by kdp connect if missing
  external:
    kcpBaseURL: "https://kcp.example.com:8443"    # kcp's front-proxy, no /clusters path
  tenantUnderlayInterfaces:
    - "ens[0-9]*"

Step 2: Install KubeV

kubermatic-virtualization apply -f cluster.yaml

In this mode, apply installs KubeV and sets its dashboard and API server up for your kcp, and changes nothing on your KDP. At the end, it prints the kdp connect command to run next. It writes the KubeV cluster’s kubeconfig, kubev-cluster-kubeconfig, to the directory it runs in: kdp connect uses it.

Until you run kdp connect, the KubeV dashboard opens but answers 502 for every project: its API server has no kcp CA yet. Without external.caSecretName, apply warns about it too.

kdp connect reaches the KubeV cluster at the address in kubev-cluster-kubeconfig, apiEndpoint.host. To run it from another machine, or through a tunnel or a port-forward, that address must work from there and be in the API server’s certificate: list it in apiEndpoint.alternativeNames before you run apply, for example 127.0.0.1 and localhost for a port-forward, and point the kubeconfig’s server at it.

Step 3: Connect KubeV to KDP

From the same directory, run kdp connect with the kcp administrator’s kubeconfig:

kubermatic-virtualization kdp connect -f cluster.yaml --kcp-kubeconfig kcp-admin.kubeconfig

kdp connect works in this order, and stops at the first problem:

1. It checks. KubeV must be installed on the KubeV cluster, the kcp the kubeconfig reaches must run KDP, and the kubeconfig must be allowed every write that follows in kcp. kcp answers these questions itself, through access reviews. A missing right is named, with its workspace:

   Error: the kcp kubeconfig (user "oidc:jane@example.com") may not do everything kdp connect does in kcp; use the kubeconfig of a kcp admin. Missing:
     - create workspaces.tenancy.kcp.io in workspace root
     - use workspacetypes.tenancy.kcp.io kdp-organization in workspace root
   

2. It shows what it will write, and asks. The output includes lines like:

   kdp connect will register KubeV in the KDP at https://kcp.example.com:8443 as <kcp user>:
     in kcp, workspace root:kubev (created if missing): the KDP Service virtualization.k8c.io, the KubeV admission policies, and RBAC letting every signed-in user bind it
     in kcp, workspace root: RBAC letting every signed-in user bind KDP's core export, so new organizations and projects are ready at once
     on the KubeV cluster at https://203.0.113.10:6443: the api-syncagent in namespace kubev-syncagent, its RBAC, the KubeV admission policies and the PublishedResources
     on the KubeV cluster: the kcp CA for the api-server, in Secret kubermatic-virtualization/kubev-kcp-front-proxy-ca, unless you wrote that Secret yourself
   

3. It writes everything in What kdp connect writes, in this order: kcp’s CA for the KubeV API server; the RBAC in root that lets every signed-in user bind KDP’s core API, before the provider workspace exists, so that kcp can finish setting it up; the provider workspace and the KubeV service; then, once KDP has issued the api-syncagent’s kubeconfig, the kcp admission policies and everything on the KubeV cluster; and last, the RBAC that lets users bind KubeV. Waiting for KDP can take several minutes on the first run, while kcp sets up the provider workspace.

4. It waits until the api-syncagent runs and has published every KubeV resource type in kcp, for up to five minutes. You stop it with Ctrl-C at any point: running it again finishes the job.

   Waiting for the api-syncagent to publish the KubeV resources in kcp...
   ✓ The api-syncagent published every KubeV resource in root:kubev
   

5. It prints the client KubeV needs on your Dex, for Step 4, and what to do next.

It also warns when your KDP’s front-proxy URL, or kcp’s virtual workspace address, looks like an address inside the KDP cluster, which the api-syncagent on the KubeV cluster can’t reach.

What kdp connect writes

kcp’s CA

The KubeV API server verifies kcp’s certificate with the CA in a Secret in its own namespace, and can’t reach kcp at all while that Secret is missing. An empty ca.crt means the publicly trusted CAs. The API server reads the Secret again when it changes, without a restart.

• With external.caSecretName, the Secret is yours: kdp connect never writes it, and warns while it is missing.
• A kubev-kcp-front-proxy-ca Secret you created yourself is left as it is too.
• Otherwise, kdp connect fills kubev-kcp-front-proxy-ca from the kcp kubeconfig:
  • when the kubeconfig names the same address as kdp.external.kcpBaseURL, with the kubeconfig’s CA, or an empty one when the kubeconfig names none;
  • when it names another address, such as a kcp shard, with the kubeconfig’s CA if kcp’s certificate at kdp.external.kcpBaseURL verifies with it, or an empty one if the publicly trusted CAs verify it. If neither does, it stores nothing and asks you to name your own Secret;
  • when it can’t reach kdp.external.kcpBaseURL from where it runs, it keeps a CA it stored earlier. With none stored yet, it stores the kubeconfig’s CA, unchecked, and warns.

Rights kdp connect checks

kdp connect asks kcp whether the kubeconfig may do each of these, and lists every one it may not:

kcp keeps RBAC per workspace: cluster-admin in root covers root only. When kdp connect creates the provider workspace, kcp makes the kubeconfig’s user its administrator, which covers the second row. Later, the provider workspace’s administrators are whoever created it, kdp.admin or kdp.adminGroup once kdp connect has run, and anyone granted cluster-admin there.

Step 4: Register KubeV on your Dex

Your Dex knows nothing of KubeV yet. Add KubeV’s callback to the client kcp trusts, next to the KDP dashboard’s own. Edit that client: a second client with the same ID doesn’t work. kdp connect prints the whole entry; the change is:

staticClients:
  - id: kdp-kubelogin                     # the client kcp trusts, dashboard.auth.oidc.clientID
    redirectURIs:
      - https://kdp.example.com/api/auth/callback/oidc   # already there: the KDP dashboard's
      - https://kubev.example.com/auth/callback          # add: KubeV's
oauth2:
  skipApprovalScreen: true

skipApprovalScreen stops Dex from asking users to approve KubeV each time they open it from KDP.

Skip Dex’s chooser

If your Dex offers more than one login method, users pick theirs on Dex’s page each time they open KubeV from KDP. To skip that page, KDP’s link can name the login method the user signed in to KDP with:

1. Have your KDP dashboard ask Dex for the federated:id scope, in its authentication.oidc.additionalScopes setting. Without it, KDP doesn’t know the login method and shows no button at all for such a link.

2. List your Dex’s login methods, its connector IDs, in dashboard.auth.oidc.connectors, and run apply again:

   dashboard:
     auth:
       oidc:
         connectors: ["local", "github", "google"]
   

3. Run kdp connect again, with --forward-connector. Pass it on every later run too: a run without it takes the login method back out of KDP’s link.

A user whose KDP session started before you added the scope sees no button until they sign in to KDP again.

Step 5: Enable KubeV in a project and check it

1. In the KDP dashboard, open a project, and enable KubeV Virtualization on its Services page. In the wizard, keep Create default roles for resources managed by this service selected, and grant every resource listed under Resources: KubeV needs each of them.

2. On the project, choose Open KubeV Virtualization. KubeV opens in the project, signed in as you.

3. Create an SSH key in KubeV. It appears on the KubeV cluster in the host namespace of its project namespace, kv-<project hash>-<namespace hash>, which carries the labels kdp.kubev.k8c.io/organization, kdp.kubev.k8c.io/project and kdp.kubev.k8c.io/namespace:

   kubectl --kubeconfig kubev-cluster-kubeconfig get sshkeys.virtualization.k8c.io -A
   kubectl --kubeconfig kubev-cluster-kubeconfig get namespaces -l kdp.kubev.k8c.io/project-hash \
     -L kdp.kubev.k8c.io/organization,kdp.kubev.k8c.io/project,kdp.kubev.k8c.io/namespace
   

Upgrading KubeV

Run kdp connect again after every apply that upgrades KubeV. In this mode, apply doesn’t touch the api-syncagent, its rights, the admission policies or the published resource types, and a new release may change any of them. The second run:

• brings every object in What kdp connect writes to what the new version ships;
• deletes the PublishedResources the new version no longer publishes;
• restarts the api-syncagent only when its kubeconfig, its rights or what it publishes changed, or when the new release changes the agent itself.

Pass the same flags as before, --forward-connector included.

Troubleshooting

If KubeV may not write to your KDP

If you can’t give kdp connect an administrator’s kubeconfig, an administrator of your KDP can create the same objects by hand: everything in What kdp connect writes, with the admission policies before the PublishedResources. Get the manifests for your KubeV version from Kubermatic. apply prints the list of steps in this mode.

Removing KubeV from your KDP

First disable the KubeV service in every project in KDP, so that no project uses it any more. Then remove what kdp connect wrote. On the KubeV cluster:

export KUBECONFIG=kubev-cluster-kubeconfig
kubectl -n kubev-syncagent delete deployment kubev-api-syncagent
kubectl delete publishedresources.syncagent.kcp.io -l kdp.k8c.io/syncagent=kubev
kubectl delete namespace kubev-syncagent
kubectl delete clusterrolebinding,clusterrole kubev-syncagent:resources
kubectl delete crd publishedresources.syncagent.kcp.io
kubectl delete validatingadmissionpolicybinding kubev-kdp-kubevirt kubev-kdp-forklift --ignore-not-found
kubectl delete validatingadmissionpolicy kubev-kdp-kubevirt kubev-kdp-forklift --ignore-not-found
# Only if kdp connect created it: without it, the KubeV API server can't reach kcp.
kubectl -n kubermatic-virtualization delete secret kubev-kcp-front-proxy-ca
unset KUBECONFIG

In kcp, with the administrator’s kubeconfig:

KCP=https://kcp.example.com:8443
k() { kubectl --kubeconfig kcp-admin.kubeconfig --server "$KCP/clusters/root" "$@"; }
# The provider workspace, with the KubeV service and the admission policies in it.
k delete workspace kubev
# Only with kdp.admin or kdp.adminGroup: the platform administrators' grants in root.
k delete clusterrolebinding kubev:tenancy:workspaces-manager
k delete clusterrole kubev:tenancy:workspaces-manager kubev:tenancy:project-type-user

The copies of the projects’ objects stay on the KubeV cluster, in their host namespaces, until you delete them. kubectl get namespaces -l kdp.kubev.k8c.io/project-hash lists those namespaces. Keep kubev:kdp-core-binder in root: it helps every new organization and project in your KDP, not only the ones that use KubeV.