DNS Management

Enterprise Edition

Setup

Install External-dns

We leverage External-dns to manage DNS records for the tenant clusters.

This is just an example to give you a headstart. For more details on setting up external-dns for different providers, visit Official Documentation.

Update the values.yaml for KubeLB manager chart to enable the external-dns addon.

kubelb-addons:
  enabled: true

  external-dns:
    enabled: true
    domainFilters:
      - example.com
    extraVolumes:
      - name: credentials
        secret:
          secretName: route53-credentials
    extraVolumeMounts:
      - name: credentials
        mountPath: /.aws
        readOnly: true
    env:
      - name: AWS_SHARED_CREDENTIALS_FILE
        value: /.aws/credentials
    txtOwnerId: kubelb-example-aws
    registry: txt
    provider: aws
    policy: sync
    sources:
      - service
      - ingress
      # Comment out the below resources if you are not using Gateway API.
      - gateway-httproute
      - gateway-grpcroute
      - gateway-tlsroute
      - gateway-tcproute
      - gateway-udproute

Credentials secret

apiVersion: v1
kind: Namespace
metadata:
  name: external-dns
---
apiVersion: v1
data:
  credentials: W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gTk9UVEhBVERVTUIKYXdzX3NlY3JldF9hY2Nlc3Nfa2V5ID0gTUFZQkVJVFNBU0VDUkVU
kind: Secret
metadata:
  name: route53-credentials
  namespace: external-dns
type: Opaque

Enable DNS automation

DNS can be enabled/disabled at global or tenant level. For automation purposes, you can configure allowed domains for DNS per tenant.

apiVersion: kubelb.k8c.io/v1alpha1
kind: Tenant
metadata:
  name: shroud
spec:
  # These domains are allowed to be used for Ingress, Gateway API, DNS, and certs.
  allowedDomains:
    - "kube.example.com"
    - "*.kube.example.com"
    - "*.shroud.example.com"
  dns:
    # If not empty, only the domains specified here will have automation for DNS. Everything else will be ignored.
    allowedDomains:
    - "*.shroud.example.com"
    # The wildcard domain to use for auto-generated hostnames for Load balancers
    # In EE Edition, this is also use to generated dynamic hostnames for tunnels.
    wildcardDomain: "*.apps.example.com"
    # Allow tenants to specify explicit hostnames for Load balancers and tunnels(in EE Edition)
    allowExplicitHostnames: false
  gatewayAPI:
    class: "eg"
    defaultGateway:
      name: "default"
      namespace: "kubelb"

Users can then either use external-dns annotations or the annotation kubelb.k8c.io/manage-dns: true on their resources to automate DNS management.

The additional validation at the tenant level allows us to use a single instance of external-dns for multiple tenants. Although, if required, external-dns can be installed per tenant as well.

Configure Gateway

Gateway resource needs to be configured for this automation to work. For example, if you are using Gateway API, you can configure the Gateway resource to manage DNS as follows:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: default
  namespace: kubelb
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
spec:
  gatewayClassName: eg
  listeners:
    ## HTTP listener to solve DNS challenge for cert-manager
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - protocol: HTTPS
      port: 443
      name: https
      hostname: "*.apps.example.com"
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: eg-https
    # Required in EE for tunneling
    - protocol: HTTPS
      port: 443
      name: https-connection-manager
      hostname: "connection-manager.example.com"
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: eg-https-connection-manager

Usage

Using external-dns annotations:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: example
  annotations:
    external-dns.alpha.kubernetes.io/hostname: example.com
spec:
  gatewayClassName: kubelb
  listeners:
    - name: http
      hostname: example.com
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: All

Delegate DNS management to KubeLB:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: example
  annotations:
    kubelb.k8c.io/manage-dns: true
spec:
  gatewayClassName: kubelb
  listeners:
    - name: http
      hostname: example.com
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: All

Services can also be annotated to manage DNS:

apiVersion: v1
kind: Service
metadata:
  name: backend
  annotations:
    external-dns.alpha.kubernetes.io/hostname: backend.example.com
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: backend
  type: LoadBalancer