Securing System Services

Access to Prometheus, Grafana and all other system services included with Kubermatic Kubernetes Platform (KKP) is secured by running them behind OAuth2-Proxy and using Dex as the authentication provider.

Configuration

Dex can then be configured to use external authentication sources like GitHub’s or Google’s OAuth endpoint, LDAP or OpenID Connect. For this to work you have to configure both Dex (the dex Helm chart) and OAuth2-Proxy (called “IAP”, Identity-Aware Proxy) in your Helm values.yaml.

Dex

For each service that is supposed to use Dex as an authentication provider, configure a client. The callback URL is called after authentication has been completed and must point to https://<domain>/oauth/callback. Remember that this will point to OAuth2-Proxy and is therefore independent of the actual underlying application (Gatekeeper will intercept the request and not forward it to the upstream service). Generate a secure random secret for each client, for example by doing cat /dev/urandom | tr -dc A-Za-z0-9 | head -c32.

A sample configuration for Prometheus and Alertmanager could look like this:

httpRoute:
  domain: kkp.example.com
  path: /dex
  pathType: PathPrefix

dex:
  config:
    staticClients:
    # keep the KKP client for the login to the KKP dashboard
    - id: kubermaticIssuer
      # ...

    # new client used for authenticating Prometheus
    - id: prometheus # a unique identifier
      name: Prometheus
      secret: <generate random secret key here>
      RedirectURIs:
      - 'https://prometheus.kkp.example.com/oauth/callback'

    # new client used for authenticating Alertmanager
    - id: alertmanager # a unique identifier
      name: Alertmanager
      secret: <generate another random secret key here>
      RedirectURIs:
      - 'https://alertmanager.kkp.example.com/oauth/callback'

Each service should have its own credentials (i.e. a different secret for every client). Re-deploying the dex chart with Helm will now prepare Dex to act as an authentication provider, but there is no OAuth2-Proxy yet to make use of this:

Helm 3

helm --namespace dex upgrade --install --wait --values /path/to/your/helm-values.yaml dex charts/dex/

OAuth2-Proxy (IAP)

Now that you have setup Dex, you need to configure OAuth2-Proxy to sit in front of the system services and use it for authentication. The configuration for this happens in the iap Helm chart. For each client that we configured in Dex, add a deployment to the IAP configuration. Use the client’s secret (from Dex) as the client_secret and generate another random, secure encryption key to encrypt the client state with (which is then stored as a cookie in the user’s browser).

Extend your values.yaml with a section for the IAP:

iap:
  deployments:
    prometheus:
      # will be used to create kubernetes Deployment object
      name: prometheus

      ingress:
        host: prometheus.kkp.example.com

      # the Kubernetes service and port the IAP should point to
      upstream_service: prometheus.monitoring.svc.cluster.local
      upstream_port: 9090

      # OAuth configuration from Dex
      # client_id is the "id" from the Dex config
      client_id: prometheus
      # client_secret is the "secret" from the Dex config
      client_secret: <copy value from Dex>

      # generate a fresh secret key here
      encryption_key: <generate random secret key here>

      # see https://github.com/oauth2-proxy/oauth2-proxy/blob/master/docs/configuration/configuration.md
      # this example configures that only users belong to a special GitHub
      # organization can access the service behind the proxy (Prometheus in this case)
      config:
        scope: "groups openid email"
        github_org: mygithuborg
        github_team: mygroup

    alertmanager:
      name: alertmanager
      ingress:
        host: alertmanager.kkp.example.com
      upstream_service: alertmanager.monitoring.svc.cluster.local
      upstream_port: 9093
      client_id: alertmanager
      client_secret: <copy value from Dex>
      encryption_key: <generate another random secret key here>
      config:
        scope: "groups openid email"
        github_org: mygithuborg
        github_team: mygroup

The KKP installer deploys the iap chart together with the monitoring stack: release iap in namespace monitoring on the master or seed, and for User Cluster MLA release iap in namespace mla. The installer labels these namespaces for Gateway access automatically. If you prefer to manage the chart yourself, install it manually. Using namespace monitoring and release name iap matches what the installer would create:

Helm 3

helm --namespace monitoring upgrade --install --wait --values /path/to/your/helm-values.yaml iap charts/iap/

This will create one HTTPRoute per deployment you configured, attached to the KKP Gateway (kubermatic in the kubermatic namespace by default; configure the top-level httpRoute.gatewayName and httpRoute.gatewayNamespace values to point elsewhere, for example at a seed Gateway). Each deployment’s ingress.host value is reused as the HTTPRoute hostname. If all your hosts are subdomains of your primary domain, the wildcard DNS record we already set up earlier will be enough. Otherwise you will need to update your DNS accordingly.

The Gateway accepts HTTPRoutes only from namespaces labeled kubermatic.io/gateway-access=true. The KKP installer labels the required namespaces automatically; when you install the iap chart manually with Helm as shown above, make sure the namespace carries the label:

kubectl label namespace monitoring kubermatic.io/gateway-access=true --overwrite

TLS for each IAP hostname is provided by the shared Gateway listener: the HTTPRoute-Gateway sync controller adds a listener per hostname and cert-manager issues the certificates based on the Gateway’s issuer annotation. Once you setup the required DNS records (see next steps) you can check that the HTTPRoutes are accepted, that the sync controller added listeners for the IAP hostnames, and that the certificates were issued:

watch kubectl -n monitoring get httproute
kubectl -n kubermatic get gateway kubermatic -o jsonpath='{.spec.listeners[*].hostname}'
kubectl get certificates -A

DNS Records

To allow incoming traffic and to acquire a TLS certificate, DNS records must be in place. This can be either a single wildcard entry for all IAP deployments or individual records. Refer to the installation instructions for more information on what records to create.

Alternative Authentication Provider

It’s possible to use a different authentication provider than Dex. Please refer to the OIDC provider chapter for more information on how to configure KKP and OAuth2-Proxy accordingly.

Alternatively, if you want to configure a different or additional JWT identity provider for your user cluster API server(s), you can define an AuthenticationConfiguration file within a Kubernetes Secret and refer to it within the Seed’s spec.authenticationConfiguration or, to configure it per datacenter, within the Seed’s spec.datacenters.spec.authenticationConfiguration or within the Cluster’s spec.authenticationConfiguration directly. The AuthenticationConfiguration precedence order is as follows: 1. Cluster, 2. Datacenter, 3. Seed.

Security Considerations

The IAP does not protect services against access from within the cluster. Sensitive services should therefore be configured to require further authentication. Grafana, the Master / Seed Monitoring, Logging & Alerting Stack’s dashboard UI, requires proper authentication by default, for example.