Addons

Addons are a mechanism used to deploy Kubernetes resources after provisioning the cluster. Addons allow operators to use KubeOne to deploy various components such as CNI and CCM, and various stacks such as logging and monitoring, backups and recovery, log rotating, and more.

This document explains how to use addons in your workflow.

Writing Addons

Addons are represented as Kubernetes YAML manifests. To deploy an addon, the operator needs to put a YAML manifest in a directory and provide it as the addons directory in the KubeOne cluster configuration.

Templating

Manifests support templating based on Go templates. The following data is available out of the box:

KubeOne cluster configuration - .Config
Credentials - .Credentials

On top of that, you can use the sprig functions in your templates. For list of available functions, consider the sprig docs.

Example

The following snippet shows how an addon looks like and how to use templating:

apiVersion: v1
kind: Namespace
metadata:
  name: example-{{ .Config.Name }} # will be rendered as 'example-cluster_name'
---
apiVersion: v1
kind: Secret
metadata:
  name: credentials
  namespace: kube-system
type: Opaque
data:
  AWS_ACCESS_KEY_ID: {{ .Credentials.AWS_ACCESS_KEY_ID | b64enc }} # will be rendered as base64-encoded AWS access key
  AWS_SECRET_ACCESS_KEY: {{ .Credentials.AWS_SECRET_ACCESS_KEY | b64enc }} # will be rendered as base64-encoded AWS secret access key

Note: The b64enc function is a sprig function.

Addons API Reference

Take a look at the KubeOne’s autogenerated API reference for more details about available options in the Addons API.

Enabling Addons

To enable addons, you need to modify the KubeOne cluster configuration to add the addons config:

apiVersion: kubeone.io/v1beta1
kind: KubeOneCluster
versions:
  kubernetes: 1.20.1
cloudProvider:
  aws: {}
# Addons are Kubernetes manifests to be deployed after provisioning the cluster
addons:
  enable: true
  # In case when the relative path is provided, the path is relative
  # to the KubeOne configuration file.
  path: "./addons"

The addons path is normalized on the runtime. If you provide a relative path, the path is relative to the KubeOne configuration file. This means that ./addons will be parsed depending on the kubeone command you use:

kubeone install -m config.yaml - ./addons
kubeone install -m other/dir/config.yaml - ./other/dir/addons/config.yaml

Addons can be organized into subdirectories, but only one level of subdirectories is supported. For example, ./addons/example-addon-1 is supported and YAML manifest in that directory will be deployed, but ./addons/example-addon-1/subdirectory-2 directory will be entirely ignored.

Embedded Addons

Since KubeOne v1.3 release, there’s a new kind of addons that are always carried with the binary itself – embedded addons. This is done thanks to the new Go’s embed package.

Some of those addons are directly and automatically used by the KubeOne itself (for example machinecontroller), but it’s not the case for all of them. Some very useful addons can be activated by the user to be deployed to the cluster, like:

Activate Embedded Eddons

To activate the embedded addons, the user needs to use the new Addons API.

The addons directory must exist even if you plan to only use embedded addons without providing your custom addons. This is a known bug and can be tracked on GitHub.

Example:

apiVersion: kubeone.io/v1beta1
kind: KubeOneCluster
versions:
  kubernetes: 1.20.1

addons:
  enable: true
  path: "./addons"
  addons:
  - name: cluster-autoscaler
  - name: unattended-upgrades

Up to date (and possibly unreleased yet) list of addons.

Overriding Embedded Eddons

Some of those embedded addons used to be written in Go structures form. Now KubeOne has them in YAML form so those addons are easier to maintain and to let users to override them.

For example, if you wish to have your own machine-controller manifests being deployed to the cluster, you can override the machinecontroller addon. Enable addons like described above and in the addons directory place your machinecontroller manifests in the same directory name, e.g. ./addons/machinecontroller/yourmanifest.yaml.

Cleanup Addons

To delete embedded addon from the cluster, use the new delete field from the Addons API.

apiVersion: kubeone.io/v1beta1
kind: KubeOneCluster
versions:
  kubernetes: 1.20.1

addons:
  enable: true
  addons:
  - name: unattended-upgrades
    delete: true

That will cleanup all the traced of unattended-upgrades addon, but only it’s manifests, not the contents that addons workloads might have produced, i.e. backups themselves.

delete field works only on embedded addons for now.

Parameters

It’s possible to pass down user defined parameters (in key/value form) down to an addon. There are two fields (of type map[string]string):

.addons.globalParams
.addons.addons[] .params

.addons.addons[] .params has higher priority over .addons.globalParams, so you can use it to override globally defined parameters.

apiVersion: kubeone.io/v1beta1
kind: KubeOneCluster
versions:
  kubernetes: 1.20.1

addons:
  enable: true
  globalParams:
    key1: value1
  addons:
  - name: unattended-upgrades
    params:
      key2: value2

Reconciling Addons

The addons are reconciled after initializing and joining the control plane nodes nodes when running kubeone install, kubeone upgrade, or kubeone apply. You can also reconcile addons after the cluster is provisioned by using kubeone apply.

kubeone apply --manifest kubeone.yaml -t .

The reconciliation is done using kubectl over SSH, using a command such as:

kubectl apply -f addons.yaml --prune -l kubeone.io/addon

Using the --prune options means that the next time you run kubeone:

if you updated any manifest, the corresponding resources in the cluster will be updated,
if you removed a resource from a manifest, the resource will be removed from the cluster as well
if you removed a whole manifest, all resources defined in that manifest will be removed from the cluster

The --prune option can be dangerous. Always make sure that you have all needed manifests present in the addons directory and correct addons configuration before running kubeone.

The addons are applied in the alphabetical order. This means that you can control in which order addons will be applied by setting the appropriate file name.

Example Addons

We provide the example addons that you can use as a template or to handle various tasks, such as cluster backups. You can find the example addons in the addons directory.