Backups Addon

The backups addon can be used to backup the most important parts of a cluster, including:

  • etcd
  • etcd PKI (certificates and keys used by Kubernetes to access the etcd cluster)
  • Kubernetes PKI (certificates and keys used by Kubernetes and clients)

The addon uses Restic to upload backups, encrypt them, and handle backup rotation.

By default, backups are done every 30 minutes and are kept for 48 hours. If you need renention, please adjust the restic CLI flags restic forget --prune --keep-last <NEW AMOUNT OF HOURS>.

Prerequisites

In order to use this addon, you need an S3 bucket or Restic-compatible repository for storing backups.

Using The Addon

Original addon source can be found in kubeone repository.

apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
  namespace: kube-system
type: Opaque
data:
  AWS_ACCESS_KEY_ID: {{ .Credentials.AWS_ACCESS_KEY_ID | b64enc }}
  AWS_SECRET_ACCESS_KEY: {{ .Credentials.AWS_SECRET_ACCESS_KEY | b64enc }}
---
apiVersion: v1
kind: Secret
metadata:
  name: restic-config
  namespace: kube-system
type: Opaque
data:
  password: {{ "<<RESTIC_PASSWORD>>" | b64enc }}
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: etcd-s3-backup
  namespace: kube-system
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  schedule: '@every 30m'
  successfulJobsHistoryLimit: 0
  suspend: false
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          nodeSelector:
            node-role.kubernetes.io/master: ""
          tolerations:
          - key: node-role.kubernetes.io/master
            effect: NoSchedule
            operator: Exists
          restartPolicy: OnFailure
          volumes:
          - name: etcd-backup
            emptyDir: {}
          - name: host-pki
            hostPath:
              path: /etc/kubernetes/pki
          initContainers:
          - name: snapshoter
            image: gcr.io/etcd-development/etcd:v3.4.3
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - |-
              set -euf
              mkdir -p /backup/pki/kubernetes
              mkdir -p /backup/pki/etcd
              cp -a /etc/kubernetes/pki/etcd/ca.crt /backup/pki/etcd/
              cp -a /etc/kubernetes/pki/etcd/ca.key /backup/pki/etcd/
              cp -a /etc/kubernetes/pki/ca.crt /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/ca.key /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/front-proxy-ca.crt /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/front-proxy-ca.key /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/sa.key /backup/pki/kubernetes
              cp -a /etc/kubernetes/pki/sa.pub /backup/pki/kubernetes
              etcdctl snapshot save /backup/etcd-snapshot.db
            env:
            - name: ETCDCTL_API
              value: "3"
            - name: ETCDCTL_DIAL_TIMEOUT
              value: 3s
            - name: ETCDCTL_CACERT
              value: /etc/kubernetes/pki/etcd/ca.crt
            - name: ETCDCTL_CERT
              value: /etc/kubernetes/pki/etcd/healthcheck-client.crt
            - name: ETCDCTL_KEY
              value: /etc/kubernetes/pki/etcd/healthcheck-client.key
            - name: ETCD_HOSTNAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            volumeMounts:
            - mountPath: /backup
              name: etcd-backup
            - mountPath: /etc/kubernetes/pki
              name: host-pki
              readOnly: true
          containers:
          - name: uploader
            image: docker.io/restic/restic:0.9.6
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - |-
              set -euf
              restic snapshots -q || restic init -q
              restic backup --tag=etcd --host=${ETCD_HOSTNAME} /backup
              restic forget --prune --keep-last 48
            env:
            - name: ETCD_HOSTNAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: RESTIC_REPOSITORY
              value: "s3:s3.amazonaws.com/<<S3_BUCKET>>"
            - name: RESTIC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: restic-config
                  key: password
            - name: AWS_DEFAULT_REGION
              value: "<<AWS_DEFAULT_REGION>>"
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: AWS_ACCESS_KEY_ID
                  name: s3-credentials
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: AWS_SECRET_ACCESS_KEY
                  name: s3-credentials
            volumeMounts:
            - mountPath: /backup
              name: etcd-backup

You need to replace the following values with the actual ones:

  • <<RESTIC_PASSWORD>> - a password used to encrypt the backups
  • <<S3_BUCKET>> - the name of the S3 bucket to be used for backups
  • <<AWS_DEFAULT_REGION>> - default AWS region

Credentials are fetched automatically if you are deploying on AWS. If you want to use non-default credentials or you’re not deploying on AWS, update the s3-credentials secret (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY keys).