Starting with version v2.17, KKP introduced experimental support for new etcd Backup and Restore controller based on the etcd-launcher experimental feature.
The legacy backup controller is based on a simple cron job and didn’t support automated restore operations. The new experimental controllers utilize new CRDs for backups and restore, support multiple backup configurations per cluster, immediate backups and automated restore operations.
The new controllers try to be as backward compatible as possible with the legacy controller. However, it’s not possible to manage or restore legacy backups with the new controllers.
The new controllers manage backup, restore and cleanup operations using Kubernetes Jobs. All jobs triggered use containers that can be passed to the seed controller manager similar to the legacy backup controller.
Currently, only S3 compatible backup backends are supported.
To use the new backup/restore controller, the etcd-launcher experimental feature must be enabled along with specifically enabling the controllers by passing the flag
--enable-etcd-backups-restores to the seed controller manager. This can be achieved by setting the following values in the KubermaticConfiguration:
spec: # SeedController configures the seed-controller-manager. seedController: # BackupRestore contains the setup of the new backup and restore controllers. backupRestore: # Enabled enables the new etcd backup and restore controllers. enabled: true # S3BucketName is the S3 bucket name to use for backup and restore. s3BucketName: "" # S3Endpoint is the S3 API endpoint to use for backup and restore. Defaults to s3.amazonaws.com. s3Endpoint: "" # BackupDeleteContainer is the container used for deleting etcd snapshots from a backup location. backupDeleteContainer: ""
The new Backup controller is designed to be a drop-in replacement for the legacy backup controller. To achieve this, the legacy
backupCleanupContainer are supported by the new controller. However, it’s best to use the new containers to enable the full functionality of the new controller.
backupStoreContainer: it’s recommended to update the seed controller configuration to use the new backup container. The new container spec is provided in the
chartsdirectory shipped with the KKP release:
backupDeleteContainer: backup deletions are performed using a customizable container that is passed to the seed controller via new optional argument
backupDeleteContainer. If this option is not set, the controller will not perform deletions and the legacy
backupCleanupContainerwill be used instead. A container spec is provided in the
chartsdirectory shipped with the KKP release:
You can’t have both
backupDeleteContainer will take precedence when the new controller is enabled.
The new containers expect that the S3 backend is SSL enabled. Unfortunately, the released minio chart doesn’t support that by default, and will need to be modified to run over minio over SSL.
The new controllers will use the credentials from the secret
kube-system/s3-credentials similar to the legacy backup controller.
However, when using the new containers, creating an additional ConfigMap with the S3 backend settings is required:
apiVersion: v1 kind: ConfigMap metadata: name: s3-settings namespace: kube-system data: BUCKET_NAME: ENDPOINT:
By default a new backup configuration resource is created in the cluster namespace for each user cluster once the controllers are enabled. The new backup configuration resource is created with a default of
@every 20m interval and 20 backup retention.
However, it’s possible to create additional backup configuration if needed, simply by creating a backup configuration resource:
apiVersion: kubermatic.k8s.io/v1 kind: EtcdBackupConfig metadata: name: daily-backup namespace: cluster-zvc78xnnz7 spec: cluster: apiVersion: kubermatic.k8s.io/v1 kind: Cluster name: zvc78xnnz7 schedule: '0 1 * * *' keep: 40
Once created, the controller will start a new backup every time the schedule demands it (i.e. at 1:00 AM in the example), and delete backups once more than
.spec.keep (40 in the example) have been created. The created backups are named based on the configuration name and the backup timestamp
Backup creations and deletions are done by spawning Kubernetes jobs. For each backup, a job is created with an init container that creates a snapshot and a main container that will run the customizable store container. It will receive the cluster name and backup name in two environment variables. The store container should upload the snapshot to an S3 compatible backend and return exit code 0 if successful.
When a backup needs to be deleted, the controller starts another job with the delete container configured in the seed controller manager. The delete container receives the cluster and backup names in environment variables and must delete the backup at the place where it was uploaded by the create job earlier, and exit 0 if successful.
The controller maintains list of backups related to this configuration in the resource status spec, where it tracks the creation and deletion status of all backups that haven’t been successfully deleted yet.
It’s also possible to trigger a one-time backup if needed. Simply by creating a backup configuration that doesn’t have a
schedule set in the spec. The backup controller will immediately create a backup job for the user cluster and will not attempt to run it again or delete it later.
The Backup controller tracks all backups related to each configuration in a list in the backup configuration resource status spec. The status spec shows details about the backup status, cleanup status and the jobs triggered to run it and clean it up:
status: lastBackups: - backupFinishedTime: "2021-03-16T01:48:22Z" backupName: zvc78xnnz7-jobtest1-2021-03-16T03:48:00 backupPhase: Completed deleteFinishedTime: "2021-03-16T02:01:43Z" deleteJobName: zvc78xnnz7-backup-jobtest1-delete-s8sb8mdsp5 deletePhase: Completed jobName: zvc78xnnz7-backup-jobtest1-create-rjjq9d4k6d scheduledTime: "2021-03-16T01:48:00Z" - backupFinishedTime: "2021-03-16T01:51:25Z" backupName: zvc78xnnz7-jobtest1-2021-03-16T03:51:00 backupPhase: Completed deleteFinishedTime: "2021-03-16T02:04:08Z" deleteJobName: zvc78xnnz7-backup-jobtest1-delete-wcwvc8nmdt deletePhase: Completed jobName: zvc78xnnz7-backup-jobtest1-create-96tblznngk scheduledTime: "2021-03-16T01:51:00Z" - backupFinishedTime: "2021-03-16T01:54:21Z" backupName: zvc78xnnz7-jobtest1-2021-03-16T03:54:00 backupPhase: Completed deleteFinishedTime: "2021-03-16T02:08:24Z" deleteJobName: zvc78xnnz7-backup-jobtest1-delete-cgmqhlw5fw deletePhase: Completed jobName: zvc78xnnz7-backup-jobtest1-create-v2gz2ntn46 scheduledTime: "2021-03-16T01:54:00Z"
To restore a cluster from am existing backup, you simply create a restore resource in the cluster namespace:
apiVersion: kubermatic.k8s.io/v1 kind: EtcdRestore metadata: name: cluster-restore namespace: cluster-zvc78xnnz7 spec: cluster: apiVersion: kubermatic.k8s.io/v1 kind: Cluster name: zvc78xnnz7 backupName: daily-backup-2021-03-02T15:24:00
Once this resource is created, the controller will reconcile it and apply the following restore procedure:
EtcdClusterInitializedcluster condition to false.
kube-system/s3-credentialssecret and the information from the
EtcdClusterInitializedcluster condition to true. At this point the
Currently, The restore controller only support S3 compatible backends for downloading backups.