Working with Operating System Manager

Enable/Disable OSM

Starting with KubeOne 1.5, OSM will be enabled by default for new and existing clusters. This change will be applied upon running kubeone apply with KubeOne 1.5 or newer

To fallback to legacy user-data from Machine Controller, we can disable OSM for a KubeOne cluster. Although this is not recommended for production environments because the legacy user-data is considered deprecated and will be removed in a future release.

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: 1.23.6
addons:
  enable: true
operatingSystemManager:
  deploy: false

Migrating existing MachineDeployments

To perform this rotation please follow the guide at Rolling Restart MachineDeploments.

OSM doesn’t support using cloud-init as a provisioning utility for Flatcar machines. Flatcar use coreos-cloudinit instead of the upstream cloud-init. coreos-cloudinit is deprecated and its usage is not recommended in production environments.

After upgrading to KubeOne 1.5, when OSM is enabled, the newly created MachineDeployments will default to ignition as the provisioning utility unless cloud-init is specified explicitly in the MachineDeployment.

To migrate existing MachineDeployments, make sure that you either leave the provisioningUtility field blank or set it to ignition explicitly.

apiVersion: cluster.k8s.io/v1alpha1
kind: MachineDeployment
metadata:
  name: << MACHINE_NAME >>
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      name: << MACHINE_NAME >>
  template:
    metadata:
      labels:
        name: << MACHINE_NAME >>
    spec:
      providerSpec:
        value:
          cloudProvider: "aws"
          cloudProviderSpec:
            region: "eu-central-1"
            availabilityZone: "eu-central-1a"
            vpcId: "vpc-name"
            subnetId: "subnet-id"
            instanceType: "t2.micro"
            instanceProfile: "kubernetes-v1"
            isSpotInstance: false
            diskSize: 50
            diskType: "gp2"
            ebsVolumeEncrypted: false
            ami: "my-custom-ami"
          operatingSystem: flatcar
          operatingSystemSpec:
            # 'provisioningUtility` is only used for flatcar os, can be set to ignition or cloud-init. Defaults to ignition.
            provisioningUtility: ignition
      versions:
        kubelet: "<< KUBERNETES_VERSION >>"

Using custom OperatingSystemProfile

The annotation k8c.io/operating-system-profile is used at the MachineDeployment level to select the Operating System Profile that will be used to generate configurations for provisioning the machines.

When creating a new cluster using Terraform

For new clusters, a dedicated terraform variable initial_machinedeployment_operating_system_profile was introduced for this purpose. It’ll propagate the specified value to the initial MachineDeployments in the cluster.

It can be used as follows:

1. Create a custom OSP and place it in a folder inside your custom addons path, if it exists, otherwise create one. Let’s take this minimalistic OSP as an example:

apiVersion: operatingsystemmanager.k8c.io/v1alpha1
kind: OperatingSystemProfile
metadata:
  name: osp-install-curl
  namespace: kube-system
spec:
  osName: "ubuntu"
  osVersion: "20.04"
  version: "v1.0.0"
  supportedCloudProviders:
    - name: "aws"
  bootstrapConfig:
    files:
      - path: /opt/bin/bootstrap
        permissions: 755
        content:
          inline:
            encoding: b64
            data: |
              #!/bin/bash

              apt update && apt install -y curl jq              

      - path: /etc/systemd/system/bootstrap.service
        permissions: 644
        content:
          inline:
            encoding: b64
            data: |
              [Install]
              WantedBy=multi-user.target

              [Unit]
              Requires=network-online.target
              After=network-online.target
              [Service]
              Type=oneshot
              RemainAfterExit=true
              ExecStart=/opt/bin/bootstrap              

    modules:
      runcmd:
        - systemctl restart bootstrap.service

  provisioningConfig:
    files:
      - path: /opt/hello-world
        permissions: 644
        content:
          inline:
            encoding: b64
            data: echo "hello world"

NOTE: This is just for demonstration purposes and the custom OSPs would most likely be more complicated than this.

2. Create your infrastructure

terraform apply -var=initial_machinedeployment_operating_system_profile=osp-install-curl

The variable initial_machinedeployment_operating_system_profile can also be configured using terraform.tfvars.

3. Now create your cluster and make sure that you are using the custom addons folder.

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: "1.22.5"
cloudProvider:
  aws: {}
addons:
  enable: true
  path: "./custom-addons"

NOTE: The path for addons can be anything. But make sure that the custom OSP exists in that directory.

For new MachineDeployments

Create a new MachineDeployment and make sure that you add the k8c.io/operating-system-profile annotation with the required OSP name.

For an existing Cluster

For existing MachineDeployments; edit the MachineDeployment and add k8c.io/operating-system-profile annotation with the required OSP name.

To perform this rotation for existing MachineDeployments please follow the guide at Rolling Restart MachineDeploments.

Additional Information

• An update to OperatingSystemProfile will not result in an automatic rotation of the machines. This is an intentional design decision since an OperatingSystemProfile can be associated with multiple MachineDeployments. It is the user’s responsibility to rotate the machines when the OperatingSystemProfile is updated. To perform this rotation for existing MachineDeployments please follow the guide at Rolling Restart MachineDeploments.