Infrastructure Management

This document describes some possible approaches for managing the infrastructure needed for a Kubernetes cluster.

Infrastructure For Control Plane

It’s the user’s responsibility to create and manage the infrastructure for the control plane. In order to help with this, KubeOne integrates with Terraform by reading the information about the infrastructure from the Terraform state, and provides example Terraform configurations that can be used to get started. The example Terraform configs can be found in the KubeOne’s GitHub repository.

The example Terraform configurations are optimized for the CI and might not be production-ready out of the box. We advise checking the Production Recommendations document for more details about making the example configurations suitable for the production usage.

If you’re using our example Terraform configs, requirements are satisfied out of the box.

The following infrastructure requirements must be satisfied to successfully provision a Kubernetes cluster using KubeOne:

  • You need the appropriate number of instances dedicated for the control plane
    • You need odd number of instances with a minimum of three instances for the Highly-Available control plane
    • If you decide to use a single-node control plane instead, one instance is enough, however, highly-available control plane is highly advised, especially in the production environments
  • All control plane instances must satisfy the system requirements for a kubeadm cluster
    • Minimum 2 vCPUs
    • Minimum 2 GB RAM
    • Operating system must be a officially-supported by KubeOne (Ubuntu, Debian, CentOS, RHEL, Flatcar Linux)
    • Full network connectivity between all machines in the cluster (private network is recommended, but public is supported as well)
    • Unique hostname, MAC address, and product_uuid for every node
      • You can get the MAC address of the network interfaces using the command ip link or ifconfig -a
      • The product_uuid can be checked by using the command sudo cat /sys/class/dmi/id/product_uuid
    • Swap disabled. You MUST disable swap in order for the kubelet to work properly.
    • The following ports open: 6443, 2379, 2380, 10250, 10251, 10252
  • For highly-available control plane, a load balancer pointing to the control plane instances (the Kubernetes API server) is required
    • Load balancer must include all control plane instances and distribute traffic to the TCP port 6443 (default port of the Kubernetes API server)
    • It’s recommended to use a provider’s offering for load balancers if such is available
    • If provider doesn’t offer load balancer, you can create an instance and setup a solution such as HAProxy
    • Check out the Load Balancer for Highly-Available Cluster example to learn more about possible setups
  • You must have an SSH key deployed on all control plane instances and SSH configured as described in the Configuring SSH document

Depending on the environment, you may need additional objects, such as VPCs, firewall rules, or images. For officially-supported providers, we recommended checking our example Terraform configs as a reference what objects you should be created. In addition, options for multi-zone deployments are shown for selected providers.

Infrastructure For Worker Nodes

Instances for worker nodes can be managed in two ways:

  • using Kubermatic machine-controller, which creates and provisions instances, and joins them a cluster, automatically
  • using KubeOne Static Workers, by using the preferred tooling to create instances (e.g. Terraform) and then provision them using KubeOne

Using Kubermatic machine-controller is highly advised if your provider is officially-supported. Otherwise, KubeOne Static Workers are recommended instead. More details about the machine-controller and the Cluster-API can be found in the Concepts document.

The requirements for the worker instances are similar as for the control plane instances:

  • All instances must satisfy the system requirements for a kubeadm cluster
    • Minimum 2 vCPUs
    • Minimum 2 GB RAM
    • Operating system must be a officially-supported by KubeOne (Ubuntu, Debian, CentOS, RHEL, Flatcar Linux)
    • Full network connectivity between all machines in the cluster (private network is recommended, but public is supported as well)
    • Unique hostname, MAC address, and product_uuid for every node
      • You can get the MAC address of the network interfaces using the command ip link or ifconfig -a
      • The product_uuid can be checked by using the command sudo cat /sys/class/dmi/id/product_uuid
    • Swap disabled. You MUST disable swap in order for the kubelet to work properly.
    • The following ports open: 10250, and optionally 30000-32767 for NodePort Services
  • You must have an SSH key deployed on all control plane instances and SSH configured as described in the Configuring SSH document

Cloud Provider-specific Requirements

Some providers have specific requirements for Kubernetes to work properly. Those are set automatically by the machine-controller or the provided Terraform examples, therefore this section is only relevant when machines are created manually.

VMware vSphere

For the vSphere CSI driver to work correctly, machines created in vSphere need to meet the CSI driver prerequisites.

Make sure that all VMs have the disk.enableUUID flag set to 1. This can be done using the govc tool with the following command:

govc vm.change -e="disk.enableUUID=1" -vm='/PATH/TO/VM'

The vSphere CSI driver requires that the Virtual Machine Hardware Version is 15 or newer, and the VMware Tools are installed.

Terraform Integration

KubeOne integrates with Terraform by reading the Terraform state for the information about the cluster, including:

  • the Kubernetes API server load balancer endpoint
  • nodes’ public and private IP addresses, and hostnames
  • SSH parameters (username, port, key)
  • bastion/jump host parameters if the bastion host is used
  • information needed to generate the MachineDeployment objects which define worker nodes

To use the integration, you need to generate a Terraform state file using the terraform output -json command. KubeOne consumes the generated Terraform state file and reads the needed information. Therefore, the generated file must strictly follow the format used by KubeOne. To accomplish this, you must have the appropriate output.tf file co-located with other Terraform files. The output.tf file defines the template for generating the state file, including where to look for the information about the infrastructure.

For more information about how the output.tf file should look like, you can check our example Terraform configs and the Terraform Integration Reference.

The needed output.tf file already comes with all our example Terraform configs.