Ceph Storage

This guide sets up Ceph as storage for VM disks, on its own or next to Longhorn, and covers its day-2 operations. KubeV runs Ceph with Rook, on empty disks of the cluster’s own nodes.

KubeV sets Ceph up for block storage. It installs:

• the ceph-block StorageClass, which becomes the cluster’s default StorageClass;
• a ceph-block VolumeSnapshotClass, so VM backups and snapshots work on Ceph;
• the Ceph toolbox, for the ceph commands below.

Ceph spreads every VM disk over all the disks of the cluster and keeps copies of it on several nodes. VM disks on Ceph live-migrate, and a disk cloned from another disk on Ceph is created almost at once, whatever its size.

Before You Start

Disks and Partitions

Ceph stores data only on block devices that hold nothing else: an empty disk, an empty partition, or an LVM logical volume without a file system. Unlike Longhorn, it cannot keep its data in a directory of the system disk, so every node that is to store Ceph data needs one of these. A node without one still runs Ceph, but stores no data.

• A disk must be empty: no partitions and no file system. A partition must have no file system. Ceph never takes a device that has a file system, and it formats the devices it takes.
• Each disk or partition must be at least 5 GiB. Use SSD or NVMe: Ceph on hard disks is slow for VMs.
• For deviceFilter, disks and partitions must have names a regular expression can tell apart from the others, such as nvme1n1 and nvme2n1 for the data disks next to the system disk nvme0n1. Otherwise, list each node’s devices on its host, see Choosing Devices per Node.

List a node’s disks and partitions with lsblk -f. A disk Ceph can take shows no FSTYPE and no partitions below it, and a partition Ceph can take shows no FSTYPE. To clear a disk that was used before, and whose data you no longer need:

sudo wipefs --all /dev/nvme1n1
sudo sgdisk --zap-all /dev/nvme1n1

Nodes with a Single Disk

A node whose only disk holds the operating system can give Ceph a partition of it, if the disk has unallocated space. Leave space free when you install the operating system, then create a partition in that space, without a file system:

# Show the free space on the disk
sudo parted /dev/sda unit GiB print free

# Create a partition from the start of the free space (here 200 GiB) to the end of the disk
sudo parted /dev/sda mkpart primary 200GiB 100%

# The new partition, for example sda2, shows no FSTYPE
lsblk -f /dev/sda

Do not create a file system on it.

A system disk installed with LVM, as Ubuntu Server does by default, often keeps its free space in the volume group rather than on the disk. Give Ceph a logical volume from it, without a file system, and list the volume in the host’s cephDevices: no filter matches a logical volume.

# Show the free space in the volume groups
sudo vgs

# Create a logical volume of 200 GiB for Ceph in the volume group ubuntu-vg
sudo lvcreate --name ceph --size 200G ubuntu-vg

    - address: 10.0.0.11
      cephDevices: ["/dev/ubuntu-vg/ceph"]

A system disk whose space is all in use, with no free space in a volume group either, cannot give Ceph anything: shrinking the root file system is risky, and XFS cannot be shrunk at all. Add a disk to such nodes instead.

Ceph on the system disk shares the disk with the operating system, and with Longhorn when it runs too. That suits test clusters and the move from Longhorn to Ceph. While a VM disk moves from Longhorn to Ceph, it needs room on both.

Nodes

Ceph keeps each copy of the data on a different node, and keeps 3 copies by default. Production clusters need at least 3 nodes with matching disks. A cluster of fewer nodes must set replicas to 1 or 2, which suits test clusters only: a failed node can then take data with it. With fewer than 3 nodes, Ceph also runs a single monitor, and every Ceph volume goes offline when its node fails.

Ceph runs on every node, the control plane included. A node without a matching disk runs Ceph’s monitors and managers, but stores no data.

Resources

Ceph’s daemons run next to the VMs and take their share of the nodes:

Network

Ceph uses the nodes’ own network, not the Kube-OVN overlay. Every write is copied to as many nodes as Ceph keeps copies, so give the nodes at least 10 GbE.

Enabling Ceph

Add ceph to the storage section of the configuration file, with a filter that matches the data disks:

storage:
  ceph:
    deviceFilter: "^nvme[1-9]n1$"

deviceFilter is matched against the disk and partition names lsblk shows, without /dev/. The filter is the same for every node, and each node matches it against its own disks and partitions, so one filter can name different devices on different nodes. Some examples:

Ceph takes every empty disk and partition that matches, on every node. Keep the filter as narrow as the names allow, so that it does not also match an empty device on another node that you meant to keep for something else. When no filter can tell the devices apart, list them per host, see Choosing Devices per Node.

Then apply the configuration:

kubermatic-virtualization apply -f cluster.yaml

After installing Ceph, the installer waits until Ceph serves storage: its pool is ready, and Ceph stores data on as many nodes as it keeps copies. This takes a few minutes. When Ceph cannot take enough disks, apply stops after 20 minutes and says how many nodes Ceph found disks on.

See Storage in the Configuration Guide for all the fields.

Choosing Devices per Node

A host can list the devices Ceph takes on its node, in cephDevices next to its address, instead of the filter. That is how to take different devices on different nodes, such as sdc on one node and sdb on another while the first node’s own sdb stays out of Ceph, and the only way to give Ceph an LVM logical volume.

controlPlane:
  hosts:
    - address: 10.0.0.10
staticWorkers:
  hosts:
    - address: 10.0.0.11
      cephDevices: ["sdc"]
    - address: 10.0.0.12
      cephDevices: ["/dev/disk/by-id/wwn-0x5000c500a1b2c3d4"]
    - address: 10.0.0.13
storage:
  ceph:
    deviceFilter: "^nvme1n1$"   # for the hosts that list no devices; can be left out

• Name a device by its kernel name, such as sdc or nvme1n1p3, or by a path: a path under /dev/disk/by-id/ or /dev/disk/by-path/ for a disk or partition, or /dev/<volume group>/<logical volume> for a logical volume. Kernel names such as sdb can change between reboots, while a /dev/disk/by-id/ path always names the same disk. ls -l /dev/disk/by-id/ shows a node’s paths and the kernel names they point to.
• Ceph takes a listed device only when it is empty, as it does a matched one.
• Without deviceFilter, only the hosts that list devices store data, so replicas can be at most their number.
• You never name the nodes. The installer finds the node of each host and gives Rook the name Rook knows it by. A node that has not joined the cluster stops apply with an error naming its host.
• Once a host lists devices, Ceph takes devices only on the nodes the table gives some to, and on no node outside the configuration. apply refuses a configuration that would leave out a node Ceph already keeps data on, such as a cluster that ran on a filter alone, gets its first cephDevices and drops deviceFilter. Keep the filter, list the node’s devices, or take its disks out of Ceph first.
• A device added to a list joins Ceph on the next apply. A device removed from a list does not leave Ceph: take it out first, as described in Removing Nodes and Replacing Disks.

Running Ceph Next to Longhorn

Ceph and Longhorn can run in the same cluster:

storage:
  longhorn: {}
  ceph:
    deviceFilter: "^nvme[1-9]n1$"

• ceph-block becomes the default StorageClass, and longhorn stays. VM disks that do not name a StorageClass go to Ceph. Disks already on Longhorn stay there.
• The filter must not match sd names. Longhorn attaches its volumes to the nodes as sd disks, and a blank one could match. The installer warns about such a filter.
• With KDP installed on the same cluster, KDP’s own volumes stay on Longhorn. On Ceph alone, they go to Ceph. apply refuses a storage change that would move them, because a volume cannot change its StorageClass.

Moving VMs from Longhorn to Ceph

KubeV does not move disks between storage on its own. To move a VM’s disk, give the VM a disk on ceph-block in place of the Longhorn one, either by migrating the volume while the VM runs, or by restoring a backup to ceph-block.

Once no volume is left on Longhorn, remove longhorn from the configuration and apply it. apply refuses to remove Longhorn while volumes are still on it.

Checking Ceph

# Rook's view of the cluster
kubectl -n rook-ceph get cephcluster,cephblockpool

# Ceph's own view: health, monitors, disks and capacity
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph df

When Ceph takes fewer disks than expected, the logs of the rook-ceph-osd-prepare pods say which disks Rook skipped, and why:

kubectl -n rook-ceph logs -l app=rook-ceph-osd-prepare --tail=-1

Day-2 Operations

Capacity

Usable space is the raw space of the disks divided by the number of copies. Ceph warns when a disk is 85% full and stops all writes, for every VM on Ceph, at 95%. Keep it below about 75%, and add disks or nodes before it fills up.

Adding Disks and Nodes

Add a node to the configuration and apply it: Rook takes the devices its host lists, or those the filter matches. A device added to a host’s list joins Ceph on the next apply. A disk the filter matches, added to a node that is already in the cluster, is found when Rook next looks at the cluster. To make it look now, restart its operator:

kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator

Ceph then spreads the existing data over the new disks, which slows VM disks down for a while.

Removing Nodes and Replacing Disks

Before you remove a node from the configuration, or replace a failed disk, take its disks out of Ceph as Rook’s OSD management guide describes, and let Ceph copy their data elsewhere first. Removing a node with data on it can leave Ceph without enough copies, or without some data. When hosts list cephDevices, apply refuses to remove a node Ceph still keeps data on.

A cluster that goes below 3 nodes must also lower replicas: apply refuses a configuration that keeps more copies than there are nodes.

Removing Ceph

apply refuses to remove ceph from the configuration while the Ceph cluster KubeV installed runs, because uninstalling it would take the storage away from every volume on it. Move the volumes off Ceph and tear the Ceph cluster down as Rook’s teardown guide describes, then remove ceph and apply the configuration.

kubermatic-virtualization reset tears the whole KubeV cluster down. It deletes the Ceph cluster and has Rook wipe /var/lib/rook and Ceph’s disks on every node.

Limitations

• Ceph serves block storage only. KubeV sets up no shared file system (CephFS) and no object storage (S3).
• Ceph runs on the cluster’s own nodes. Connecting KubeV to an existing Ceph cluster outside it is not supported yet.
• Ceph uses the nodes’ main addresses. A separate storage network cannot be configured yet.