> ## Documentation Index
> Fetch the complete documentation index at: https://docs.siderolabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ceph Storage Cluster with Rook

> Guide on how to create a simple Ceph storage cluster with Rook for Kubernetes

export const version = 'v1.14';

This guide walks through deploying a Ceph storage cluster on Talos Linux using [Rook](https://rook.io), the Kubernetes operator that manages Ceph so you don't have to run it by hand.

Ceph's actual storage work is done by **OSDs** (Object Storage Daemons) — one OSD per disk, each responsible for writing, replicating, and reporting on the data on that disk. Ceph's total capacity is the sum of its OSDs: mons and the manager keep the cluster coordinated, but no OSDs means no usable storage. Every spare disk mentioned in this guide becomes an OSD.

## Prerequisites

Before deploying, make sure your cluster and machines meet the following:

* Talos Linux reserves an entire disk for the OS installation, so you need machines with at least one additional, unused disk for Ceph to use. Only nodes with a spare disk can host an OSD, Rook will not create one on a node without unused storage.
* The block devices or partitions Ceph will use must have no existing partitions or formatted filesystems.
* A supported Kubernetes version. Current Rook (v1.20.x) supports Kubernetes `v1.31` through `v1.36`, check the [Rook compatibility matrix](https://rook.io/docs/rook/latest/Getting-Started/Prerequisites/prerequisites/) for the version you're installing, since this range shifts with each Rook release.
* Helm `v3`.
* Enough non-control-plane nodes for your mon count. Rook runs 3 mons by default, one per node, and control-plane nodes are tainted against them, plan for at least 3 worker nodes, or see the mon quorum note under step 5 for smaller clusters.
* Headroom for the CSI driver stack. On top of the mons, the CSI drivers (RBD, CephFS, NFS, NVMe-oF) add real overhead. Since an OSD pod can only run on the node holding its disk, it has nowhere to go if that node is already full.
* Read the [Rook Ceph overview](https://rook.io/docs/rook/latest/Getting-Started/intro/) before deploying, Ceph is a large system, and the overview explains concepts this guide doesn't repeat.

## Install Rook and create a Ceph cluster

Current Rook installs in three parts: the operator, the Ceph-CSI drivers, and the cluster itself.

1. Add the Rook Helm repository:

   ```bash theme={null}
   helm repo add rook-release https://charts.rook.io/release
   ```

2. Install the Rook Operator into the `rook-ceph` namespace:

   ```bash theme={null}
   helm install --create-namespace --namespace rook-ceph rook-ceph rook-release/rook-ceph
   ```

3. Label the `rook-ceph` namespace so its privileged pods (Ceph daemons need privileged access) aren't blocked by default Pod Security admission:

   ```bash theme={null}
   kubectl label namespace rook-ceph pod-security.kubernetes.io/enforce=privileged
   ```

4. Install the Ceph-CSI drivers chart:

   ```bash theme={null}
   helm repo add ceph-csi-operator https://ceph.github.io/ceph-csi-operator/
   helm install --create-namespace --namespace rook-ceph ceph-csi-drivers ceph-csi-operator/ceph-csi-drivers
   ```

5. Install the Ceph cluster chart. The defaults will attempt to use all nodes and all unused disks in the cluster — see the [Rook cluster CRD documentation](https://rook.io/docs/rook/latest/CRDs/Cluster/ceph-cluster-crd/) for more specific node, device, and cluster configuration options.

   ```bash theme={null}
   helm install --create-namespace --namespace rook-ceph rook-ceph-cluster --set operatorNamespace=rook-ceph rook-release/rook-ceph-cluster
   ```

   <Note>
     If your cluster has fewer non-control-plane nodes than the default mon count (3), cluster creation will stall indefinitely with an event like:

     ```
     failed to perform validation before cluster creation: cannot start 3 mons on 2 node(s) when allowMultiplePerNode is false
     ```

     Relaxing this via `helm upgrade --set cephClusterSpec.mon.allowMultiplePerNode=true` will fail with a field-ownership conflict, since the operator's own reconciliation has already taken ownership of several `CephCluster` spec fields. Patch the resource directly instead:

     ```bash theme={null}
     kubectl -n rook-ceph patch cephcluster rook-ceph --type merge -p '{"spec":{"mon":{"allowMultiplePerNode":true}}}'
     ```

     Note that this direct patch means the Helm release's recorded values will drift from the live `CephCluster` spec; a later `helm upgrade` on this release may hit the same conflict again.
   </Note>

## Verify the cluster

The Rook operator needs time to bring the Ceph cluster online after the cluster chart is installed. Watch its progress:

```bash theme={null}
kubectl --namespace rook-ceph get cephcluster rook-ceph --watch
```

<Note>
  The `cephcluster` resource's `PHASE`/`MESSAGE` fields can lag noticeably behind what the operator is actually doing. For a more real-time view, check the operator's own logs:

  ```bash theme={null}
  kubectl -n rook-ceph logs deploy/rook-ceph-operator --tail=30
  ```

  Storage classes (`ceph-block`, `ceph-bucket`, `ceph-filesystem`) are created by the Helm chart at install time, independent of cluster health — their presence does not confirm the cluster is actually ready. To check whether OSDs are genuinely up, look for OSD pods directly:

  ```bash theme={null}
  kubectl -n rook-ceph get pods -l app=rook-ceph-osd
  ```
</Note>

Once the cluster reaches `HEALTH_OK`, the storage classes it creates are ready to use:

```bash theme={null}
kubectl --namespace rook-ceph get cephcluster rook-ceph
kubectl get storageclass
```

## Maintain the cluster during Talos upgrades

By default, Rook configures Ceph with 3 `mon` instances, so the data in `dataDirHostPath` can be regenerated from the other mons. Even so, when performing maintenance on a Talos node running Rook (such as a Talos upgrade), take care to keep the Ceph cluster healthy throughout.

Before upgrading a node, confirm the cluster is healthy:

```bash theme={null}
kubectl --namespace rook-ceph get cephclusters.ceph.rook.io rook-ceph
```

Only proceed if `HEALTH` reports `HEALTH_OK`. Upgrade one node at a time, and wait for the cluster to return to `HEALTH_OK` before touching the next node:

<Tabs>
  <Tab title="Talos-native">
    ```bash theme={null}
    talosctl upgrade --nodes <node-ip> --image ghcr.io/siderolabs/installer:<talos-version>
    ```
  </Tab>

  <Tab title="Omni">
    Omni manages Talos upgrades itself rather than through a direct `talosctl upgrade` call:

    * For a cluster not managed by a template, open the cluster in the Omni UI and click **Update Talos** (or the version indicator), select the target version, and click **Upgrade**.
    * For a template-managed cluster, update the `talos.version` field in the cluster template and sync it:

      ```bash theme={null}
      omnictl cluster template sync --file cluster-template.yaml
      ```

    Omni performs the upgrade one node at a time and waits for each node to be healthy before continuing, which pairs well with the mon-health caution above.
  </Tab>
</Tabs>

Wait for the cluster to report `HEALTH_OK` again before continuing to the next node:

```bash theme={null}
kubectl --namespace rook-ceph wait --timeout=1800s --for=jsonpath='{.status.ceph.health}=HEALTH_OK' cephclusters.ceph.rook.io rook-ceph
```

## Clean up

Cleaning up a Rook Ceph deployment happens in two stages: removing the Ceph cluster itself, then removing the metadata and disk data it left behind on each Talos node.

### Remove the Ceph cluster

To remove the ceph cluster:

1. Confirm cluster deletion, then delete the storage classes and Ceph storage types:

   ```bash theme={null}
   kubectl --namespace rook-ceph patch cephcluster rook-ceph --type merge -p '{"spec":{"cleanupPolicy":{"confirmation":"yes-really-destroy-data"}}}'
   kubectl delete storageclasses ceph-block ceph-bucket ceph-filesystem
   kubectl --namespace rook-ceph delete cephblockpools ceph-blockpool
   kubectl --namespace rook-ceph delete cephobjectstore ceph-objectstore
   kubectl --namespace rook-ceph delete cephfilesystem ceph-filesystem
   ```

2. Delete the Ceph cluster and its Helm release:

   ```bash theme={null}
   kubectl --namespace rook-ceph delete cephcluster rook-ceph
   helm --namespace rook-ceph uninstall rook-ceph-cluster
   helm --namespace rook-ceph uninstall ceph-csi-drivers
   ```

3. If you also want to remove the operator and everything it created, uninstall it, then find and delete the CRDs it left behind. The exact CRD list has changed across Rook versions (the CSI-related ones especially, given the chart restructuring above), so list them from your cluster rather than relying on a hardcoded list:

   ```bash theme={null}
   helm --namespace rook-ceph uninstall rook-ceph
   kubectl get crds -o name | grep -E 'ceph\.rook\.io|objectbucket\.io'
   ```

   Review the output, then delete the CRDs it returns.

### Remove Rook metadata and wipe the disks

If the Rook Operator was cleanly removed following the steps above, node metadata and disks should already be clean. If cleanup was interrupted, some metadata may remain on the system disk and partition information may remain on the storage disks.

1. Remove leftover node metadata. Replace `<storage-node-name>` with the node's name and `<dataDirHostPath>` with the Rook `dataDirHostPath` value used when installing the chart (`/var/lib/rook` if you used the default values). Repeat this for each storage node:

   ```bash theme={null}
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: disk-clean
   spec:
     restartPolicy: Never
     nodeName: <storage-node-name>
     volumes:
     - name: rook-data-dir
       hostPath:
         path: <dataDirHostPath>
     containers:
     - name: disk-clean
       image: busybox
       securityContext:
         privileged: true
       volumeMounts:
       - name: rook-data-dir
         mountPath: /node/rook-data
       command: ["/bin/sh", "-c", "rm -rf /node/rook-data/*"]
   EOF
   kubectl wait --timeout=900s --for=jsonpath='{.status.phase}=Succeeded' pod disk-clean
   kubectl delete pod disk-clean
   ```

2. Wipe the partition and filesystem data from each disk used in the Ceph cluster. On Talos, <a href={`../../talos/${version}/reference/cli#talosctl-wipe`}>disks are wiped directly</a> using `talosctl`:

<Tabs>
  <Tab title="Talos-native">
    ```bash theme={null}
    talosctl wipe disk <device> -n <node-ip>
    ```

    **Example:**

    ```bash theme={null}
    talosctl wipe disk sdb -n 192.168.1.10
    ```
  </Tab>

  <Tab title="Omni">
    Download a talosconfig for the cluster, then run the same command with it:

    ```bash theme={null}
    omnictl talosconfig --cluster <cluster-name> ./talosconfig
    talosctl --talosconfig ./talosconfig wipe disk <device> -n <node-ip>
    ```
  </Tab>
</Tabs>