Create Etcd Backups - Sidero Documentation

Etcd is the key-value store that holds all Kubernetes cluster state. If etcd becomes corrupted or unavailable, your cluster will stop functioning. Omni supports creating etcd backups so that you can restore a cluster to a known good state if something goes wrong. Omni supports two backup storage backends: local and s3. The backend is selected during Omni initialization. This guide covers how to configure S3 backup storage, create a manual backup, and enable automatic backups. To restore a cluster from a backup, see Restore Etcd of a Cluster Managed by Cluster Templates.

Check backup status

Before configuring backups, check the current state of the backup subsystem:

omnictl get etcdbackupoverallstatus

On a freshly created Omni instance, the output will look similar to this:

NAMESPACE   TYPE                      ID                          VERSION   CONFIGURATION NAME   CONFIGURATION ERROR   LAST BACKUP STATUS   LAST BACKUP ERROR   LAST BACKUP TIME   CONFIGURATION ATTEMPT
ephemeral   EtcdBackupOverallStatus   etcdbackup-overall-status   1         s3                   not initialized

The key fields to pay attention to are:

Field	Description
`CONFIGURATION NAME`	The backup backend currently configured (`s3` or `local`).
`CONFIGURATION ERROR`	Any error with the current backup configuration. `not initialized` means the backend has been selected but not yet configured.
`LAST BACKUP STATUS`	The result of the most recent backup attempt.
`LAST BACKUP ERROR`	Any error from the most recent backup attempt.
`LAST BACKUP TIME`	When the most recent backup completed.
`CONFIGURATION ATTEMPT`	The number of times Omni has attempted to apply the backup configuration.

In the output above, CONFIGURATION NAME is s3 and CONFIGURATION ERROR shows not initialized, which means the S3 backend has been selected but the credentials have not yet been configured. The remaining fields are empty because no backups have been created yet.

Configure S3 backup storage

Omni stores etcd backups in an S3-compatible bucket. Before creating any backups, you need to provide Omni with the credentials and configuration it needs to access your bucket. This section covers compatibility considerations for non-AWS providers and how to apply the S3 configuration.

S3-compatible providers

Omni uses the AWS S3 API for storing etcd backups. Non-AWS providers that advertise S3 compatibility may work when configured with a custom endpoint, but Omni does not maintain a compatibility list for third-party providers. If you use a non-AWS S3 provider:

Test backup and restore workflows before relying on them in production.
Some providers may require additional configuration, such as a region value or provider-specific credentials.
Ensure the endpoint and credentials are valid and accessible from Omni.

During configuration, Omni validates access by attempting to list objects in the bucket. If validation fails, the configuration is not applied.

Apply the S3 configuration

To configure S3 as the backup backend, create an EtcdBackupS3Configs.omni.sidero.dev resource. The following example configures Omni to use a MinIO S3 instance, edit this example to fit your S3 instance:

cat <<EOF > etcd-backup-s3-conf.yaml
metadata:
  namespace: default
  type: EtcdBackupS3Configs.omni.sidero.dev
  id: etcd-backup-s3-conf
spec:
  bucket: mybucket
  region: us-east-1
  endpoint: http://127.0.0.1:9000
  accesskeyid: access
  secretaccesskey: secret123
  sessiontoken: ""
EOF

The fields in the spec are:

Field	Required	Description
`bucket`	Yes	The name of the S3 bucket to store backups in.
`region`	No	The S3 bucket region. Defaults to the AWS default region if not set.
`endpoint`	No	The S3 endpoint URL. Defaults to the AWS S3 endpoint if not set. Required for non-AWS providers.
`accesskeyid`	No	The access key ID for the S3 bucket. If not set, Omni assumes it is running on an EC2 instance with an IAM role that has access to the bucket.
`secretaccesskey`	No	The secret access key for the S3 bucket.
`sessiontoken`	No	A session token for temporary credentials, if required.

Apply the S3 configuration:

omnictl apply -f etcd-backup-s3-conf.yaml

Omni validates the credentials during resource creation by attempting to list objects in the bucket. If validation fails, the resource will not be created and an error will be returned. To confirm the configuration was applied successfully, check the backup status again:

omnictl get etcdbackupoverallstatus

The CONFIGURATION ERROR field should now be empty:

NAMESPACE   TYPE                      ID                          VERSION   CONFIGURATION NAME   CONFIGURATION ERROR   LAST BACKUP STATUS   LAST BACKUP ERROR   LAST BACKUP TIME   CONFIGURATION ATTEMPT
ephemeral   EtcdBackupOverallStatus   etcdbackup-overall-status   2         s3

Create a manual backup

To trigger a backup immediately, create and apply an EtcdManualBackups.omni.sidero.dev resource.

The <unix-timestamp> value must be no more than one minute in the future or past. The nanos field must always be 0.

Run the following command, replacing <cluster-name> with the name of your cluster:

cat <<EOF > etcd-manual-backup.yaml
metadata:
  namespace: ephemeral
  type: EtcdManualBackups.omni.sidero.dev
  id: <cluster-name>
spec:
  backupat:
    seconds: $(date +%s)
    nanos: 0
EOF

Apply the resource:

omnictl apply -f etcd-manual-backup.yaml

After a few seconds, check the per-cluster backup status:

omnictl get etcdbackupstatus -o yaml

The output will look similar to this:

metadata:
  namespace: ephemeral
  type: EtcdBackupStatuses.omni.sidero.dev
  id: <cluster-name>
  version: 1
  owner: EtcdBackupController
  phase: running
spec:
  status: 1
  error: ""
  lastbackuptime:
    seconds: 1702166400
    nanos: 985220192
  lastbackupattempt:
    seconds: 1702166400
    nanos: 985220192

To check the overall backup subsystem status:

omnictl get etcdbackupoverallstatus -o yaml

Enable automatic backups

Omni supports scheduled automatic backups. You can configure the backup interval using either cluster templates or by editing the cluster resource directly.

Cluster templates
Cluster resource

To enable automatic etcd backups using a cluster template:

Add a features.backupConfiguration block to your cluster template:

cat <<EOF > etcd-backup.yaml
kind: Cluster
name: talos-default
kubernetes:
  version: v1.28.2
talos:
  version: v1.5.5
# add the backupConfiguration block to your cluster templates
features:
  backupConfiguration:
    interval: 1h
---
kind: ControlPlane
machines:
  - <controlplane-machine-uuid> # control plane machine uuid
---
kind: Workers
machines:
  - <worker-machine-one-uuid> # worker machine uuid
  - <worker-machine-two-uuid> # worker machine uuid
EOF

Replace <controlplane-machine-uuid>, <worker-machine-one-uuid>, and <worker-machine-two-uuid> with the UUIDs of your control plane and worker machines.For more information on configuring cluster features with cluster templates, see the Cluster Template reference documentation.

Before applying, check whether automatic backups are already enabled on the cluster:

omnictl cluster template -f etcd-backup.yaml diff

If automatic backups are not yet enabled, the output will show the backup configuration being added:

--- Clusters.omni.sidero.dev(default/talos-default)
+++ Clusters.omni.sidero.dev(default/talos-default)
@@ -19,4 +19,7 @@
   features:
     enableworkloadproxy: false
     diskencryption: false
-  backupconfiguration: null
+  backupconfiguration:
+    interval:
+      seconds: 3600
+      nanos: 0

If automatic backups are already enabled, the diff will show an existing backupconfiguration block with a non-null value.In that case, you can update the interval value in your template and re-run the diff to confirm the change before applying, or skip to step 4 to verify the existing backup schedule is working as expected.

Apply the template:

omnictl cluster template -f etcd-backup.yaml sync

If no backups exist for the cluster yet, Omni will create one immediately rather than waiting for the first scheduled interval.

Verify that the backup was created by running the following command, replacing <cluster-name> with the name of your cluster:

omnictl get etcdbackup --selector omni.sidero.dev/cluster=<cluster-name>

This command will output something similar to this:

NAMESPACE   TYPE         ID                         VERSION     CREATED AT
external    EtcdBackup   talos-default-1702166400   undefined   {"nanos":0,"seconds":1702166400}

If your cluster is not managed by cluster templates, you can enable automatic backups by editing the cluster resource directly.To enable automatic etcd backups by editing the cluster resource directly:

Retrieve the current cluster resource, replace the <cluster-name> placeholder with the name of your cluster:

omnictl get cluster <cluster-name> -o yaml > cluster-backup.yaml

In the cluster-backup.yaml file, add the backupconfiguration block while preserving all existing fields:

# cluster-backup.yaml
metadata:
  namespace: default
  type: Clusters.omni.sidero.dev
  id: talos-default
  finalizers:
    - KubernetesUpgradeStatusController
    - TalosUpgradeStatusController
    - SecretsController
    - ClusterController
spec:
  installimage: ""
  kubernetesversion: 1.28.2
  talosversion: 1.5.5
  features:
    enableworkloadproxy: false
    diskencryption: false
  # add the backupconfiguration block
  backupconfiguration:
    interval:
      seconds: 3600
      nanos: 0

Apply the updated file:

omnictl apply -f cluster-backup.yaml

Next steps

To restore a cluster from an etcd backup, see Restore Etcd of a Cluster Managed by Cluster Templates.

Documentation Index

​Check backup status

​Configure S3 backup storage

​S3-compatible providers

​Apply the S3 configuration

​Create a manual backup

​Enable automatic backups

​Next steps

Check backup status

Configure S3 backup storage

S3-compatible providers

Apply the S3 configuration

Create a manual backup

Enable automatic backups

Next steps