Skip to main content
The Omni exporter reads an Omni instance and exposes the state of each object it finds as metrics: one set of series per cluster, machine, machine set, upgrade, and etcd backup. It is a Prometheus exporter, a small service that sits next to something you want to monitor and publishes its state over HTTP in the format Prometheus scrapes. Running it is how you bring the state of your Omni instance into your own monitoring stack, where you can alert on conditions such as a cluster that is no longer ready or a machine that has disconnected, and build dashboards in Grafana alongside the rest of your infrastructure.

​
Prerequisites

You need:
  • An Omni instance, either SaaS or self-hosted.
  • omnictl, installed and configured. See Install and configure omnictl.
  • A Prometheus instance that can reach the machine you run the exporter on.

​
Set up the exporter

Setting up the exporter takes three steps:
  • Create an identity for it to authenticate with.
  • Start it against your instance.
  • Point Prometheus at it.

​
Step 1: Create a service account

Create a service account for the exporter:
The Reader role grants read-only access to the resources of your instance, which is everything the exporter needs. It cannot create, change, or delete anything, and the exporter never writes to Omni. For more on what each role allows, see the Omni security model.
--use-user-role defaults to true, which clones the role of the user running the command instead of the role you pass. Set it to false, as shown above, or the account ends up with your own role.
The command prints an OMNI_SERVICE_ACCOUNT_KEY. It is shown once, so store it somewhere safe. Service account keys expire after a year by default, and rotating the key requires restarting the exporter.

​
Step 2: Run the exporter

Point the exporter at your instance and give it the key. In the command below, replace <account>.omni.siderolabs.io with the address of your instance and <key> with the key from Step 1:
Pin the image to a release tag, as the example does. Use the newest release, which the exporter releases page lists. Avoid the latest tag, which follows the main branch of the exporter and not the newest release. The restart policy keeps the exporter running, since it is meant to stay up continuously. By default, metrics are served on port 10048, which --web.listen-address can change. Confirm the exporter is working before moving on:
A value of 1 confirms that the exporter reached Omni and that every resource type finished its initial sync:
Resource types sync independently, so the value starts at 0 and reaches 1 only once the last of them completes. Retry the command if the exporter has only just started.

​
Step 3: Add a scrape job to Prometheus

Add the exporter as a target in your Prometheus configuration:
Each scrape renders from data the exporter already holds in memory, so scraping more often costs little. The one exception is a small reachability check the exporter makes against Omni on every scrape, which is bounded by a timeout of a few seconds. Keep your Prometheus scrape timeout above that.

​
Configure the exporter

The defaults suit most instances, and --help lists every flag. The settings below are the ones most often worth changing. Passing the key as an environment variable is convenient, but it exposes the key to anything that can read the process environment. Use --omni.service-account-key-file to read it from a file instead, which also fits a Kubernetes secret mount better. With docker run, mount the file into the container and give the flag the path inside it:
The metrics endpoint is unauthenticated by default. Because the metrics describe your infrastructure, restrict access to it. The exporter uses the Prometheus exporter toolkit for this, so TLS and basic authentication are configured through a web configuration file passed with --web.config.file, not through flags of the exporter itself. See the exporter toolkit web configuration reference for the file format. If your self-hosted instance serves a certificate the exporter does not trust, add the certificate authority to the trust store of the container. There is an --omni.insecure-skip-tls-verify flag, but it disables certificate checking on a connection that carries a service account key, so keep it to temporary testing.

​
Understand the metrics

The exporter emits one set of series per object. The table is a representative subset. The metrics reference in the exporter repository is the authoritative list. Metrics that represent a state with several possible values, such as omni_exporter_cluster_phase, emit one series for every possible value, with the current one set to 1 and the rest to 0. A cluster changing phase therefore changes the values of existing series instead of creating and removing them, which keeps queries to simple equality matches such as omni_exporter_cluster_phase{phase="running"} == 1. Descriptive attributes that change often, such as a hostname or a Talos version, live on the _info metrics. Keeping them off the state metrics means a machine being renamed does not disturb the series you alert on. Join them back in your query when you need them, for example to see the hostnames of disconnected machines:
The exporter performs no aggregation of its own. PromQL does that, so counting the machines of each cluster by stage is a query:
Gate alerts on object metrics with omni_exporter_up == 1, and alert on omni_exporter_up == 0 separately. When Omni is unreachable the exporter stops serving object metrics, which is indistinguishable from the objects having been deleted. An alert that fires on the absence of a series will otherwise treat an Omni outage as every cluster disappearing at once.

​
Troubleshoot

Once it is running, the exporter degrades instead of stopping, so an unreachable instance or a failing watch shows up in the metrics and not in a container that has exited. Configuration is the exception: a missing endpoint or an unreadable service account key fails at startup, and the container exits immediately. For anything else, start with omni_exporter_up, then narrow down:
  • omni_exporter_up is 0 and omni_exporter_reachable is 0. The exporter cannot reach Omni. Check the endpoint, network connectivity, and whether the service account key has expired or been deleted. An expired key looks the same as an unreachable instance from the metrics alone, so check the exporter logs, where the underlying error is reported.
  • omni_exporter_up is 0 but omni_exporter_reachable is 1. Omni is reachable, and at least one resource type has not finished its initial sync or failed to render. This is normal for a few seconds after startup. If it persists, omni_exporter_collector_success shows which resource type is affected.
  • Some object metrics are missing while others are served. Suppression is per resource type, so a collector that has not completed its initial sync withholds only its own metrics and the rest keep serving. Use omni_exporter_collector_success to identify which one. An unreachable Omni is the case that suppresses every resource type at once.
The exporter logs to stderr in JSON by default. Use --log.format=text for readable output while troubleshooting, and --log.level=debug for more detail.