Run Omni Air-Gapped

This document will walk through each component to run the “Sidero stack” in an offline environment which includes the following components.

Omni
Image Factory
Container registry
Authentication service

When running Talos with Omni you do not need to run additional services such as the Discovery Service because discovery functionality is built in to Omni.

If you already have services such as a container registry, authentication service (SAML or OIDC), or a trusted certificate authority you can skip those sections of the guide. This guide will set up proof-of-concept deployment to get you started. We recommend talking with the Sidero team for production deployments.

Omni is licensed under the Business Source License and requires a support contract for production use.

Prerequisites

There are some expectations your environment has the following supporting services for the services to run. These include:

Networks that can route Talos nodes to the Omni service endpoints
Basic networking services such as DNS, DHCP, and NTP
An admin system that can connect to the internet to download assets
A Linux server (e.g. RHEL, Ubuntu) to run the Sidero stack

The Sidero stack can be run on a single server or multiple servers, but we don’t recommend running Omni inside of Kubernetes for air gapped environments.

In addition to these services you’ll need the following tools installed on the administrator machine and server.

talosctl
docker or podman
cfssl and cfssljson
yq
crane
htpasswd

Podman is known to work but has some flags that are different than docker and you may have to translate them for your version of podman.

Download required tools

Download static binaries of the required tools. Docker and htpasswd should be provided by your distributions package manager.talosctlcfssl

CFSSL_VERSION=$(curl -sI https://github.com/cloudflare/cfssl/releases/latest | grep -i location | awk -F '/' '{print $NF}' | tr -d '\r')
curl -L -o cfssl https://github.com/cloudflare/cfssl/releases/download/${CFSSL_VERSION}/cfssl_${CFSSL_VERSION#v}_linux_amd64
curl -L -o cfssljson https://github.com/cloudflare/cfssl/releases/download/${CFSSL_VERSION}/cfssljson_${CFSSL_VERSION#v}_linux_amd64
chmod +x cfssl cfssljson

yq

YQ_VERSION=$(curl -sI https://github.com/mikefarah/yq/releases/latest | grep -i location | awk -F '/' '{print $NF}' | tr -d '\r')
curl -L -o yq "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_amd64"
chmod +x yq

crane

CRANE_VERSION=$(curl -sI https://github.com/google/go-containerregistry/releases/latest | grep -i location | cut -d/ -f8 | tr -d '\r')
curl -sL "https://github.com/google/go-containerregistry/releases/download/${CRANE_VERSION}/go-containerregistry_Linux_x86_64.tar.gz" | tar -xz crane
chmod +x crane

Export endpoints

To make this guide easier to follow we will set global variables for each of the endpoints and ports we will use. Update the hostnames and ports if you change any of them from the defaults.

REGISTRY_ENDPOINT=registry.internal:5000
FACTORY_ENDPOINT=factory.internal:8080
AUTH_ENDPOINT=auth.internal:5556
OMNI_ENDPOINT=omni.internal

1. Generate certificates

In order to run services securely, even in an air gapped environment, you should run with encrypted data in transit and at rest. There are multiple certificates and keys needed to secure your infrastructure.

CA certificate (root of trust)
Domain certificates for the following endpoints
- Omni
- Authentication
- Image factory
- Container registry
Container signing certificate
Omni database encryption key

Create Root CA certificate (optional)

If you already have a trusted, internal root CA you can skip generating the CA (root of trust). You will need to use your existing CA to create certificates for the services in this guide. Skip to Generate endpoint certificates

We will use the cfssl command to make the CA and certificate signing easier, but openssl can be used if you have existing CA infrastructure.

cat <<EOF > ca-csr.json
{
  "CN": "Internal Root CA",
  "key": {
    "algo": "rsa",
    "size": 4096
  },
  "names": [
    {
      "C": "US",
      "O": "Internal Infrastructure",
      "OU": "Security"
    }
  ]
}
EOF

With the configuration create a CA certificate.

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

This will give you a private key, ca-key.pem, a public key ca.pem, and a signing request ca.csr. For any client that will be calling the internal services you will need to install the ca.pem file into your trusted store.

Linux (RHEL)
Linux (Ubuntu)
macOS
Windows

On Red Hat and Fedora based distros you can copy the ca.pem file into the /etc/pki/ca-trust/source/anchors/ folder and then run the following command to generate the trusted root store:

sudo cp ca.pem /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

On Ubuntu and Debian based Linux distros you can copy the ca.pem file into the /usr/local/share/ca-certificates/ directory and rename it to ca.crt

sudo cp ca.pem /usr/local/share/ca-certificates/ca.crt
sudo update-ca-certificates

For macOS you should open the Keychain Access application and drag the ca.pem file into the window to install it.

On Windows you should rename the certificate extension from .pem to .crt. You can then double click on the file and select Install Certificate -> Local Machine. You then need to select “Place all certificates in the following store” and select the Trusted Root Certification Authorities.If you’re using Windows Subsystem for Linux (WSL) you should follow the Linux guide for installing the certificate.

Generate endpoint certificates

Generate a single certificate that all services can use based on the CA we just created. For a production deployment you should generate individual certificates for each service. Create a signing configuration to let cfssl know we want a web server certificate that should expire in 1 year.

cat <<EOF > ca-config.json
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "web-server": {
        "usages": ["signing", "key encipherment", "server auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

When using .internal domains you will need to update your DNS server or /etc/hosts file to make sure the endpoints resolve properly. The file should look something like this.

# Example config in /etc/hosts
127.0.0.1 localhost registry.internal factory.internal auth.internal omni.internal

Now create a wildcard certificate for your services. We’ll be using the ICANN reserved .internal TLD for service domains, but you can use any domain if you have it registered.

cat <<EOF > wildcard-csr.json
{
  "CN": "Internal Wildcard",
  "hosts": [
    "internal",
    "${AUTH_ENDPOINT%:*}",
    "${REGISTRY_ENDPOINT%:*}",
    "${FACTORY_ENDPOINT%:*}",
    "${OMNI_ENDPOINT%:*}",
    "127.0.0.1",
    "$(hostname -I | awk '{print $1}')"
  ],
  "key": {
    "algo": "rsa",
    "size": 4096
  }
}
EOF

Generate the wildcard certificates for services.

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=web-server wildcard-csr.json \
  | cfssljson -bare server

Create a certificate chain with server and CA.

cat server.pem ca.pem > server-chain.pem

This will create a private server key, server-key.pem, a public server key, server.pem, a server signing request, server.csr, and a server-chain.pem, a server certificate with CA. Services in this guide run with different user IDs so we will update the certificates to allow all users to read them. This is not suitable or secure for a production environment.

chmod 644 server*.pem

2. Image factory and container registry

Using the certificates we just created, follow the guide Deploy Image Factory On-prem. This will create a container registry and host the Image Factory in your environment. It will also sign container images with an offline key for verification.

If you do not have a working Image Factory with Talos images and extensions seeded do not continue with the guide. That is a pre-requisite for running Omni in an air gapped environment.

3. Authentication

If you have existing SAML or OIDC authentication available you can use that with Omni. Please see the configuration guides in the Authentication and Authorization section. For a PoC environment we will run dex with static users configured. Dex can be used for static configuration or to communicate with upstream providers. For this guide we will configure static users.

Deploy dex (optional)

Because Omni does not have any user authentication Dex will be configured so we can log in to Omni with a static user. You will need to download the dex container from a machine that has internet access and push it to your internal registry.

Download dex

Download the dex container image.

docker pull ghcr.io/dexidp/dex:v2.41.1

If your machine has access to the internal registry you can push the image directly.

docker tag ghcr.io/dexidp/dex:v2.41.1 ${REGISTRY_ENDPOINT}/dexidp/dex:v2.41.1
docker push ${REGISTRY_ENDPOINT}/dexidp/dex:v2.41.1

If you need to send the image to a remote machine or transfer it via an offline method you can export the container image.

docker save -o dex.tar ghcr.io/dexidp/dex:v2.41.1

On the remote machine load the archive into the local image storage and then push it to the registry.

docker load -i dex.tar
docker push ${REGISTRY_ENDPOINT}/dexidp/dex:v2.41.1

Configure dex

We will create an example dex configuration to use with Omni. Create a password. Make sure you remember this password for logging in later.

export OMNI_USER_PASSWORD=$(htpasswd -BnC 15 user42 | cut --delimiter=: --fields=1 --complement)

Create a dex configuration file.

cat <<EOF > dex.yaml
issuer: https://${AUTH_ENDPOINT}
storage:
  type: memory

web:
  https: 0.0.0.0:5556
  tlsCert: /etc/dex/tls/server-chain.pem
  tlsKey: /etc/dex/tls/server-key.pem

enablePasswordDB: true

staticClients:
  - name: Omni
    id: omni
    secret: aW50ZXJuYWwtc2lkZXJvLXN0YWNrCg== #internal-sidero-stack
    redirectURIs: [https://omni.internal/oidc/consume]

staticPasswords:
  - email: "[email protected]"
    username: "admin"
    preferredUsername: "admin"
    hash: "$OMNI_USER_PASSWORD"
EOF

Run dex

Run dex with the provided configuration and certificate.

docker run -d \
  --name dex \
  -p 5556:5556 \
  -v $(pwd)/dex.yaml:/etc/dex/dex.yaml \
  -v $(pwd)/server-key.pem:/etc/dex/tls/server-key.pem \
  -v $(pwd)/server-chain.pem:/etc/dex/tls/server-chain.pem \
  ${REGISTRY_ENDPOINT}/dexidp/dex:v2.41.1 \
    dex serve /etc/dex/dex.yaml

If your machine has SELinux in enforcing mode you may need to add :Z to the volume mounts in the docker command.

4. Run Omni

Omni will depend on the following URLs. If these services are not running or the hostnames do not resolve you may need to add them to your /etc/hosts file.

omni.internal
factory.internal
registry.internal
auth.internal

Create etcd encryption key

Data in Omni’s database is encrypted and we need an encryption key to provide to Omni. Generate a GPG key:

gpg --quick-generate-key "Omni (Used for etcd data encryption) [email protected]" rsa4096 cert never
FINGERPRINT=$(gpg --with-colons --list-keys "[email protected]" | awk -F: '$1 == "fpr" {print $10; exit}')
gpg --quick-add-key ${FINGERPRINT} rsa4096 encr never
gpg --export-secret-key --armor [email protected] > omni.asc

Note: Do not add passphrases to keys during creation.

Download Omni

Download the Omni container image. If your machine has access to the internal registry you can push the image directly. If you need to send the image to a remote machine or transfer it via an offline method you can export the container image. On the remote machine load the archive into the local image storage and then push it to the registry.

Start Omni container

This will run Omni with an embedded etcd database mounted to the host. It is not recommended for production use cases. The command assumes the certificates generated earlier are available in the local directory where you run this command.

We changed the --metrics-bind-addr to use port :2123 to avoid port conflicts with Image Factory (if it’s running on the same host).

These flags mount the files and directories needed by Omni (e.g. certificates, etcd storage) and set flags to connect Omni to the upstream services (e.g. factory, authentication).

5. Create a cluster

Before you create a cluster you will need to generate installation media. When creating a cluster you’ll need to make sure you patch the machine config to redirect container registries to your internal registry.

Download Kubernetes containers

Seed the internal registry with Kubernetes container images.

talosctl images k8s-bundle > k8s-images.txt

Download the images and push them into the internal registry.

Internet available
Air-gapped

If your machine can reach the public internet and the internal registry at the same time you can copy the images internally with this command.

for SOURCE_IMAGE in $(cat images.txt)
  do
    IMAGE_WITHOUT_DIGEST=${SOURCE_IMAGE%%@*}
    IMAGE_WITH_NEW_REG="${REGISTRY_ENDPOINT}/${IMAGE_WITHOUT_DIGEST#*/}"
    crane copy \
      $SOURCE_IMAGE \
      $IMAGE_WITH_NEW_REG
done

If you don’t have direct access to an internal container registry (e.g. air gapped environment) you need to download the container images while connected to the internet with this command:

cat images.txt \
  | talosctl images cache-create \
      --layout flat \
      --image-cache-path ./image-cache \
      --images=-

Move the image-cache folder to an air gapped machine and serve the images on a read only, temporary container registry with:

IP=$(hostname -I | awk '{print $1}')

talosctl image cache-cert-gen \
  --advertised-address $IP

talosctl image cache-serve \
  --address $IP:5000 \
  --image-cache-path ./image-cache \
  --tls-cert-file tls.crt \
  --tls-key-file tls.key

A temporary image registry will run on your local machine IP address port 5000 with self-signed certificates. Copy the images to an internal, permanent container registry.

for SOURCE_IMAGE in $(cat images.txt)
  do
    IMAGE_WITHOUT_DIGEST=${SOURCE_IMAGE%%@*}
    IMAGE_WITH_NEW_REG="${REGISTRY_ENDPOINT}/${IMAGE_WITHOUT_DIGEST#*/}"
    LOCALHOST_IMAGE="localhost:5000/${IMAGE_WITHOUT_DIGEST#*/}"
    crane copy \
      $LOCALHOST_IMAGE \
      $IMAGE_WITH_NEW_REG
done

Create installation media

The first step to create a cluster is to boot machines and connect them to Omni. In order to do that we will need to embed the self-signed CA certificate into Talos. Create a configuration for a TrustedRootsConfig:

yq eval --null-input '
.apiVersion = "v1alpha1" |
.kind = "TrustedRootsConfig" |
.name = "internal-ca" |
.certificates = load_str("ca.pem")
' > trustedrootsconfig.yaml

You can use this config two different ways. Use kernel arguments for Talos 1.11 and older and use embedded config for Talos 1.12.

Embedded config (Talos 1.12+)
Kernel args (Talos 1.11)

Create an output directory for installation media and config.

mkdir _out
mv trustedrootsconfig.yaml _out/machine-config.yaml
echo "---" >> _out/machine-config.yaml

Download machine join configuration from Omni. You can do this from the Omni web interface home page by clicking on the Download Machine Join Config button or if you have omnictl installed you can download it with

omnictl jointoken machine-config >> _out/machine-config.yaml

Create a static hosts configuration for Omni and the registry.

yq --null-input '
.apiVersion = "v1alpha1" |
.kind = "StaticHostConfig" |
.name = "'$(hostname -I | awk '{print $1}')'" |
.hostnames = ["'${OMNI_ENDPOINT%:*}'", "'${REGISTRY_ENDPOINT%:*}'"]
' > hosts-config.yaml

Append this configuration to the others.

echo "---" >> _out/machine-config.yaml
cat hosts-config.yaml >> _out/machine-config.yaml

Embed both configurations and create an installation ISO with imager.

Get the kernel arguments from Omni with omnictl. You can also copy them from the Omni web interface with the Copy Kernel Parameters button on the home page.

OMNI_KERNEL_ARGS=$(omnictl jointoken kernel-args)

Compress and base64 encode the configuration for a kernel argument.

TRUSTED_ROOT_CONFIG=$(cat trustedrootsconfig.yaml | zstd --compress --ultra -21 | base64 -w 0)

Create an ISO with imager with both of the kernel arguments.

No matter which version of Talos you use you should have a Talos iso file in the _out directory. You can use this to boot a machine and it will connect to Omni and trust the self-signed CA certificate.

Create cluster in Omni

Guides on creating a cluster on Omni can be found at creating an Omni cluster. Because we’re working in an airgapped environment we will need the following values added to our cluster configs so they know where to pull images from. We also need to provide the CA certificate to the node so it will trust the certificate that signed omni.internal endpoint.

NOTE: In this example, cluster discovery is also disabled. You may also configure cluster discovery via Omni. More information on the Discovery Service can be found here.

machine:
  registries:
    mirrors:
    docker.io:
      endpoints:
      - https://${REGISTRY_ENDPOINT}
    gcr.io:
      endpoints:
      - https://${REGISTRY_ENDPOINT}
    ghcr.io:
      endpoints:
      - https://${REGISTRY_ENDPOINT}
    registry.k8s.io:
      endpoints:
      - https://${REGISTRY_ENDPOINT}
cluster:
  discovery:
    enabled: false
---
apiVersion: v1alpha1
kind: RegistryTLSConfig
name: ${REGISTRY_ENDPOINT}
ca: |-
  -----BEGIN CERTIFICATE-----
  MIID...IDAQAB
  -----END CERTIFICATE-----

The machine patch should be added to the cluster during cluster creation and should be applied cluster wide.

Overview

Getting Started

Self Hosted

Infrastructure and Extensions

Omni Cluster Setup

Cluster Management

Security and Authentication

Reference

Troubleshooting and Support

Prerequisites

Export endpoints

1. Generate certificates

Create Root CA certificate (optional)

Generate endpoint certificates

2. Image factory and container registry

3. Authentication

Deploy dex (optional)

Download dex

Configure dex

Run dex

4. Run Omni

Create etcd encryption key

Download Omni

Start Omni container

5. Create a cluster

Download Kubernetes containers

Create installation media

Create cluster in Omni

Overview

Getting Started

Self Hosted

Infrastructure and Extensions

Omni Cluster Setup

Cluster Management

Security and Authentication

Reference

Troubleshooting and Support

​Prerequisites

​Export endpoints

​1. Generate certificates

​Create Root CA certificate (optional)

​Generate endpoint certificates

​2. Image factory and container registry

​3. Authentication

​Deploy dex (optional)

​Download dex

​Configure dex

​Run dex

​4. Run Omni

​Create etcd encryption key

​Download Omni

​Start Omni container

​5. Create a cluster

​Download Kubernetes containers

​Create installation media

​Create cluster in Omni

Prerequisites

Export endpoints

1. Generate certificates

Create Root CA certificate (optional)

Generate endpoint certificates

2. Image factory and container registry

3. Authentication

Deploy dex (optional)

Download dex

Configure dex

Run dex

4. Run Omni

Create etcd encryption key

Download Omni

Start Omni container

5. Create a cluster

Download Kubernetes containers

Create installation media

Create cluster in Omni