> ## Documentation Index
> Fetch the complete documentation index at: https://docs.siderolabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Image Factory On-Prem

> Run Image Factory on-prem to generate Talos images locally

export const image_factory_release = 'v1.0.2';

export const version = 'v1.13';

export const release = 'v1.13.2';

The [Image Factory](https://github.com/siderolabs/image-factory) is a way for you to dynamically create Talos Linux images. There is a public, hosted version of the Image Factory at [factory.talos.dev](https://factory.talos.dev) and it can also be run in your environment.

The Image Factory is a critical component of Omni to generate installation media and update Talos nodes, but it is not required to use Omni to use the Image Factory. It is a web interface and API for the <a href={`../../../talos/${version}/platform-specific-installations/boot-assets#imager`}>`imager`</a> command which is used to customize Talos from the command line.

## Prerequisites

* [`talosctl`](../../talos/latest/getting-started/talosctl)
* [`crane`](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md)
* `docker` or `podman`

> Podman is known to work but has some flags that are different than docker and you may have to translate them for your version of podman.

### Container registry

If you already have a container registry available you can export your registry to an environment variable. and skip to create an [image cache signing key](#image-cache-signing-key).

```shell theme={null}
REGISTRY_ENDPOINT=registry.internal:5000
```

If you don't have a container registry available to push images to you can temporarily run one with the `registry` container. We recommend using the official `registry:2` registry from docker as some registries do not support all OCI images.

<Note>This example doesn't have persistent storage.</Note>

<Tabs>
  <Tab title="Secure registry">
    We recommend using certificates for your temporary registry you will need to provide your own certificates and mount them into the container at run time. If you do not have certificates, follow the steps in the [Omni air-gapped documentation](./run-omni-airgapped).

    ```bash theme={null}
    docker run -d \
      --name registry \
      -p 5000:5000 \
      -v ${PWD}/server-key.pem:/certs/server-key.pem:ro,Z \
      -v ${PWD}/server-chain.pem:/certs/server-chain.pem:ro,Z \
      -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
      -e REGISTRY_HTTP_TLS_KEY=/certs/server-key.pem \
      -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server-chain.pem \
      registry:2
    ```

    Make sure the CA certificate is in your system pki path and `docker` has restarted to trust the certificate.
  </Tab>

  <Tab title="Insecure registry">
    A registry can be run without certificates or encrypted communication. Running this way will require your to add a flag to `crane` and `docker` to allow insecure communication.

    ```bash theme={null}
    docker run -d -p 5000:5000 --name registry registry:2
    ```
  </Tab>

  <Tab title="Air-gapped">
    Without internet access you will need to download the container image, transfer it internally, and load it on the target machine.

    <CodeBlock language="bash">
      {`docker save -o registry.tar registry:2 \ndocker save -o image-factory.tar ghcr.io/siderolabs/image-factory:${image_factory_release}`}
    </CodeBlock>

    Transfer the `registry.tar` and `image-factory.tar` files to an internal system.

    ```bash theme={null}
    docker load -i registry.tar
    docker load -i image-factory.tar
    ```

    Run the registry with certificates.

    ```bash theme={null}
    docker run -d \
      --name registry \
      -p 5000:5000 \
      -v ${PWD}/server-key.pem:/certs/server-key.pem:ro,Z \
      -v ${PWD}/server-chain.pem:/certs/server-chain.pem:ro,Z \
      -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
      -e REGISTRY_HTTP_TLS_KEY=/certs/server-key.pem \
      -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server-chain.pem \
      registry:2
    ```
  </Tab>
</Tabs>

### Image cache signing key

You need to create a Cache Signing Key to sign cached Talos image artifacts, ensuring they haven’t been tampered with before being served.

```shell theme={null}
openssl ecparam -name prime256v1 -genkey -noout -out signing-key.key
```

### Image cache storage (optional)

There are a variety of image cache locations to store built images. Without an image cache, each asset will be built on demand which can consume a high amount of CPU on the image factory machine.

Some supported cache storage options include:

* CDN
* s3 bucket (or compatible API)

Please view the `--help` output for cache options.

## Run Image Factory

There are two supported methods to run the Image Factory:

* Connected to the upstream Sidero container registry
* Using a custom container registry

A custom container registry is required for air-gapped environments or custom Talos builds.

<Tabs>
  <Tab title="Image factory connected">
    <Note>Run with the official, upstream container registry if your machine is connected to the internet and you don't need custom Talos images.</Note>

    The official Sidero Labs registry has all of the required Talos installation containers, extensions, and tools.

    Export the URL for where you will be hosting the Image Factory.

    ```bash theme={null}
    export FACTORY_URL="http://factory.internal:8080"
    ```

    You need to provide a registry where image schematics can be stored. Create a config file.

    ```bash theme={null}
    cat <<EOF > image-factory.yaml
    artifacts:
      schematic:
        registry: $REGISTRY_ENDPOINT
    cache:
      signingKeyPath: "/signing-key.key"
    http:
      externalURL: $FACTORY_URL
    EOF
    ```

    If you want to run image factory connected to the upstream container registry you can do it with:

    <CodeBlock language="bash">
      {`
            docker run -p 8080:8080 -d \\\n  --name image-factory \\\n  -v $PWD/image-factory.yaml:/image-factory.yaml:ro,Z \\\n  -v $PWD/signing-key.key:/signing-key.key:ro,Z \\\n  ghcr.io/siderolabs/image-factory:${image_factory_release} \\\n    --config image-factory.yaml
            `}
    </CodeBlock>

    This will run the image factory on your machine on port 8080 and automatically pull container images from Sidero's registry. It will also validate image signatures using [`cosign`](https://edu.chainguard.dev/open-source/sigstore/cosign/an-introduction-to-cosign/) to validate pulled images.

    This will not allow you to create or publish custom system extensions.
    To do that you will need to run your own container registry with the necessary images. See the **disconnected** instructions for Image Factory.
  </Tab>

  <Tab title="Image Factory air-gapped">
    <Note>Run with an internal container registry if your machine is not connected to the internet, or you need custom Talos images and extensions.</Note>

    Running the image factory in an air-gapped environment has more requirements than running in a connected mode. Make sure you have a registry running from the [Internal container registry](#container-registry) section. You will need to download container images and seed them into the internal registry and sign the container images.

    > This is just an example and should not be used in a production environment. If you want to test locally on your machine you can also see the [developer documentation](https://github.com/siderolabs/image-factory#air-gapped-mode) in the repository.

    ### Download container images

    Starting with Talos 1.12 you can get a list of images needed to seed the image factory directly from `talosctl`.

    Get a list of Talos base images needed for the image factory with:

    <CodeBlock lang="sh">
      {`talosctl image talos-bundle ${release} > images.txt`}
    </CodeBlock>

    This will give you a list of all images and extensions for Talos {release}.
    You will need to repeat this command for each version of Talos you want to download images for.
    If you don't need specific extensions you can delete them from the `images.txt` file.

    Push the images to your `$REGISTRY_ENDPOINT`

    <Tabs>
      <Tab title="Internet available">
        If your machine can reach the public internet and the internal registry at the same time you can copy the images internally with this command.

        ```bash theme={null}
        for SOURCE_IMAGE in $(cat images.txt)
          do
            IMAGE_WITHOUT_DIGEST=${SOURCE_IMAGE%%@*}
            IMAGE_WITH_NEW_REG="${REGISTRY_ENDPOINT}/${IMAGE_WITHOUT_DIGEST#*/}"
            crane copy \
              $SOURCE_IMAGE \
              $IMAGE_WITH_NEW_REG
        done
        ```
      </Tab>

      <Tab title="Air-gapped">
        If you don't have direct access to an internal container registry (e.g. air gapped environment) you need to download the container images while connected to the internet.

        Download all images from `images.txt`. Create file names that don't contain special characters.

        ```bash theme={null}
        cat images.txt \
          | talosctl images cache-create \
              --layout flat \
              --image-cache-path ./image-cache \
              --images=-
        ```

        Move to a network that has access to the `$REGISTRY_ENDPOINT` endpoint and push the local images to the registry.
        This serves the container layers on your local admin machine port `:5000` so you can push them with crane.
        Run this command command from your admin machine so you do not have port conflicts with the `$REGISTRY_ENDPOINT`.

        <Info>This method of copying containers is important because Talos releases pin to a container digest which needs to match in your internal registry</Info>

        ```bash theme={null}
        export IP=$(hostname -I | awk '{print $1}')

        talosctl image cache-cert-gen \
          --advertised-address $IP

        talosctl image cache-serve \
          --address $IP:5000 \
          --image-cache-path ./image-cache \
          --tls-cert-file tls.crt \
          --tls-key-file tls.key &
        ```

        Push the container images with `crane`.

        ```bash theme={null}
        for SOURCE_IMAGE in $(cat images.txt)
          do
            IMAGE_WITHOUT_DIGEST=${SOURCE_IMAGE%%@*}
            IMAGE_WITH_NEW_REG="${REGISTRY_ENDPOINT}/${IMAGE_WITHOUT_DIGEST#*/}"
            LOCALHOST_IMAGE="${IP}:5000/${IMAGE_WITHOUT_DIGEST#*/}"
            crane copy --insecure \
              $LOCALHOST_IMAGE \
              $IMAGE_WITH_NEW_REG
        done
        ```
      </Tab>
    </Tabs>

    ### Sign container images

    The Image Factory verifies container image signatures when being used. Generate a cosign singing key and sign each container pushed to the registry.

    <Note>Image factory currently only supports `cosign` v2 signatures.</Note>

    Generate a cosign key.

    ```bash theme={null}
    docker run --rm -it \
      -v $PWD:/keys:ro,Z -w /keys \
      -e COSIGN_PASSWORD="" \
      --user $(id -u):$(id -g) \
      ghcr.io/sigstore/cosign/cosign:v2.6.1 \
      generate-key-pair
    ```

    Sign each image and tag in your internal registry. This will allow the registry to validate images without reaching out to any external services for key validation.

    ```bash theme={null}
    KEY_FILE="cosign.key"
    export COSIGN_PASSWORD="" # Leave empty for empty password
    ```

    Sign all of the images using the `images.txt` file as a list.

    > If your registry is running with a self-signed CA certificate (i.e. from the [Installing Airgapped Omni](./run-omni-airgapped) guide) you need to mount the CA certificate into the cosign container for it to be trusted.

    ```bash theme={null}
    for IMAGE in $(cat images.txt)
      do
        NEW_IMAGE="${REGISTRY_ENDPOINT}/${IMAGE#*/}"
        if [[ "$NEW_IMAGE" != *"@sha256:"* ]]; then
          NEW_IMAGE="${NEW_IMAGE}@$(crane digest $NEW_IMAGE)"
        fi
        docker run --rm -it --net=host \
          -v $PWD:/keys -w /keys \
          -v "$PWD/ca.pem:/etc/ssl/certs/ca-certificates.crt:ro,Z" \
          -e COSIGN_PASSWORD="" \
          -e COSIGN_YES=true \
          --user $(id -u):$(id -g) \
          ghcr.io/sigstore/cosign/cosign:v2.6.1 \
            sign --key /keys/$KEY_FILE \
            --tlog-upload=false \
            $NEW_IMAGE
    done
    ```

    <Info>This guide assumes the container registry and image factory are running on the same machine. Because of this we will run the Image Factory with `--net=host` which is not recommended for a production, multi-host deployment.</Info>

    <Tabs>
      <Tab title="Self-signed CA">
        Set a internal factory endpoint.

        ```bash theme={null}
        FACTORY_URL=https://factory.internal:8080
        ```

        Create a configuration file for the Image Factory.

        ```bash theme={null}
        cat <<EOF > image-factory.yaml
        artifacts:
          core:
            registry: $REGISTRY_ENDPOINT
          schematic:
            registry: $REGISTRY_ENDPOINT
          installer:
            internal:
              registry: $REGISTRY_ENDPOINT
            external:
              registry: $REGISTRY_ENDPOINT
              repository: "talos"
              namespace: "siderolabs"
        containerSignature:
          publicKeyFile: /cosign.pub
          subjectRegExp: ""
        cache:
          oci:
            registry: $REGISTRY_ENDPOINT
          signingKeyPath: "/signing-key.key"
        http:
          externalURL: $FACTORY_URL
          certFile: "/certs/server-chain.pem"
          keyFile: "/certs/server-key.pem"
        EOF
        ```

        To run the image factory with a self-signed CA certificate you need to mount them into the container image at run time.

        <CodeBlock language="bash">
          {`
                    docker run -p 8080:8080 -d \\\n  --name image-factory \\\n  --net=host \\\n  -v $PWD/image-factory.yaml:/image-factory.yaml:ro,Z \\\n  -v $PWD/signing-key.key:/signing-key.key:ro,Z \\\n  -v $PWD/cosign.pub:/cosign.pub:ro,Z \\\n  -v $PWD/server-chain.pem:/certs/server-chain.pem:ro,Z \\\n  -v $PWD/server-key.pem:/certs/server-key.pem:ro,Z \\\n  -v $PWD/ca.pem:/etc/ssl/certs/ca.pem:ro,Z \\\n  ghcr.io/siderolabs/image-factory:${image_factory_release} \\\n    --config /image-factory.yaml
                    `}
        </CodeBlock>

        You should now be able to browse to [https://registry.internal:8080](https://registry.internal:8080) and view the Image Factory web interface. If your server or network has any firewall rules you may need to allow TCP traffic to the host.
      </Tab>

      <Tab title="Insecure">
        Set a internal factory endpoint.

        ```bash theme={null}
        FACTORY_URL=http://factory.internal:8080
        ```

        Create a configuration for the Image Factory

        ```bash theme={null}
        cat <<EOF > image-factory.yaml
        artifacts:
          core:
            registry: $REGISTRY_ENDPOINT
            insecure: true
          schematic:
            registry: $REGISTRY_ENDPOINT
            insecure: true
          installer:
            internal:
              registry: $REGISTRY_ENDPOINT
              insecure: true
            external:
              registry: $REGISTRY_ENDPOINT
              insecure: true
        containerSignature:
          publicKeyFile: /cosign.pub
          subjectRegExp: ""
        cache:
          oci:
            registry: $REGISTRY_ENDPOINT
            insecure: true
          signingKeyPath: "/signing-key.key"
        http:
          externalURL: $FACTORY_URL
        EOF
        ```

        If your image factory and container registry do not have certificates run the following command:

        <CodeBlock language="bash">
          {`
                    docker run -p 8080:8080 -d \\\n  --name image-factory \\\n  --net=host \\\n  -v $PWD/image-factory.yaml:/image-factory.yaml:ro,Z \\\n  -v $PWD/signing-key.key:/signing-key.key:ro,Z \\\n  -v $PWD/cosign.pub:/cosign.pub:ro,Z \\\n ghcr.io/siderolabs/image-factory:${image_factory_release} \\\n    --config /image-factory.yaml
                    `}
        </CodeBlock>

        You should now be able to browse to [http://registry.internal:8080](http://registry.internal:8080) and view the Image Factory web interface. If your server or network has any firewall rules you may need to allow TCP traffic to the host.
      </Tab>
    </Tabs>
  </Tab>
</Tabs>

## Run Omni

After the image factory is running you can continue to the [Omni Airgapped documentation](./run-omni-airgapped).
