# Building Custom Talos Images Source: https://docs.siderolabs.com/talos/v1.12/build-and-extend-talos/custom-images-and-development/building-images How to build a custom Talos image from source. There might be several reasons to build Talos images from source: * verifying the [image integrity](../../security/verifying-images) * building an image with custom configuration ## Checkout Talos source ```bash theme={null} git clone https://github.com/siderolabs/talos.git ``` If building for a specific release, checkout the corresponding tag: ## Set up the build environment See [Developing Talos](./developing-talos) for details on setting up the buildkit builder. ## Architectures By default, Talos builds for `linux/amd64`, but you can customize that by passing `PLATFORM` variable to `make`: ```bash theme={null} make PLATFORM=linux/arm64 # build for arm64 only make PLATFORM=linux/arm64,linux/amd64 # build for arm64 and amd64, container images will be multi-arch ``` ## Custom `PKGS` When [customizing Linux kernel](./customizing-the-kernel), the source for the [`siderolabs/pkgs`](https://github.com/siderolabs/pkgs) repository can be overridden with: * if you built and pushed only a custom `kernel` package, the reference can be overridden with `PKG_KERNEL` variable: `make PKG_KERNEL=//kernel:` * if any other single package was customized, the reference can be overridden with `PKG_` (e.g. `PKG_IPTABLES`) variable: `make PKG_=//:` * if the full `pkgs` repository was built and pushed, the references can be overridden with `PKGS_PREFIX` and `PKGS` variables: `make PKGS_PREFIX=/ PKGS=` ## Customizations Some of the build parameters can be customized by passing environment variables to `make`, e.g. `GOAMD64=v1` can be used to build Talos images compatible with old AMD64 CPUs: ```bash theme={null} make GOAMD64=v1 ``` ## Building kernel and initramfs The most basic boot assets can be built with: ```bash theme={null} make kernel initramfs ``` Build result will be stored as `_out/vmlinuz-` and `_out/initramfs-.xz`. ## Building container images Talos container images should be pushed to the registry as the result of the build process. The default settings are: * `IMAGE_REGISTRY` is set to `ghcr.io` * `USERNAME` is set to the `siderolabs` (or value of environment variable `USERNAME` if it is set) The image can be pushed to any registry you have access to, but the access credentials should be stored in `~/.docker/config.json` file (e.g. with `docker login`). Building and pushing the image can be done with: ```bash theme={null} make installer-base PUSH=true IMAGE_REGISTRY=docker.io USERNAME= # ghcr.io/siderolabs/installer-base make imager PUSH=true IMAGE_REGISTRY=docker.io USERNAME= # ghcr.io/siderolabs/imager make installer IMAGE_REGISTRY=docker.io USERNAME= # ghcr.io/siderolabs/installer ``` The [local registry](./developing-talos) running on `127.0.0.1:5005` can be used as well to avoid pushing/pulling over the network: ```bash theme={null} make installer REGISTRY=127.0.0.1:5005 ``` When building `imager` container, by default Talos will include the boot assets for both `amd64` and `arm64` architectures, if building only for single architecture, specify `INSTALLER_ARCH` variable: ```bash theme={null} make imager INSTALLER_ARCH=targetarch PLATFORM=linux/amd64 ``` ## Building ISO The [ISO image](../../platform-specific-installations/boot-assets) is built with the help of `imager` container image, by default `ghcr.io/siderolabs/imager` will be used with the matching tag: ```bash theme={null} make iso ``` The ISO image will be stored as `_out/talos-.iso`. If ISO image should be built with the custom `imager` image, it can be specified with `IMAGE_REGISTRY`/`USERNAME` variables: ```bash theme={null} make iso IMAGE_REGISTRY=docker.io USERNAME= ``` ## Building disk images The disk image is built with the help of `imager` container image, by default `ghcr.io/siderolabs/imager` will be used with the matching tag: ```bash theme={null} make image-metal ``` Available disk images are encoded in the `image-%` target, e.g. `make image-aws`. Same as with ISO image, the custom `imager` image can be specified with `IMAGE_REGISTRY`/`USERNAME` variables. # Customizing the Kernel Source: https://docs.siderolabs.com/talos/v1.12/build-and-extend-talos/custom-images-and-development/customizing-the-kernel Guide on how to customize the kernel used by Talos Linux. Talos Linux configures the kernel to allow loading only cryptographically signed modules. The signing key is generated during the build process, it is unique to each build, and it is not available to the user. The public key is embedded in the kernel, and it is used to verify the signature of the modules. So if you want to use a custom kernel module, you will need to build your own kernel, and all required kernel modules in order to get the signature in sync with the kernel. ## Overview In order to build a custom kernel (or a custom kernel module), the following steps are required: * build a new Linux kernel and modules, push the artifacts to a registry * build a new Talos base artifacts: kernel and initramfs image * produce a new Talos boot artifact (ISO, installer image, disk image, etc.) We will go through each step in detail. ## Building a custom kernel First, you might need to prepare the build environment, follow the [Building Custom Images](./building-images) guide. Checkout the [`siderolabs/pkgs`](https://github.com/siderolabs/pkgs) repository: The kernel configuration is located in the files `kernel/build/config-ARCH` files. It can be modified using the text editor, or by using the Linux kernel `menuconfig` tool: ```shell theme={null} make kernel-menuconfig ``` The kernel configuration can be cleaned up by running: ```shell theme={null} make kernel-olddefconfig ``` Both commands will output the new configuration to the `kernel/build/config-ARCH` files. Once ready, build the kernel any out-of-tree modules (if required, e.g. `zfs`) and push the artifacts to a registry: ```shell theme={null} make kernel REGISTRY=127.0.0.1:5005 PUSH=true ``` By default, this command will compile and push the kernel both for `amd64` and `arm64` architectures, but you can specify a single architecture by overriding a variable `PLATFORM`: ```shell theme={null} make kernel REGISTRY=127.0.0.1:5005 PUSH=true PLATFORM=linux/amd64 ``` This will create a container image `127.0.0.1:5005/siderolabs/kernel:$TAG` with the kernel and modules. ## Building Talos base artifacts Follow the [Building Custom Images](./building-images) guide to set up the Talos source code checkout. If some new kernel modules were introduced, adjust the list of the default modules compiled into the Talos `initramfs` by editing the file `hack/modules-ARCH.txt`. Try building base Talos artifacts: ```shell theme={null} make kernel initramfs PKG_KERNEL=127.0.0.1:5005/siderolabs/kernel:$TAG PLATFORM=linux/amd64 ``` This should create a new image of the kernel and initramfs in `_out/vmlinuz-amd64` and `_out/initramfs-amd64.xz` respectively. > Note: if building for `arm64`, replace `amd64` with `arm64` in the commands above. As a final step, produce the new `imager` container image which can generate Talos boot assets: ```shell theme={null} make imager PKG_KERNEL=127.0.0.1:5005/siderolabs/kernel:$TAG PLATFORM=linux/amd64 INSTALLER_ARCH=targetarch ``` > Note: if you built the kernel for both `amd64` and `arm64`, a multi-arch `imager` container can be built as well by specifying `INSTALLER_ARCH=all` and `PLATFORM=linux/amd64,linux/arm64`. ## Building Talos boot assets Follow the [Boot Assets](../../platform-specific-installations/boot-assets) guide to build Talos boot assets you might need to boot Talos: ISO, `installer` image, etc. Replace the reference to the `imager` in guide with the reference to the `imager` container built above. > Note: if you update the `imager` container, don't forget to `docker pull` it, as `docker` caches pulled images and won't pull the updated image automatically. # Developing Talos Source: https://docs.siderolabs.com/talos/v1.12/build-and-extend-talos/custom-images-and-development/developing-talos Learn how to set up a development environment for local testing and hacking on Talos itself! This guide outlines steps and tricks to develop Talos operating systems and related components. The guide assumes macOS or a Linux operating system on the development host. ## Prepare Check out the [Talos repository](https://github.com/siderolabs/talos). Try running `make help` to see available `make` commands. You would need Docker and `buildx` installed on the host. > Note: Usually it is better to install up to date Docker from Docker apt repositories, e.g. [Ubuntu instructions](https://docs.docker.com/engine/install/ubuntu/). > > If `buildx` plugin is not available with OS docker packages, it can be installed [as a plugin from GitHub releases](https://docs.docker.com/buildx/working-with-buildx/#install). Set up a builder with access to the host network: ```bash theme={null} docker buildx create --driver docker-container --driver-opt network=host --name local1 --buildkitd-flags '--allow-insecure-entitlement security.insecure' --use ``` > Note: `network=host` allows buildx builder to access host network, so that it can push to a local container registry (see below). Make sure the following steps work: * `make talosctl` * `make initramfs kernel` Set up a local docker registry: ```bash theme={null} docker run -d -p 5005:5000 \ --restart always \ --name local registry:2 ``` Try to build and push to local registry an installer image: ```bash theme={null} make installer-base IMAGE_REGISTRY=127.0.0.1:5005 PUSH=true make imager IMAGE_REGISTRY=127.0.0.1:5005 PUSH=true INSTALLER_ARCH=targetarch make installer IMAGE_REGISTRY=127.0.0.1:5005 PUSH=true ``` Record the image name output in the step above. > Note: it is also possible to force a stable image tag by using `TAG` variable: `make installer-base IMAGE_REGISTRY=127.0.0.1:5005 TAG=v1.0.0-alpha.1 PUSH=true`. ## Running Talos cluster Set up local caching docker registries (this speeds up Talos cluster boot a lot), script is in the Talos repo: ```bash theme={null} bash hack/start-registry-proxies.sh ``` Start your local cluster with: ```bash theme={null} sudo --preserve-env=HOME _out/talosctl- cluster create \ --provisioner=qemu \ --cidr=172.20.0.0/24 \ --registry-mirror docker.io=http://172.20.0.1:5000 \ --registry-mirror registry.k8s.io=http://172.20.0.1:5001 \ --registry-mirror gcr.io=http://172.20.0.1:5003 \ --registry-mirror ghcr.io=http://172.20.0.1:5004 \ --registry-mirror 127.0.0.1:5005=http://172.20.0.1:5005 \ --install-image=127.0.0.1:5005/siderolabs/installer: \ --controlplanes 3 \ --workers 2 \ --with-bootloader=false ``` * `--provisioner` selects QEMU vs. default Docker * custom `--cidr` to make QEMU cluster use different network than default Docker setup (optional) * `--registry-mirror` uses the caching proxies set up above to speed up boot time a lot, last one adds your local registry (installer image was pushed to it) * `--install-image` is the image you built with `make installer` above * `--controlplanes` & `--workers` configure cluster size, choose to match your resources; 3 controlplanes give you HA control plane; 1 controlplane is enough, never do 2 controlplanes * `--with-bootloader=false` disables boot from disk (Talos will always boot from `_out/vmlinuz-` and `_out/initramfs-.xz`). This speeds up development cycle a lot - no need to rebuild installer and perform an install, rebooting is enough to get new code changes. > Note: when configuration changes are introduced and the old installer doesn't validate the config, or the installation flow itself is being worked on `--with-bootloader=false` should not be used > > `talosctl cluster create` derives Talos machine configuration version from the install image tag, so sometimes early in the development cycle (when new minor tag is not released yet), machine config version can be overridden with `--talos-version=${version_v1_12}`. ## Console logs Watching console logs is easy with `tail`: ```bash theme={null} tail -F ~/.talos/clusters/talos-default/talos-default-*.log ``` ## Interacting with Talos Once `talosctl cluster create` finishes successfully, `talosconfig` and `kubeconfig` will be set up automatically to point to your cluster. Start playing with `talosctl`: ```bash theme={null} talosctl -n 172.20.0.2 version talosctl -n 172.20.0.3,172.20.0.4 dashboard talosctl -n 172.20.0.4 get members ``` Same with `kubectl`: ```bash theme={null} kubectl get nodes -o wide ``` You can deploy some Kubernetes workloads to the cluster. You can edit machine config on the fly with `talosctl edit mc --immediate`, config patches can be applied via `--config-patch` flags, also many features have specific flags in `talosctl cluster create`. ## Quick reboot To reboot whole cluster quickly (e.g. to pick up a change made in the code): ```bash theme={null} for socket in ~/.talos/clusters/talos-default/talos-default-*.monitor; do echo "q" | sudo socat - unix-connect:$socket; done ``` Sending `q` to a single socket allows to reboot a single node. > Note: This command performs immediate reboot (as if the machine was powered down and immediately powered back up), for normal Talos reboot use `talosctl reboot`. ## Development cycle Fast development cycle: * bring up a cluster * make code changes * rebuild `initramfs` with `make initramfs` * reboot a node to pick new `initramfs` * verify code changes * more code changes... Some aspects of Talos development require to enable bootloader (when working on `installer` itself), in that case quick development cycle is no longer possible, and cluster should be destroyed and recreated each time. ## Running integration tests If integration tests were changed (or when running them for the first time), first rebuild the integration test binary: ```bash theme={null} rm -f _out/integration-test-linux-amd64; make _out/integration-test-linux-amd64 ``` Running short tests against QEMU provisioned cluster: ```bash theme={null} _out/integration-test-linux-amd64 \ -talos.provisioner=qemu \ -test.v \ -test.short \ -talos.talosctlpath=$PWD/_out/talosctl-linux-amd64 ``` Whole test suite can be run removing `-test.short` flag. Specfic tests can be run with `-test.run=TestIntegration/api.ResetSuite`. ## Build flavors `make WITH_RACE=1` enables Go race detector, Talos runs slower and uses more memory, but memory races are detected. `make WITH_DEBUG=1` enables Go profiling and other debug features, useful for local development. `make initramfs WITH_DEBUG_SHELL=true` adds bash and minimal utilities for debugging purposes. Combine with `--with-debug-shell` flag when creating cluster to obtain shell access. This is uncommonly used as in this case the bash shell will run in place of machined. ## Destroying cluster ```bash theme={null} sudo --preserve-env=HOME ../talos/_out/talosctl-linux-amd64 cluster destroy --provisioner=qemu ``` This command stops QEMU and helper processes, tears down bridged network on the host, and cleans up cluster state in `~/.talos/clusters`. > Note: if the host machine is rebooted, QEMU instances and helpers processes won't be started back. > In that case it's required to clean up files in `~/.talos/clusters/` directory manually. ## Optional Set up cross-build environment with: ```bash theme={null} docker run --rm --privileged multiarch/qemu-user-static --reset -p yes ``` > Note: the static qemu binaries which come with Ubuntu 21.10 seem to be broken. ## Unit tests Unit tests can be run in buildx with `make unit-tests`, on Ubuntu systems some tests using `loop` devices will fail because Ubuntu uses low-index `loop` devices for snaps. Most of the unit-tests can be run standalone as well, with regular `go test`, or using IDE integration: ```bash theme={null} go test -v ./internal/pkg/circular/ ``` This provides much faster feedback loop, but some tests require either elevated privileges (running as `root`) or additional binaries available only in Talos `rootfs` (containerd tests). Running tests as root can be done with `-exec` flag to `go test`, but this is risky, as test code has root access and can potentially make undesired changes: ```bash theme={null} go test -exec sudo -v ./internal/app/machined/pkg/controllers/network/... ``` ## Go profiling Build `initramfs` with debug enabled: `make initramfs WITH_DEBUG=1`. Launch Talos cluster with bootloader disabled, and use `go tool pprof` to capture the profile and show the output in your browser: ```bash theme={null} go tool pprof http://172.20.0.2:9982/debug/pprof/heap ``` The IP address `172.20.0.2` is the address of the Talos node, and port `:9982` depends on the Go application to profile: * 9981: `apid` * 9982: `machined` * 9983: `trustd` ## Testing air-gapped environments There is a hidden `talosctl debug air-gapped` command which launches two components: * HTTP proxy capable of proxying HTTP and HTTPS requests * HTTPS server with a self-signed certificate The command also writes down Talos machine configuration patch to enable the HTTP proxy and add a self-signed certificate to the list of trusted certificates: ```shell theme={null} $ talosctl debug air-gapped --advertised-address 172.20.0.1 2022/08/04 16:43:14 writing config patch to air-gapped-patch.yaml 2022/08/04 16:43:14 starting HTTP proxy on :8002 2022/08/04 16:43:14 starting HTTPS server with self-signed cert on :8001 ``` The `--advertised-address` should match the bridge IP of the Talos node. Generated machine configuration patch looks like: ```yaml theme={null} machine: env: http_proxy: http://172.20.0.1:8002 https_proxy: http://172.20.0.1:8002 no_proxy: 172.20.0.1/24 cluster: extraManifests: - https://172.20.0.1:8001/debug.yaml --- apiVersion: v1alpha1 kind: TrustedRootsConfig name: air-gapped-ca certificates: | -----BEGIN CERTIFICATE----- MIIBiTCCAS+gAwIBAgIBATAKBggqhkjOPQQDAjAUMRIwEAYDVQQKEwlUZXN0IE9u bHkwHhcNMjUwMTE1MTE1OTI3WhcNMjUwMTE2MTE1OTI3WjAUMRIwEAYDVQQKEwlU ZXN0IE9ubHkwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAReznBeEcQFcB/y1yqI HQcP0IWBMvgwGTeaaTBM6rV+AjbnyxgCrXAnmJ0t45Eur27eW9J/1T5tzA6fe24f YyY9o3IwcDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG AQUFBwMCMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEGBbafXsyzxVhVqfjzy 7aBmVvtaMA8GA1UdEQQIMAaHBKwUAAEwCgYIKoZIzj0EAwIDSAAwRQIhAPAFm6Lv 1Bw+M55Z1SEDLyILJSS0En5F6n8Q9LyGGT4fAiBi+Fm3wSQcvgGPG9OfokFaXmGp Pa6c4ZrarKO8ZxWigA== -----END CERTIFICATE----- ``` The first section appends a self-signed certificate of the HTTPS server to the list of trusted certificates, followed by the HTTP proxy setup (in-cluster traffic is excluded from the proxy). The last section adds an extra Kubernetes manifest hosted on the HTTPS server. The machine configuration patch can now be used to launch a test Talos cluster: ```shell theme={null} talosctl cluster create ... --config-patch @air-gapped-patch.yaml ``` The following lines should appear in the output of the `talosctl debug air-gapped` command: * `CONNECT discovery.talos.dev:443`: the HTTP proxy is used to talk to the discovery service * `http: TLS handshake error from 172.20.0.2:53512: remote error: tls: bad certificate`: an expected error on Talos side, as self-signed cert is not written yet to the file * `GET /debug.yaml`: Talos successfully fetches the extra manifest successfully There might be more output depending on the registry caches being used or not. ## Running upgrade integration tests Talos has a separate set of provision upgrade tests, which create a cluster on older versions of Talos, perform an upgrade, and verify that the cluster is still functional. Build the test binary: ```bash theme={null} rm -f _out/integration-test-provision-linux-amd64; make _out/integration-test-provision-linux-amd64 ``` Prepare the test artifacts for the upgrade test: ```bash theme={null} make release-artifacts ``` Build and push an installer image for the development version of Talos: ```bash theme={null} make installer-base IMAGE_REGISTRY=127.0.0.1:5005 PUSH=true make imager IMAGE_REGISTRY=127.0.0.1:5005 PUSH=true make installer IMAGE_REGISTRY=127.0.0.1:5005 ``` Run the tests (the tests will create the cluster on the older version of Talos, perform an upgrade, and verify that the cluster is still functional): ```bash theme={null} sudo --preserve-env=HOME _out/integration-test-provision-linux-amd64 \ -test.v \ -talos.talosctlpath _out/talosctl-linux-amd64 \ -talos.provision.target-installer-registry=127.0.0.1:5005 \ -talos.provision.registry-mirror 127.0.0.1:5005=http://172.20.0.1:5005,docker.io=http://172.20.0.1:5000,registry.k8s.io=http://172.20.0.1:5001,quay.io=http://172.20.0.1:5002,gcr.io=http://172.20.0.1:5003,ghcr.io=http://172.20.0.1:5004 \ -talos.provision.cidr 172.20.0.0/24 ``` ## SELinux policy debugging and development Here are some tips about how Talos SELinux policy is built, which should mainly help developers troubleshoot denials and assess policy rules for security against different threats. ### Obtaining and processing denial logs If SELinux has blocked some event from happening, it will log it to the audit log. If the mode is permissive, the only implication of would be a denial message, so permissive mode is useful for prototyping the policy. You can check the logs with: `talosctl --nodes 172.20.0.2 logs auditd > audit.log` The obtained logs can be processed with `audit2allow` to obtain a CIL code that would allow the denied event to happen, alongside an explanation of the denial. For this we use SELinux userspace utilities, which can be ran in a container for cases you use a Linux system without SELinux or another OS. Some of the useful commands are: ```bash theme={null} audit2why -p ./internal/pkg/selinux/policy/policy.33 -i audit.log audit2allow -C -e -p ./internal/pkg/selinux/policy/policy.33 -i audit.log ``` However, please do not consider the output of `audit2allow` as a final modification for the policy. It is a good starting point to understand the denial, but the generated code should be reviewed and correctly reformulated once confirmed to be needed and not caused by mislabeling. ### Iterating on the policy `make generate` generates the compiled SELinux files. However, if you want to iterate on the policy rapidly, you might want to consider only rebuilding the policy during the testing: ```bash theme={null} make local-selinux-generate DEST=./internal/pkg/selinux PLATFORM=linux/amd64 PROGRESS=plain ``` ### Debugging locally with many denials happening Sometimes, e.g. during a major refactor, the policy can be broken and many denials can happen. This can cause the audit ring buffer to fill up, losing some messages. These are some kernel cmdline parameters that redirect the audit logs to the console, which is saved to your development cluster directory: `talos.auditd.disabled=1 audit=1 audit_backlog_limit=65535 debug=1 sysctl.kernel.printk_ratelimit=0 sysctl.kernel.printk_delay=0 sysctl.kernel.printk_ratelimit_burst=10000` ### SELinux policy structure The SELinux policy is built using the CIL language. The CIL files are located in `internal/pkg/selinux/policy/selinux` and are compiled into a binary format (e.g. `33` for the current kernel policy format version) using the `secilc` tool from Talos tools bundle. The policy is embedded into the initramfs init and loaded early in the boot process. For understanding and modifying the policy, [CIL language reference](https://github.com/SELinuxProject/selinux-notebook/blob/dfabf5f1bcdc72e440c1f7010e39ae3ce9f0c364/src/notebook-examples/selinux-policy/cil/CIL_Reference_Guide.pdf) is a recommended starting point to get familiar with the language. [Object Classes and Permissions](https://github.com/SELinuxProject/selinux-notebook/blob/dfabf5f1bcdc72e440c1f7010e39ae3ce9f0c364/src/object_classes_permissions.md) is another helpful document, listing all SELinux entities and the meaning of all the permissions. The policy directory contains the following main subdirectories: * `immutable`: contains the preamble parts, mostly listing SELinux SIDs, classes, policy capabilities and roles, not expected to change frequently. * `common`: abstractions and common rules, which are used by the other parts of the policy or by all objects of some kind.: * classmaps: contains class maps, which are a SELinux concept for easily configuring the same list of permissions on a list of classes. Our policy frequently uses `fs_classes` classmap for enabling a group of file operations on all types of files. * files: labels for common system files, stored on squashfs. Mostly used for generalized labels not related to a particular service. * network: rules that allow basically any network activity, as Talos does not currently use SELinux features like IPsec labeling for network security. * typeattributes: this file contains typeattributes, which are a SELinux concept for grouping types together to have the same rules applied to all of them. This file also contains macros used to assign objects into typeattributes. When such a macro exists its use is recommended over using the typeattribute directly, as it allows for grepping for the macro call. * processes: common rules, applied to all processes or typeattribute of processes. We only add rules that apply widely here, with more specific rules being added to the service policy files. * `services`: policy files for each service. These files contain the definitions and rules that are specific to the service, like allowing access to its configuration files or communicating over sockets. Some specific parts not being a service in the Talos terms are: * `selinux` - selinuxfs rules protecting SELinux settings from modifications after the OS has started. * `system-containerd` - a containerd instance used for `apid` and similar services internal to Talos. * `system-containers` - `apid`, `trustd`, `etcd` and other system services, running in system containerd instance. #### classmaps overview * `fs_classes` - contains file classes and their permissions, used for file operations. * `rw` - all operations, except SELinux label management. * `ro` - read-only operations. * others - just a class permission applied to all supported file classes. * `netlink_classes (full)` - full (except security labels) access to all netlink socket classes. * `process_classes` - helpers to allow a wide range of process operations. * `full` - all operations, except ptrace (considered to be a rare requirement, so should be added specifically where needed). * `signal` - send any signal to the target process. #### typeattributes overview * Processes: * `service_p` - system services. * `system_container_p` - containerized system services. * `pod_p` - Kubernetes pods. * `system_p` - kernel, init, system services (not containerized). * `any_p` - any process registered with the SELinux. * Service-specific types and typeattributes in service policy files. * Files: * `common_f` - world-rw files, which can be accessed by any process. * `protected_f` - mostly files used by specific services, not accessible by other processes (except e.g. machined) * `system_f` - files and directories used by the system services, also generally to be specified by precise type and not typeattribute. * `system_socket_f` - sockets used for communication between system services, not accessible by workload processes. * `device_f`: * `common_device_f` - devices not considered protected like GPUs. * `protected_device_f` - protected devices like TPM, watchdog timers. * `any_f` - any file registered with the SELinux. * `filesystem_f` - filesystems, generally used for allowing mount operations. * `service_exec_f` - system service executable files. * Service-specific types and typeattributes in service policy files. * General: * `any_f_any_p` - any file or any process, the widest typeattribute. # Extension Services Source: https://docs.siderolabs.com/talos/v1.12/build-and-extend-talos/custom-images-and-development/extension-services Use extension services in Talos Linux. Talos provides a way to run additional system services early in the Talos boot process. Extension services should be included into the Talos root filesystem (e.g. using [system extensions](./system-extensions)). Extension services run as privileged containers with ephemeral root filesystem located in the Talos root filesystem. Extension services can be used to use extend core features of Talos in a way that is not possible via [static pods](../../configure-your-talos-cluster/images-container-runtime/static-pods) or Kubernetes DaemonSets. Potential extension services use-cases: * storage: Open iSCSI, software RAID, etc. * networking: BGP FRR, etc. * platform integration: VMWare open VM tools, etc. ## Configuration Talos on boot scans directory `/usr/local/etc/containers` for `*.yaml` files describing the extension services to run. Format of the extension service config: ```yaml theme={null} name: hello-world container: entrypoint: ./hello-world environment: - XDG_RUNTIME_DIR=/run args: - -f mounts: - # OCI Mount Spec depends: - configuration: true - service: cri - path: /run/machined/machined.sock - network: - addresses - connectivity - hostname - etcfiles - time: true restart: never|always|untilSuccess logToConsole: true|false ``` ### `name` Field `name` sets the service name, valid names are `[a-z0-9-_]+`. The service container root filesystem path is derived from the `name`: `/usr/local/lib/containers/`. The extension service will be registered as a Talos service under an `ext-` identifier. ### `container` * `entrypoint` defines the container entrypoint relative to the container root filesystem (`/usr/local/lib/containers/`) * `environmentFile` (**deprecated**) defines the path to a file containing environment variables, the service waits for the file to exist before starting. Use `ExtensionServiceConfig` instead. * `environment` defines the container environment variables. * `args` defines the additional arguments to pass to the entrypoint * `mounts` defines the volumes to be mounted into the container root #### `container.mounts` The section `mounts` uses the standard OCI spec: ```yaml theme={null} - source: /var/log/audit destination: /var/log/audit type: bind options: - rshared - bind - ro ``` All requested directories will be mounted into the extension service container mount namespace. If the `source` directory doesn't exist in the host filesystem, it will be created (only for writable paths in the Talos root filesystem). #### `container.security` The section `security` follows this example: ```yaml theme={null} maskedPaths: - "/should/be/masked" readonlyPaths: - "/path/that/should/be/readonly" - "/another/readonly/path" writeableRootfs: true writeableSysfs: true rootfsPropagation: shared ``` > * The rootfs is readonly by default unless `writeableRootfs: true` is set. > * The sysfs is readonly by default unless `writeableSysfs: true` is set. > * Masked paths if not set defaults to [containerd defaults](https://github.com/containerd/containerd/tree/main/pkg/oci/spec.go). > Masked paths will be mounted to `/dev/null`. > To set empty masked paths use: > > ```yaml theme={null} > container: > security: > maskedPaths: [] > ``` > > * Read Only paths if not set defaults to [containerd defaults](https://github.com/containerd/containerd/tree/main/pkg/oci/spec.go). > Read-only paths will be mounted to `/dev/null`. > To set empty read only paths use: > > ```yaml theme={null} > container: > security: > readonlyPaths: [] > ``` > > * Rootfs propagation is not set by default (container mounts are private). ### `depends` The `depends` section describes extension service start dependencies: the service will not be started until all dependencies are met. Available dependencies: * `service: `: wait for the service `` to be running and healthy * `path: `: wait for the `` to exist * `network: [addresses, connectivity, hostname, etcfiles]`: wait for the specified network readiness checks to succeed * `time: true`: wait for the NTP time sync * `configuration: true`: wait for `ExtensionServiceConfig` resource with a name matching the extension name to be available. The mounts specified in the `ExtensionServiceConfig` will be added as extra mounts to the extension service. ### `restart` Field `restart` defines the service restart policy, it allows to either configure an always running service or a one-shot service: * `always`: restart service always * `never`: start service only once and never restart * `untilSuccess`: restart failing service, stop restarting on successful run ### `logToConsole` Field `logToConsole` defines whether the service logs should also be written to the console, i.e., to kernel log buffer (or to the container logs in container mode). This feature is particularly useful for debugging extensions that operate in maintenance mode or early in the boot process when service logs cannot be accessed yet. ## Example Example layout of the Talos root filesystem contents for the extension service: ```text theme={null} / └── usr    └── local       ├── etc      │   └── containers       │     └── hello-world.yaml       └── lib           └── containers          └── hello-world          ├── hello └── config.ini ``` Talos discovers the extension service configuration in `/usr/local/etc/containers/hello-world.yaml`: ```yaml theme={null} name: hello-world container: entrypoint: ./hello args: - --config - config.ini depends: - network: - addresses restart: always ``` Talos starts the container for the extension service with container root filesystem at `/usr/local/lib/containers/hello-world`: ```text theme={null} / ├── hello └── config.ini ``` Extension service is registered as `ext-hello-world` in `talosctl services`: ```shell theme={null} $ talosctl service ext-hello-world NODE 172.20.0.5 ID ext-hello-world STATE Running HEALTH ? EVENTS [Running]: Started task ext-hello-world (PID 1100) for container ext-hello-world (2m47s ago) [Preparing]: Creating service runner (2m47s ago) [Preparing]: Running pre state (2m47s ago) [Waiting]: Waiting for service "containerd" to be "up" (2m48s ago) [Waiting]: Waiting for service "containerd" to be "up", network (2m49s ago) ``` An extension service can be started, restarted and stopped using `talosctl service ext-hello-world start|restart|stop`. Use `talosctl logs ext-hello-world` to get the logs of the service. Complete example of the extension service can be found in the [extensions repository](https://github.com/talos-systems/extensions/tree/main/examples/hello-world-service). # Interactive Dashboard Source: https://docs.siderolabs.com/talos/v1.12/deploy-and-manage-workloads/interactive-dashboard A tool to inspect the running Talos machine state on the physical video console. Interactive dashboard is enabled for all Talos platforms except for SBC images. The dashboard can be disabled with kernel parameter `talos.dashboard.disabled=1`.