> ## Documentation Index > Fetch the complete documentation index at: https://docs.siderolabs.com/llms.txt > Use this file to discover all available pages before exploring further. # Logging > Dealing with Talos Linux logs. export const k8s_release = '1.36.0'; export const VersionWarningBanner = () => { const latestVersion = "v1.13"; const [latestUrl, setLatestUrl] = useState(null); const [currentVersion, setCurrentVersion] = useState(null); const [isBeta, setIsBeta] = useState(false); const parseVersion = v => v.replace("v", "").split(".").map(Number); const isGreaterVersion = (a, b) => { const [aMajor, aMinor] = parseVersion(a); const [bMajor, bMinor] = parseVersion(b); if (aMajor > bMajor) return true; if (aMajor === bMajor && aMinor > bMinor) return true; return false; }; useEffect(() => { if (typeof window === "undefined") return; const {pathname, hash, search} = window.location; const match = pathname.match(/\/talos\/(v\d+\.\d+)\//); if (!match) return; const detectedVersion = match[1]; if (detectedVersion === latestVersion) return; setCurrentVersion(detectedVersion); if (isGreaterVersion(detectedVersion, latestVersion)) { setIsBeta(true); } const newPath = pathname.replace(`/talos/${detectedVersion}/`, `/talos/${latestVersion}/`); setLatestUrl(`${newPath}${search}${hash}`); }, []); if (!latestUrl || !currentVersion) return null; return
{isBeta ? <> ⚠️ You are viewing a beta version of Talos ({currentVersion}). This version may be unstable. View latest stable version {latestVersion} → : <> ⚠️ You are viewing an older version of Talos ({currentVersion}). View the latest version {latestVersion} → }
; }; ## View logs Kernel messages can be retrieved with `talosctl dmesg` command: ```sh theme={null} $ talosctl -n 172.20.1.2 dmesg 172.20.1.2: kern: info: [2021-11-10T10:09:37.662764956Z]: Command line: init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on reboot=k panic=1 talos.shutdown=halt talos.platform=metal talos.config=http://172.20.1.1:40101/config.yaml [...] ``` Service logs can be retrieved with `talosctl logs` command: ```sh theme={null} $ talosctl -n 172.20.1.2 services NODE SERVICE STATE HEALTH LAST CHANGE LAST EVENT 172.20.1.2 apid Running OK 19m27s ago Health check successful 172.20.1.2 containerd Running OK 19m29s ago Health check successful 172.20.1.2 cri Running OK 19m27s ago Health check successful 172.20.1.2 etcd Running OK 19m22s ago Health check successful 172.20.1.2 kubelet Running OK 19m20s ago Health check successful 172.20.1.2 machined Running ? 19m30s ago Service started as goroutine 172.20.1.2 trustd Running OK 19m27s ago Health check successful 172.20.1.2 udevd Running OK 19m28s ago Health check successful $ talosctl -n 172.20.1.2 logs machined 172.20.1.2: [talos] task setupLogger (1/1): done, 106.109µs 172.20.1.2: [talos] phase logger (1/7): done, 564.476µs [...] ``` Kernel log is also mirrored as `talosctl logs kernel`. Container logs for Kubernetes pods can be retrieved with `talosctl logs -k` command: {` $ talosctl -n 172.20.1.2 containers -k NODE NAMESPACE ID IMAGE PID STATUS 172.20.1.2 k8s.io kube-system/kube-flannel-dk6d5 registry.k8s.io/pause:3.6 1329 SANDBOX_READY 172.20.1.2 k8s.io └─ kube-system/kube-flannel-dk6d5:install-cni:f1d4cf68feb9 ghcr.io/siderolabs/install-cni:v0.7.0-alpha.0-1-g2bb2efc 0 CONTAINER_EXITED 172.20.1.2 k8s.io └─ kube-system/kube-flannel-dk6d5:install-config:bc39fec3cbac quay.io/coreos/flannel:v0.13.0 0 CONTAINER_EXITED 172.20.1.2 k8s.io └─ kube-system/kube-flannel-dk6d5:kube-flannel:5c3989353b98 quay.io/coreos/flannel:v0.13.0 1610 CONTAINER_RUNNING 172.20.1.2 k8s.io kube-system/kube-proxy-gfkqj registry.k8s.io/pause:3.5 1311 SANDBOX_READY 172.20.1.2 k8s.io └─ kube-system/kube-proxy-gfkqj:kube-proxy:ad5e8ddc7e7f registry.k8s.io/kube-proxy:v${k8s_release} 1379 CONTAINER_RUNNING $ talosctl -n 172.20.1.2 logs -k kube-system/kube-proxy-gfkqj:kube-proxy:ad5e8ddc7e7f 172.20.1.2: 2021-11-30T19:13:20.567825192Z stderr F I1130 19:13:20.567737 1 server_others.go:138] "Detected node IP" address="172.20.0.3" 172.20.1.2: 2021-11-30T19:13:20.599684397Z stderr F I1130 19:13:20.599613 1 server_others.go:206] "Using iptables Proxier" [...] `} If some host workloads (e.g. system extensions) send syslog messages, they can be retrieved with `talosctl logs syslogd` command. ## Forward logs for aggregation Talos writes logs to files in `/var/log` directory. A pod running in Kubernetes can mount this directory and forward logs to a log aggregation system. ## Send logs over network ### Service logs You can enable log sending in machine configuration: ```yaml theme={null} machine: logging: destinations: - endpoint: "udp://127.0.0.1:12345/" format: "json_lines" - endpoint: "tcp://host:5044/" format: "json_lines" ``` Several destinations can be specified. Supported protocols are UDP and TCP. The only currently supported format is `json_lines`: ```json theme={null} { "msg": "[talos] apply config request: immediate true, on reboot false", "talos-level": "info", "talos-service": "machined", "talos-time": "2021-11-10T10:48:49.294858021Z" } ``` Messages are newline-separated when sent over TCP. Over UDP messages are sent with one message per packet. `msg`, `talos-level`, `talos-service`, and `talos-time` fields are always present; there may be additional fields. Every message sent can be enhanced with additional fields by using the `extraTags` field in the machine configuration: ```yaml theme={null} machine: logging: destinations: - endpoint: "udp://127.0.0.1:12345/" format: "json_lines" extraTags: server: s03-rack07 ``` The specified `extraTags` are added to every message sent to the destination verbatim. > `syslog` is considered a service in Talos, and so messages/logs sent to syslog (e.g., by system extensions) are considered > service logs and will be sent to any configured remote receivers without further configuration. ### Kernel logs Kernel log delivery can be enabled with the `talos.logging.kernel` kernel command line argument, which can be specified in the `.machine.installer.extraKernelArgs`: ```yaml theme={null} machine: install: extraKernelArgs: - talos.logging.kernel=tcp://host:5044/ ``` Also kernel logs delivery can be configured using the [document](../../reference/configuration/runtime/kmsglogconfig) in machine configuration: ```yaml theme={null} apiVersion: v1alpha1 kind: KmsgLogConfig name: remote-log url: tcp://host:5044/ ``` Kernel log destination is specified in the same way as service log endpoint. The only supported format is `json_lines`. Sample message: ```json theme={null} { "clock":6252819, // time relative to the kernel boot time "facility":"user", "msg":"[talos] task startAllServices (1/1): waiting for 6 services\n", "priority":"warning", "seq":711, "talos-level":"warn", // Talos-translated `priority` into common logging level "talos-time":"2021-11-26T16:53:21.3258698Z" // Talos-translated `clock` using current time } ``` > `extraKernelArgs` in the machine configuration are only applied on Talos upgrades, not just by applying the config. > (Upgrading to the same version is fine). ### Receive logs If you have configured remote service logs or kernel logs on a Talos system and want to collect the logs centrally for debugging purposes you can temporarily run the netcat `nc` command to receive logs. On a Linux host run the following command: ```shell theme={null} nc -k -l 5140 | tee -a logs.txt ``` This will print the logs to standard out and also save them to a logs.txt file. On the Talos machines make sure to configure logs to send to your machine's IP address with a logging configuration as described above. Central logging configuration will allow you to collect logs even when the Talos API is not available (e.g. during installation).