> ## Documentation Index > Fetch the complete documentation index at: https://docs.siderolabs.com/llms.txt > Use this file to discover all available pages before exploring further. # Logging > Dealing with Talos Linux logs. export const k8s_release = '1.36.1'; export const VersionWarningBanner = () => { const latestVersion = "v1.13"; const [latestUrl, setLatestUrl] = useState(null); const [currentVersion, setCurrentVersion] = useState(null); const [isBeta, setIsBeta] = useState(false); const parseVersion = v => v.replace("v", "").split(".").map(Number); const isGreaterVersion = (a, b) => { const [aMajor, aMinor] = parseVersion(a); const [bMajor, bMinor] = parseVersion(b); if (aMajor > bMajor) return true; if (aMajor === bMajor && aMinor > bMinor) return true; return false; }; useEffect(() => { if (typeof window === "undefined") return; const {pathname, hash, search} = window.location; const match = pathname.match(/\/talos\/(v\d+\.\d+)\//); if (!match) return; const detectedVersion = match[1]; if (detectedVersion === latestVersion) return; setCurrentVersion(detectedVersion); if (isGreaterVersion(detectedVersion, latestVersion)) { setIsBeta(true); } const newPath = pathname.replace(`/talos/${detectedVersion}/`, `/talos/${latestVersion}/`); setLatestUrl(`${newPath}${search}${hash}`); }, []); if (!latestUrl || !currentVersion) return null; return
{isBeta ? <> ⚠️ You are viewing a beta version of Talos ({currentVersion}). This version may be unstable. View latest stable version {latestVersion} → : <> ⚠️ You are viewing an older version of Talos ({currentVersion}). View the latest version {latestVersion} → }
; }; ## Viewing logs Kernel messages can be retrieved with `talosctl dmesg` command: ```sh theme={null} $ talosctl -n 172.20.1.2 dmesg 172.20.1.2: kern: info: [2021-11-10T10:09:37.662764956Z]: Command line: init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512 reboot=k panic=1 talos.shutdown=halt talos.platform=metal talos.config=http://172.20.1.1:40101/config.yaml [...] ``` Service logs can be retrieved with `talosctl logs` command: ```sh theme={null} $ talosctl -n 172.20.1.2 services NODE SERVICE STATE HEALTH LAST CHANGE LAST EVENT 172.20.1.2 apid Running OK 19m27s ago Health check successful 172.20.1.2 containerd Running OK 19m29s ago Health check successful 172.20.1.2 cri Running OK 19m27s ago Health check successful 172.20.1.2 etcd Running OK 19m22s ago Health check successful 172.20.1.2 kubelet Running OK 19m20s ago Health check successful 172.20.1.2 machined Running ? 19m30s ago Service started as goroutine 172.20.1.2 trustd Running OK 19m27s ago Health check successful 172.20.1.2 udevd Running OK 19m28s ago Health check successful $ talosctl -n 172.20.1.2 logs machined 172.20.1.2: [talos] task setupLogger (1/1): done, 106.109µs 172.20.1.2: [talos] phase logger (1/7): done, 564.476µs [...] ``` Container logs for Kubernetes pods can be retrieved with `talosctl logs -k` command: {` $ talosctl -n 172.20.1.2 containers -k NODE NAMESPACE ID IMAGE PID STATUS 172.20.1.2 k8s.io kube-system/kube-flannel-dk6d5 registry.k8s.io/pause:3.6 1329 SANDBOX_READY 172.20.1.2 k8s.io └─ kube-system/kube-flannel-dk6d5:install-cni:f1d4cf68feb9 ghcr.io/siderolabs/install-cni:v0.7.0-alpha.0-1-g2bb2efc 0 CONTAINER_EXITED 172.20.1.2 k8s.io └─ kube-system/kube-flannel-dk6d5:install-config:bc39fec3cbac quay.io/coreos/flannel:v0.13.0 0 CONTAINER_EXITED 172.20.1.2 k8s.io └─ kube-system/kube-flannel-dk6d5:kube-flannel:5c3989353b98 quay.io/coreos/flannel:v0.13.0 1610 CONTAINER_RUNNING 172.20.1.2 k8s.io kube-system/kube-proxy-gfkqj registry.k8s.io/pause:3.5 1311 SANDBOX_READY 172.20.1.2 k8s.io └─ kube-system/kube-proxy-gfkqj:kube-proxy:ad5e8ddc7e7f registry.k8s.io/kube-proxy:v${k8s_release} 1379 CONTAINER_RUNNING $ talosctl -n 172.20.1.2 logs -k kube-system/kube-proxy-gfkqj:kube-proxy:ad5e8ddc7e7f 172.20.1.2: 2021-11-30T19:13:20.567825192Z stderr F I1130 19:13:20.567737 1 server_others.go:138] "Detected node IP" address="172.20.0.3" 172.20.1.2: 2021-11-30T19:13:20.599684397Z stderr F I1130 19:13:20.599613 1 server_others.go:206] "Using iptables Proxier" [...] `} If some host workloads (e.g. system extensions) send syslog messages, they can be retrieved with `talosctl logs syslogd` command. ## Sending logs ### Service logs You can enable logs sendings in machine configuration: ```yaml theme={null} machine: logging: destinations: - endpoint: "udp://127.0.0.1:12345/" format: "json_lines" - endpoint: "tcp://host:5044/" format: "json_lines" ``` Several destinations can be specified. Supported protocols are UDP and TCP. The only currently supported format is `json_lines`: ```json theme={null} { "msg": "[talos] apply config request: immediate true, on reboot false", "talos-level": "info", "talos-service": "machined", "talos-time": "2021-11-10T10:48:49.294858021Z" } ``` Messages are newline-separated when sent over TCP. Over UDP messages are sent with one message per packet. `msg`, `talos-level`, `talos-service`, and `talos-time` fields are always present; there may be additional fields. Every message sent can be enhanced with additional fields by using the `extraTags` field in the machine configuration: ```yaml theme={null} machine: logging: destinations: - endpoint: "udp://127.0.0.1:12345/" format: "json_lines" extraTags: server: s03-rack07 ``` The specified `extraTags` are added to every message sent to the destination verbatim. > `syslog` is considered a service in Talos, and so messages/logs sent to syslog (e.g., by system extensions) are considered > service logs and will be sent to any configured remote receivers without further configuration. ### Kernel logs Kernel log delivery can be enabled with the `talos.logging.kernel` kernel command line argument, which can be specified in the `.machine.installer.extraKernelArgs`: ```yaml theme={null} machine: install: extraKernelArgs: - talos.logging.kernel=tcp://host:5044/ ``` Also kernel logs delivery can be configured using the [document](../../reference/configuration/runtime/kmsglogconfig) in machine configuration: ```yaml theme={null} apiVersion: v1alpha1 kind: KmsgLogConfig name: remote-log url: tcp://host:5044/ ``` Kernel log destination is specified in the same way as service log endpoint. The only supported format is `json_lines`. Sample message: ```json theme={null} { "clock":6252819, // time relative to the kernel boot time "facility":"user", "msg":"[talos] task startAllServices (1/1): waiting for 6 services\n", "priority":"warning", "seq":711, "talos-level":"warn", // Talos-translated `priority` into common logging level "talos-time":"2021-11-26T16:53:21.3258698Z" // Talos-translated `clock` using current time } ``` > `extraKernelArgs` in the machine configuration are only applied on Talos upgrades, not just by applying the config. > (Upgrading to the same version is fine). ### Filebeat example To forward logs to other Log collection services, one way to do this is sending them to a [Filebeat](https://www.elastic.co/beats/filebeat) running in the cluster itself (in the host network), which takes care of forwarding it to other endpoints (and the necessary transformations). If [Elastic Cloud on Kubernetes](https://www.elastic.co/elastic-cloud-kubernetes) is being used, the following Beat (custom resource) configuration might be helpful: ```yaml theme={null} apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: talos spec: type: filebeat version: 7.15.1 elasticsearchRef: name: talos config: filebeat.inputs: - type: "udp" host: "127.0.0.1:12345" processors: - decode_json_fields: fields: ["message"] target: "" - timestamp: field: "talos-time" layouts: - "2006-01-02T15:04:05.999999999Z07:00" - drop_fields: fields: ["message", "talos-time"] - rename: fields: - from: "msg" to: "message" daemonSet: updateStrategy: rollingUpdate: maxUnavailable: 100% podTemplate: spec: dnsPolicy: ClusterFirstWithHostNet hostNetwork: true securityContext: runAsUser: 0 containers: - name: filebeat ports: - protocol: UDP containerPort: 12345 hostPort: 12345 ``` The input configuration ensures that messages and timestamps are extracted properly. Refer to the Filebeat documentation on how to forward logs to other outputs. Also note the `hostNetwork: true` in the `daemonSet` configuration. This ensures filebeat uses the host network, and listens on `127.0.0.1:12345` (UDP) on every machine, which can then be specified as a logging endpoint in the machine configuration. ### Fluent-bit example First, we'll create a value file for the `fluentd-bit` Helm chart. ```yaml theme={null} # fluentd-bit.yaml podAnnotations: fluentbit.io/exclude: 'true' extraPorts: - port: 12345 containerPort: 12345 protocol: TCP name: talos config: service: | [SERVICE] Flush 5 Daemon Off Log_Level warn Parsers_File custom_parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 inputs: | [INPUT] Name tcp Listen 0.0.0.0 Port 12345 Format json Tag talos.* [INPUT] Name tail Alias kubernetes Path /var/log/containers/*.log Parser containerd Tag kubernetes.* [INPUT] Name tail Alias audit Path /var/log/audit/kube/*.log Parser audit Tag audit.* filters: | [FILTER] Name kubernetes Alias kubernetes Match kubernetes.* Kube_Tag_Prefix kubernetes.var.log.containers. Use_Kubelet Off Merge_Log On Merge_Log_Trim On Keep_Log Off K8S-Logging.Parser Off K8S-Logging.Exclude On Annotations Off Labels On [FILTER] Name modify Match kubernetes.* Add source kubernetes Remove logtag customParsers: | [PARSER] Name audit Format json Time_Key requestReceivedTimestamp Time_Format %Y-%m-%dT%H:%M:%S.%L%z [PARSER] Name containerd Format regex Regex ^(?