Logging

View logs

Kernel messages can be retrieved with talosctl dmesg command:

$ talosctl -n 172.20.1.2 dmesg

172.20.1.2: kern:    info: [2021-11-10T10:09:37.662764956Z]: Command line: init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on reboot=k panic=1 talos.shutdown=halt talos.platform=metal talos.config=http://172.20.1.1:40101/config.yaml
[...]

Service logs can be retrieved with talosctl logs command:

$ talosctl -n 172.20.1.2 services

NODE         SERVICE      STATE     HEALTH   LAST CHANGE   LAST EVENT
20.1.2   apid         Running   OK       19m27s ago    Health check successful
20.1.2   containerd   Running   OK       19m29s ago    Health check successful
20.1.2   cri          Running   OK       19m27s ago    Health check successful
20.1.2   etcd         Running   OK       19m22s ago    Health check successful
20.1.2   kubelet      Running   OK       19m20s ago    Health check successful
20.1.2   machined     Running   ?        19m30s ago    Service started as goroutine
20.1.2   trustd       Running   OK       19m27s ago    Health check successful
20.1.2   udevd        Running   OK       19m28s ago    Health check successful

$ talosctl -n 172.20.1.2 logs machined

20.1.2: [talos] task setupLogger (1/1): done, 106.109µs
20.1.2: [talos] phase logger (1/7): done, 564.476µs
[...]

Kernel log is also mirrored as talosctl logs kernel. Container logs for Kubernetes pods can be retrieved with talosctl logs -k command: If some host workloads (e.g. system extensions) send syslog messages, they can be retrieved with talosctl logs syslogd command.

Forward logs for aggregation

Talos writes logs to files in /var/log directory. A pod running in Kubernetes can mount this directory and forward logs to a log aggregation system.

Send logs over network

Service logs

You can enable log sending in machine configuration:

machine:
  logging:
    destinations:
      - endpoint: "udp://127.0.0.1:12345/"
        format: "json_lines"
      - endpoint: "tcp://host:5044/"
        format: "json_lines"

Several destinations can be specified. Supported protocols are UDP and TCP. The only currently supported format is json_lines:

{
  "msg": "[talos] apply config request: immediate true, on reboot false",
  "talos-level": "info",
  "talos-service": "machined",
  "talos-time": "2021-11-10T10:48:49.294858021Z"
}

Messages are newline-separated when sent over TCP. Over UDP messages are sent with one message per packet. msg, talos-level, talos-service, and talos-time fields are always present; there may be additional fields. Every message sent can be enhanced with additional fields by using the extraTags field in the machine configuration:

machine:
  logging:
    destinations:
      - endpoint: "udp://127.0.0.1:12345/"
        format: "json_lines"
        extraTags:
          server: s03-rack07

The specified extraTags are added to every message sent to the destination verbatim.

syslog is considered a service in Talos, and so messages/logs sent to syslog (e.g., by system extensions) are considered service logs and will be sent to any configured remote receivers without further configuration.

Kernel logs

Kernel log delivery can be enabled with the talos.logging.kernel kernel command line argument, which can be specified in the .machine.installer.extraKernelArgs:

machine:
  install:
    extraKernelArgs:
      - talos.logging.kernel=tcp://host:5044/

Also kernel logs delivery can be configured using the document in machine configuration:

apiVersion: v1alpha1
kind: KmsgLogConfig
name: remote-log
url: tcp://host:5044/

Kernel log destination is specified in the same way as service log endpoint. The only supported format is json_lines. Sample message:

{
  "clock":6252819, // time relative to the kernel boot time
  "facility":"user",
  "msg":"[talos] task startAllServices (1/1): waiting for 6 services\n",
  "priority":"warning",
  "seq":711,
  "talos-level":"warn", // Talos-translated `priority` into common logging level
  "talos-time":"2021-11-26T16:53:21.3258698Z" // Talos-translated `clock` using current time
}

extraKernelArgs in the machine configuration are only applied on Talos upgrades, not just by applying the config. (Upgrading to the same version is fine).

Receive logs

If you have configured remote service logs or kernel logs on a Talos system and want to collect the logs centrally for debugging purposes you can temporarily run the netcat nc command to receive logs. On a Linux host run the following command:

nc -k -l 5140 | tee -a logs.txt

This will print the logs to standard out and also save them to a logs.txt file. On the Talos machines make sure to configure logs to send to your machine’s IP address with a logging configuration as described above. Central logging configuration will allow you to collect logs even when the Talos API is not available (e.g. during installation).

Overview

Getting Started

Platform specific installation

Deploying and managing workloads

Networking

Security

Build and extend Talos

Configure your Talos cluster

Advanced guides

Reference

Troubleshooting and support

Learn more

View logs

Forward logs for aggregation

Send logs over network

Service logs

Kernel logs

Receive logs

​View logs

​Forward logs for aggregation

​Send logs over network

​Service logs

​Kernel logs

​Receive logs

View logs

Forward logs for aggregation

Send logs over network

Service logs

Kernel logs

Receive logs