Skip to main content

​
Viewing logs

Kernel messages can be retrieved with talosctl dmesg command: 
$ talosctl -n 172.20.1.2 dmesg

172.20.1.2: kern:    info: [2021-11-10T10:09:37.662764956Z]: Command line: init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on reboot=k panic=1 talos.shutdown=halt talos.platform=metal talos.config=http://172.20.1.1:40101/config.yaml
[...]
Service logs can be retrieved with talosctl logs command: 
$ talosctl -n 172.20.1.2 services

NODE         SERVICE      STATE     HEALTH   LAST CHANGE   LAST EVENT
172.20.1.2   apid         Running   OK       19m27s ago    Health check successful
172.20.1.2   containerd   Running   OK       19m29s ago    Health check successful
172.20.1.2   cri          Running   OK       19m27s ago    Health check successful
172.20.1.2   etcd         Running   OK       19m22s ago    Health check successful
172.20.1.2   kubelet      Running   OK       19m20s ago    Health check successful
172.20.1.2   machined     Running   ?        19m30s ago    Service started as goroutine
172.20.1.2   trustd       Running   OK       19m27s ago    Health check successful
172.20.1.2   udevd        Running   OK       19m28s ago    Health check successful

$ talosctl -n 172.20.1.2 logs machined

172.20.1.2: [talos] task setupLogger (1/1): done, 106.109µs
172.20.1.2: [talos] phase logger (1/7): done, 564.476µs
[...]
Kernel log is also mirrored as talosctl logs kernel. Container logs for Kubernetes pods can be retrieved with talosctl logs -k command: If some host workloads (e.g. system extensions) send syslog messages, they can be retrieved with talosctl logs syslogd command.

​
Forwarding logs for aggregation

Talos writes logs to files in /var/log directory. A pod running in Kubernetes can mount this directory and forward logs to a log aggregation system.

​
Sending logs over network

​
Service logs

You can enable logs sendings in machine configuration: 
machine:
  logging:
    destinations:
      - endpoint: "udp://127.0.0.1:12345/"
        format: "json_lines"
      - endpoint: "tcp://host:5044/"
        format: "json_lines"
Several destinations can be specified. Supported protocols are UDP and TCP. The only currently supported format is json_lines: 
{
  "msg": "[talos] apply config request: immediate true, on reboot false",
  "talos-level": "info",
  "talos-service": "machined",
  "talos-time": "2021-11-10T10:48:49.294858021Z"
}
Messages are newline-separated when sent over TCP. Over UDP messages are sent with one message per packet. msg, talos-level, talos-service, and talos-time fields are always present; there may be additional fields. Every message sent can be enhanced with additional fields by using the extraTags field in the machine configuration: 
machine:
  logging:
    destinations:
      - endpoint: "udp://127.0.0.1:12345/"
        format: "json_lines"
        extraTags:
          server: s03-rack07
The specified extraTags are added to every message sent to the destination verbatim.
syslog is considered a service in Talos, and so messages/logs sent to syslog (e.g., by system extensions) are considered service logs and will be sent to any configured remote receivers without further configuration.

​
Kernel logs

Kernel log delivery can be enabled with the talos.logging.kernel kernel command line argument, which can be specified in the .machine.installer.extraKernelArgs: 
machine:
  install:
    extraKernelArgs:
      - talos.logging.kernel=tcp://host:5044/
Also kernel logs delivery can be configured using the document in machine configuration: 
apiVersion: v1alpha1
kind: KmsgLogConfig
name: remote-log
url: tcp://host:5044/
Kernel log destination is specified in the same way as service log endpoint. The only supported format is json_lines. Sample message: 
{
  "clock":6252819, // time relative to the kernel boot time
  "facility":"user",
  "msg":"[talos] task startAllServices (1/1): waiting for 6 services\n",
  "priority":"warning",
  "seq":711,
  "talos-level":"warn", // Talos-translated `priority` into common logging level
  "talos-time":"2021-11-26T16:53:21.3258698Z" // Talos-translated `clock` using current time
}
extraKernelArgs in the machine configuration are only applied on Talos upgrades, not just by applying the config. (Upgrading to the same version is fine).

​
Receiving logs

If you have configure remote service logs or kernel logs on a Talos system and want to collect the logs centrally for debugging purposes you can temporarily run the netcat nc command to receive logs. On a Linux host run the following command: 
nc -k -l 5140 | tee -a logs.txt
This will print the logs to standard out and also sove them to a logs.txt file. On the Talos machines make sure to configure logs to send to to your machine’s IP address with a logging configuration as described above. Central logging configuration will allow you to collect logs even when the Talos API is not available (e.g. during installation).