OOM Handler

Talos Linux includes a configurable userspace low-memory monitor supplementing Linux kernel built-in OOM killer. This controller enables early detection of heavy memory use and helps prevent machine lock-up due to out-of-memory, which is especially important to enhance the stability of some special cases making the control plane more prone to OOM, such as single-node clusters or scheduling pods on control plane nodes. While the Linux kernel is already capable of handling low-memory situations, the kernel OOM killer only kicks in when the kernel has completely run out of free pages to allocate for a process – at which point a machine is already struggling (or unresponsive), and will take a while to recover. Starting v1.12, Talos Linux includes a userspace OOM controller which is enabled by default and comes pre-configured, however, it is expected that different workloads and hardware configurations might require tuning the OOM controller to further improve robustness. The CEL expression language is used for configuring the Talos OOM controller, under which conditions should it activate, and which cgroups should it prioritize when it does. Configuration reference lists all supported configuration options and a sample configuration document that can be applied to customize OOM controller behavior.

How the OOM handler works

The OOM handler is a controller running as part of machined, the Talos PID 1 process. It is enabled by default and requires no configuration. It is disabled when Talos itself runs in a container (talosctl cluster create with the Docker provisioner). Every sampleInterval the controller runs a single cycle:

Collect memory metrics: pressure stall information for the root cgroup and for each quality of service (QoS) class, plus memory usage of each class.
Evaluate the triggerExpression against those metrics. If it evaluates to false, the cycle ends here and nothing is killed.
Score every candidate cgroup with the cgroupRankingExpression.
Select a single victim cgroup out of the scored candidates.
Kill the victim: every process in it receives SIGKILL (via the cgroup.kill cgroup control file), and the kernel is asked to reclaim their memory immediately with process_mrelease.

The default sampling interval is 500ms. The value shown in the sample document of the configuration reference is an example of an override, not the default. The handler always kills at most one cgroup per cycle, and it kills the cgroup as a whole — never a single process inside it. If no cgroup is eligible, nothing is killed even though the trigger fired. The kernel OOM killer stays enabled and keeps working alongside the Talos OOM handler. The difference is what they react to: the kernel acts when an allocation can no longer be satisfied, while the Talos OOM handler acts on memory pressure — processes spending a significant fraction of their time waiting for memory — which happens well before the machine runs out of pages.

Cgroup classes

Talos assigns every cgroup it monitors to one of five QoS classes. The classes are listed here from the least important to the most important, which is the order in which they are considered for killing:

Class	Cgroups	Killed by the default configuration
`Besteffort`	`kubepods/besteffort/*` — Kubernetes pods of the BestEffort QoS class	yes, first
`Burstable`	`kubepods/burstable/*` — Kubernetes pods of the Burstable QoS class	yes, but only if the pod cgroup has no memory limit
`Guaranteed`	`kubepods/*` — Kubernetes pods of the Guaranteed QoS class	no
`Podruntime`	`podruntime/*` — `containerd`, CRI shims, `kubelet`, `etcd`	no
`System`	`system/*` — Talos services such as `apid`, `trustd`, `udevd`, extension services	no

The kubelet does not create a dedicated cgroup for the Guaranteed QoS class: Guaranteed pods live directly under kubepods, next to the besteffort and burstable cgroups. The init cgroup (machined itself) is accounted as System when metrics are collected, but it is never considered as a kill candidate. Candidates are always the immediate children of the cgroups listed above. For Kubernetes this means a whole pod with all of its containers, and for Talos a single service.

Trigger

The triggerExpression is a boolean condition used by the OOM controller to decide whether it should act. If the expression evaluates to true, the OOM controller will activate and attempt to kill processes in order to free up memory. Pressure Stall Information is the key parameter provided to the expression, it should be the primary indication for determining whether or not OOM killing is required. To find more information on the meaning of the PSI parameters, please read the linked page. The avg10, avg60 and avg300 values are percentages of time stalled over the last 10, 60 and 300 seconds, while the total values are the absolute stall time accumulated since boot, in microseconds. These variables describe the root cgroup, that is the machine as a whole:

memory_some_avg10 - double - some memory pressure value, averaged over 10 seconds
memory_some_avg60 - double - some memory pressure value, averaged over 60 seconds
memory_some_avg300 - double - some memory pressure value, averaged over 300 seconds
memory_some_total - double - some memory pressure value, absolute cumulative value
memory_full_avg10 - double - full memory pressure value, averaged over 10 seconds
memory_full_avg60 - double - full memory pressure value, averaged over 60 seconds
memory_full_avg300 - double - full memory pressure value, averaged over 300 seconds
memory_full_total - double - full memory pressure value, absolute cumulative value

The same metrics are also available per QoS class, as maps keyed by the class constants (Besteffort, Burstable, Guaranteed, Podruntime, System):

qos_memory_some_avg10, qos_memory_some_avg60, qos_memory_some_avg300, qos_memory_some_total
qos_memory_full_avg10, qos_memory_full_avg60, qos_memory_full_avg300, qos_memory_full_total

Memory usage of each class is available in the same map form, in bytes:

qos_memory_current - current memory usage
qos_memory_peak - peak registered memory usage
qos_memory_max - configured memory limit

Every class is always present in these maps: a class with no cgroups (for example, when no Guaranteed pods are scheduled) reports zeroes rather than being absent. Values for a class are the sum over all cgroups of that class. d_ prefixed variants of all of the aforementioned variables (such as d_memory_full_avg10 or d_qos_memory_full_total) are also available – these represent the current derivative of that value, in absolute units per second. The derivative is normalized by the sampling interval, so changing sampleInterval does not change the scale of these values. Additionally, time_since_trigger variable is provided, representing the time past since the previous OOM trigger as the CEL duration type. You may use this variable to rate limit OOM triggers to ensure the monitored parameters have time to reflect the updated system state before new trigger decision. The multiply_qos_vectors(values, weights) function is provided to combine a per-class map into a single number: it returns the sum of values[class] * weights[class] over the classes listed in weights. Classes which are not listed in the weights map contribute nothing, which makes it a convenient way to both select and weigh the classes of interest.

Default condition in detail

The default value for triggerExpression is:

multiply_qos_vectors(d_qos_memory_full_total, {System: 8.0, Podruntime: 4.0}) > 3000.0 &&
multiply_qos_vectors(qos_memory_full_avg10, {System: 1.0, Podruntime: 1.0}) > 5.0 &&
time_since_trigger > duration("5s")

The default condition deliberately ignores memory pressure experienced by the workloads themselves and only looks at the pressure experienced by the System and Podruntime classes — Talos services, the container runtime, kubelet and etcd. The goal is not to keep any particular pod alive, but to keep the node itself responsive and manageable: the OOM handler steps in exactly when workload memory usage starts to starve the components which have to keep running. This expression checks if all these are true to trigger the OOM killer:

The System and Podruntime cgroups are accumulating memory stall time right now, weighted 8:4 in favour of the Talos system services
- This term is built on the derivative of the cumulative stall time, measured in microseconds of stall per second, so it only holds while pressure is actively being accumulated
- Without it, the handler would keep firing on the decaying 10 second average after the pressure has already subsided
The full memory pressure of the System and Podruntime cgroups combined, averaged over 10 seconds, is over 5%
- This confirms the pressure is sustained and not a short spike
The last OOM kill happened no less than 5 seconds ago
- Prevent the OOM killer from being triggered repeatedly without waiting for it to have an effect on the metrics used

In Talos 1.13 and earlier the default expression had an additional clause which triggered on the memory pressure of the machine as a whole (memory_full_avg10 > 75.0). It was removed in Talos 1.14, because a pod running with a memory limit and doing heavy I/O can drive up the overall memory pressure of the machine (the limit leaves little room for the page cache) while plenty of RAM is free, which resulted in false triggers.

Selecting a victim

Once the trigger fires, the controller enumerates the candidate cgroups and computes an OOM score for each of them using the expression configured by the cgroupRankingExpression property. These variables are supplied to the expression and can be used for computing OOM score:

memory_max - optional<uint> - if reported for the cgroup: max allowed memory usage, in bytes
memory_current - optional<uint> - if reported for the cgroup: current memory usage, in bytes
memory_peak - optional<uint> - if reported for the cgroup: peak registered memory usage, in bytes
path - string - absolute path to the cgroup being evaluated
class - int - one of the cgroup classes, should be matched against the class constants

The class constants (Besteffort, Burstable, Guaranteed, Podruntime and System) can be used to index CEL maps or in ternary operators used to apply different expressions for different cgroup classes. A cgroup is a candidate for killing only if its score is strictly greater than zero. A score of zero does not mean “killed last” — it means the cgroup is never killed at all. If no cgroup scores above zero, the trigger is logged and nothing is killed.

Strict QoS class ordering

Among the eligible cgroups the victim is picked according to the strictCgroupClassOrdering setting:

when it is enabled, which is the default, the controller first picks the least important class which has any eligible cgroup in it, and then the highest-scoring cgroup within that class;
when it is disabled, the controller picks the highest-scoring cgroup regardless of its class.

With strict ordering enabled, the score only ever breaks ties within a single class, so a large Burstable pod can never be picked over a small BestEffort pod. This is usually what is wanted: without it, a big Burstable pod such as kube-apiserver could outscore a small BestEffort pod and be killed first, as the default ranking expression only weighs BestEffort twice over Burstable.

Default formula in detail

memory_max.hasValue() ? 0.0 :
{Besteffort: 1.0, Burstable: 0.5, Guaranteed: 0.0, Podruntime: 0.0, System: 0.0}[class] *
double(memory_current.orValue(0u))

If the cgroup has a memory limit configured, return 0 — those are processes with well-defined resource demands, and the kernel already enforces their limit, so the Talos OOM handler leaves them alone
Otherwise, score by current memory usage, weighted by the cgroup class
- A map is used here to look up a coefficient depending on the cgroup class
- orValue is a method of the optional type allowing to unwrap the option, choosing a default value in case the value is not available
Guaranteed, Podruntime and System cgroups get a coefficient of 0, so they are never eligible

Which pods can be killed

Combining the default ranking expression with the default strict class ordering, this is how a pod is treated:

Pod QoS class	Pod cgroup `memory.max`	Score	Outcome
BestEffort	not set	`memory_current`	eligible, killed first
Burstable, at least one container without a memory limit	not set	`0.5 * memory_current`	eligible, only if no BestEffort pod is eligible
Burstable, every container has a memory limit	set	0.0	never killed
Guaranteed	set	0.0	never killed
Talos services, container runtime, `kubelet`, `etcd`	irrelevant	0.0	never killed

The kubelet sets a memory limit on the pod cgroup only when every container in the pod declares resources.limits.memory — including init containers. A single container without a memory limit is enough to leave the pod cgroup unlimited, and therefore to make the whole pod a candidate for the Talos OOM handler. This is easy to miss with an init container which has already terminated: the pod cgroup keeps no memory limit for the whole lifetime of the pod.

Protecting a workload

If a workload must not be killed by the Talos OOM handler, there are two options. The first one is to set resources.limits.memory on every container of the pod, including init containers. The pod cgroup then gets a memory limit, the default ranking expression scores it zero, and it is never selected by the Talos OOM handler. The trade-off is that the pod is now bounded by that limit and the kernel OOM killer enforces it, so an overrunning container is killed by the kernel instead. The second one is to customize the cgroupRankingExpression so that the workload scores zero, for example by giving its QoS class a coefficient of zero. Note that the cgroup path available to the expression is built from the pod UID (such as /sys/fs/cgroup/kubepods/burstable/podf3a1b0c2-...), not from the pod name or namespace, so matching on path is not a practical way to exempt a specific application. Whichever option is used, keep in mind that if every cgroup is exempted, the userspace OOM handler has nothing to kill and the machine is back to relying on the kernel OOM killer alone.

Observing OOM handler activity

The OOM controller logs each trigger to the controller runtime log:

talosctl -n <IP> logs controller-runtime

Relevant messages come from the runtime.OOMController controller:

[talos] OOM controller triggered {"component": "controller-runtime", "controller": "runtime.OOMController"}
[talos] Sending SIGKILL to cgroup {"component": "controller-runtime", "controller": "runtime.OOMController", "cgroup": "/sys/fs/cgroup/kubepods/besteffort/pod..."}

A trigger which found nothing to kill is logged as no eligible cgroup to kill. The last 50 actions are also kept as resources, which record the score of the victim, the command lines of the killed processes, and a JSON dump of the metrics which caused the trigger:

talosctl -n <IP> get oomactions -o yaml

OOMActions is a sensitive resource (it contains process command lines), so reading it requires the os:admin role, which the generated talosconfig has by default. The records are kept in memory only and are lost on reboot.

Tuning examples

Make the handler act sooner, by lowering the sustained pressure threshold and sampling more often:

apiVersion: v1alpha1
kind: OOMConfig
triggerExpression: |-
  multiply_qos_vectors(d_qos_memory_full_total, {System: 8.0, Podruntime: 4.0}) > 3000.0 &&
  multiply_qos_vectors(qos_memory_full_avg10, {System: 1.0, Podruntime: 1.0}) > 2.0 &&
  time_since_trigger > duration("5s")
sampleInterval: 250ms

Rank cgroups by their peak memory usage instead of the current one, which favours killing the workload which caused the spike rather than the one which is currently the largest:

apiVersion: v1alpha1
kind: OOMConfig
cgroupRankingExpression: |-
  memory_max.hasValue() ? 0.0 :
    {Besteffort: 1.0, Burstable: 0.5, Guaranteed: 0.0, Podruntime: 0.0, System: 0.0}[class] *
      double(memory_peak.orValue(0u))

Go back to pure score-based selection, ignoring the QoS class ordering:

apiVersion: v1alpha1
kind: OOMConfig
strictCgroupClassOrdering: false

Disable the userspace OOM handler completely, leaving only the kernel OOM killer:

apiVersion: v1alpha1
kind: OOMConfig
triggerExpression: "false"

Overview

Getting Started

Platform specific installation

Deploying and managing workloads

Networking

Security

Build and extend Talos

Configure your Talos cluster

Advanced guides

Reference

Troubleshooting and support

Learn more

How the OOM handler works

Cgroup classes

Trigger

Default condition in detail

Selecting a victim

Strict QoS class ordering

Default formula in detail

Which pods can be killed

Protecting a workload

Observing OOM handler activity

Tuning examples

​How the OOM handler works

​Cgroup classes

​Trigger

​Default condition in detail

​Selecting a victim

​Strict QoS class ordering

​Default formula in detail

​Which pods can be killed

​Protecting a workload

​Observing OOM handler activity

​Tuning examples

How the OOM handler works

Cgroup classes

Trigger

Default condition in detail

Selecting a victim

Strict QoS class ordering

Default formula in detail

Which pods can be killed

Protecting a workload

Observing OOM handler activity

Tuning examples