What happened?
The function _has_memory_headroom is actually checking if the system has less than 95% of the total memory available free. If so it will return true and otherwise it will return false:
|
mem = psutil.virtual_memory() |
|
if mem.percent > 95: |
|
log.warning( |
|
"Memory limit reached (Used: %s%%). Pausing queue processing.", |
|
mem.percent, |
|
) |
|
return False |
The problem with this is, that salt-minion jobs get queued up on nodes that have a huge amount of memory still free. Let's say you have 2TB memory nodes. 5% memory headroom on such a system would be 102.4 GB. I guess that should be more than enough for salt ;-) Therefore i suggest we check that we have enough free memory for a normal average salt-minion process to spawn + some extra headroom instead of the current way. That could be done with https://psutil.readthedocs.io/stable/#psutil.Process.memory_info for example
Type of salt install
Official deb
Major version
3007.x
What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)
debian-12, ubuntu-24.04, debian-11, ubuntu-22.04
salt --versions-report output
Salt Version:
Salt: 3007.13
Python Version:
Python: 3.10.19 (main, Feb 5 2026, 07:05:38) [GCC 11.2.0]
Dependency Versions:
cffi: 2.0.0
cherrypy: unknown
cryptography: 42.0.5
dateutil: 2.8.2
docker-py: Not Installed
gitdb: 4.0.10
gitpython: 3.1.31
Jinja2: 3.1.6
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 24.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.22.3
smmap: 5.0.0
timelib: 0.3.0
Tornado: 6.5.4
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: ubuntu 22.04.5 jammy
locale: utf-8
machine: x86_64
release: 5.15.0-173-generic
system: Linux
version: Ubuntu 22.04.5 jammy
What happened?
The function
_has_memory_headroomis actually checking if the system has less than 95% of the total memory available free. If so it will returntrueand otherwise it will returnfalse:salt/salt/minion.py
Lines 2110 to 2116 in 24a4cfa
The problem with this is, that salt-minion jobs get queued up on nodes that have a huge amount of memory still free. Let's say you have 2TB memory nodes. 5% memory headroom on such a system would be 102.4 GB. I guess that should be more than enough for salt ;-) Therefore i suggest we check that we have enough free memory for a normal average salt-minion process to spawn + some extra headroom instead of the current way. That could be done with https://psutil.readthedocs.io/stable/#psutil.Process.memory_info for example
Type of salt install
Official deb
Major version
3007.x
What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)
debian-12, ubuntu-24.04, debian-11, ubuntu-22.04
salt --versions-report output
Salt Version: Salt: 3007.13 Python Version: Python: 3.10.19 (main, Feb 5 2026, 07:05:38) [GCC 11.2.0] Dependency Versions: cffi: 2.0.0 cherrypy: unknown cryptography: 42.0.5 dateutil: 2.8.2 docker-py: Not Installed gitdb: 4.0.10 gitpython: 3.1.31 Jinja2: 3.1.6 libgit2: Not Installed looseversion: 1.3.0 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.7 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 24.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.5.2 PyYAML: 6.0.1 PyZMQ: 25.1.2 relenv: 0.22.3 smmap: 5.0.0 timelib: 0.3.0 Tornado: 6.5.4 ZMQ: 4.3.4 Salt Package Information: Package Type: onedir System Versions: dist: ubuntu 22.04.5 jammy locale: utf-8 machine: x86_64 release: 5.15.0-173-generic system: Linux version: Ubuntu 22.04.5 jammy