Auto switch cgroup monitors from v1 to v2 #18705

cryptoe · 2025-10-30T08:32:52Z

Summary

This PR introduces auto-switching cgroup monitors that can dynamically detect and switch between cgroup v1 and v2 implementations, along with necessary code style fixes.

Changes

-Now CgroupCpuMonitor , CgroupCpuSetMonitor , CgroupDiskMonitor ,MemoryMonitor autoswitch to cgroups v2 and no longer fail in case cgroups v2 is detected.

All cgroupMetrics now emit a dimension 'cgroupversion" to understand which cgroup version was chosen
CgroupV2CpuMonitor also emits : cgroup/cpu/shares and cgroup/cpu/cores_quota
All cgroupsv2 monitors are NON experimental

Test Plan

All existing cgroup monitor tests updated to work with auto-switching logic
New comprehensive test suites added for cgroup v2 implementations
Tests verify correct detection and switching between cgroup versions
Tested on a cluster.

This enhancement allows Druid to seamlessly work across different containerization environments that use either cgroup v1 or v2, improving compatibility and reducing configuration overhead.

Release notes

Now CgroupCpuMonitor , CgroupCpuSetMonitor , CgroupDiskMonitor ,MemoryMonitor autoswitch to cgroups v2 and no longer fail in case cgroups v2 is detected.
"CpuAcctDeltaMonitor" fails gracefully incase cgroups v2 is detected
All cgroupMetrics now emit a dimension 'cgroupversion" to understand which cgroup version was chosen
"CgroupV2CpuMonitor" also emits : cgroup/cpu/shares and cgroup/cpu/cores_quota
All cgroupsv2 monitors are NON experimental

processing/src/test/java/org/apache/druid/java/util/metrics/cgroups/CpuSetV2Test.java

processing/src/test/java/org/apache/druid/java/util/metrics/cgroups/CpuV2Test.java

capistrant

What do we want to do about documentation in https://druid.apache.org/docs/latest/configuration/#metrics-monitors? Do we want to explicitly call out that the auto conversion will happen for the v1 monitors? Do we want to remove experimental from the v2 monitors docs?

capistrant · 2025-10-30T13:34:35Z

processing/src/test/java/org/apache/druid/java/util/metrics/cgroups/CpuSetV2Test.java

+
+package org.apache.druid.java.util.metrics.cgroups;
+
+import org.junit.Assert;


should new test file be written with junit5?

capistrant · 2025-10-30T13:42:06Z

processing/src/test/java/org/apache/druid/java/util/metrics/cgroups/CpuSetV2Test.java

+  }
+
+  @Test
+  public void testBackwardsCompatibilityWithV1Parsing() throws IOException


could you point out how this format is different than all the tests above? I'm missing what is unique about it compared to the rest of the v2 tests

processing/src/test/java/org/apache/druid/java/util/metrics/cgroups/CpuSetV2Test.java

capistrant · 2025-10-30T13:44:51Z

processing/src/test/java/org/apache/druid/java/util/metrics/cgroups/CpuV2Test.java

+
+package org.apache.druid.java.util.metrics.cgroups;
+
+import org.junit.Assert;


same as other new UT file, should we write it in junit5?

capistrant · 2025-10-30T13:51:21Z

processing/src/test/java/org/apache/druid/java/util/metrics/cgroups/CpuV2Test.java

+  }
+
+  @Test
+  public void testMicrosecondToJiffiesConversion() throws IOException


Suggested change

public void testMicrosecondToJiffiesConversion() throws IOException

public void testV2doesNotConvertMicrosecondToJiffies() throws IOException

nit: existing name feels misleading since there is no conversion, jiffies just aren't provided

capistrant · 2025-10-30T14:13:23Z

processing/src/main/java/org/apache/druid/java/util/metrics/CgroupUtil.java

+      }
+    }
+    catch (Exception e) {
+      LOG.debug(e, "Could not determine cgroups version, assuming v1");


should this actually be a warn so operators can investigate and get things properly configured?

This method is removed

capistrant · 2025-10-30T14:19:36Z

processing/src/main/java/org/apache/druid/java/util/metrics/cgroups/CpuV2.java

+  private static final String
+      CPU_MAX_FILE = "cpu.max";


Suggested change

private static final String

CPU_MAX_FILE = "cpu.max";

private static final String CPU_MAX_FILE = "cpu.max";

capistrant · 2025-10-30T20:32:26Z

processing/src/main/java/org/apache/druid/java/util/metrics/CgroupUtil.java

+   * @param cgroupDiscoverer the discoverer to check
+   * @return true if running on cgroups v2, false for v1 or unknown
+   */
+  public static boolean isRunningOnCgroupsV2(CgroupDiscoverer cgroupDiscoverer)


This method looks like it will use cgroupv1 if both exist. in processing/src/main/java/org/apache/druid/java/util/metrics/cgroups/ProcSelfCgroupDiscoverer.java detectCgroupVersion it looks like it prefers v2 even if both exist. Is this discrepancy intentional?

This method is no longer needed. Adjusted the code.

capistrant · 2025-10-30T20:53:59Z

processing/src/main/java/org/apache/druid/java/util/metrics/CgroupMemoryMonitor.java

+    }
+  }
+
+  public static boolean doMonitorInternal(


short javadoc might be helpful to note that this is a static utility method used by v1 and v2 monitor. I don't fully understand the "internal" part of the name either

capistrant · 2025-10-30T20:56:39Z

processing/src/main/java/org/apache/druid/java/util/metrics/CgroupV2CpuSetMonitor.java

Does this require an entry in https://druid.apache.org/docs/latest/configuration/#metrics-monitors? Will cover the docs topic more in the top level review comment

cryptoe

Thanks for the review @capistrant .
Responded to all your queries.
Regarding junit 4 vs junit5 frameworks, I hope that change may not block this PR.
Will need to adjust lot of the testing boiler plate code for that to work which I was hoping can be punted down the line :)

cryptoe · 2025-11-10T05:34:27Z

processing/src/main/java/org/apache/druid/java/util/metrics/CgroupUtil.java

+   * @param cgroupDiscoverer the discoverer to check
+   * @return true if running on cgroups v2, false for v1 or unknown
+   */
+  public static boolean isRunningOnCgroupsV2(CgroupDiscoverer cgroupDiscoverer)


This method is no longer needed. Adjusted the code.

cryptoe · 2025-11-10T05:34:53Z

processing/src/main/java/org/apache/druid/java/util/metrics/CgroupV2CpuSetMonitor.java

cryptoe · 2025-11-10T13:42:15Z

processing/src/main/java/org/apache/druid/java/util/metrics/CgroupUtil.java

+      }
+    }
+    catch (Exception e) {
+      LOG.debug(e, "Could not determine cgroups version, assuming v1");


This method is removed

cryptoe · 2025-11-10T13:42:53Z

processing/src/test/java/org/apache/druid/java/util/metrics/CgroupCpuSetMonitorTest.java

+
+    // doMonitor should return true but skip actual monitoring
+    Assert.assertTrue(monitor.doMonitor(emitter));
+    Assert.assertEquals(4, emitter.getNumEmittedEvents());


cryptoe · 2025-11-10T13:43:09Z

processing/src/test/java/org/apache/druid/java/util/metrics/CgroupCpuMonitorTest.java

+    // doMonitor should return true
+    Assert.assertTrue(monitor.doMonitor(emitter));
+
+    Assert.assertEquals(2, emitter.getEvents().size());


cryptoe · 2025-11-10T14:14:11Z

processing/src/test/java/org/apache/druid/java/util/metrics/cgroups/CpuV2Test.java

+  }
+
+  @Test
+  public void testZeroMicrosecondValues() throws IOException


Wanted to test if the values are initialized properly.
Added more asserts to check that.

capistrant

Left a suggestion that you should be able to batch accept to fix spell checker GHA

capistrant · 2025-11-10T14:58:43Z

docs/configuration/index.md

-|`org.apache.druid.java.util.metrics.CgroupV2CpuMonitor`| **EXPERIMENTAL** Reports CPU usage from `cpu.stat` file. Only applicable to `cgroupv2`.|Any|
-|`org.apache.druid.java.util.metrics.CgroupV2DiskMonitor`| **EXPERIMENTAL** Reports disk usage from `io.stat` file. Only applicable to `cgroupv2`.|Any|
-|`org.apache.druid.java.util.metrics.CgroupV2MemoryMonitor`| **EXPERIMENTAL** Reports memory usage from `memory.current` and `memory.max` files. Only applicable to `cgroupv2`.|Any|
+|`org.apache.druid.java.util.metrics.CgroupCpuMonitor`|Reports CPU shares and quotas as per the `cpu` cgroup. Automaticaly switches to `CgroupV2CpuMonitor` in case `cgroupv2` type is detected.|Any|


Suggested change

|`org.apache.druid.java.util.metrics.CgroupCpuMonitor`|Reports CPU shares and quotas as per the `cpu` cgroup. Automaticaly switches to `CgroupV2CpuMonitor` in case `cgroupv2` type is detected.|Any|

|`org.apache.druid.java.util.metrics.CgroupCpuMonitor`|Reports CPU shares and quotas as per the `cpu` cgroup. Automatically switches to `CgroupV2CpuMonitor` in case `cgroupv2` type is detected.|Any|

capistrant · 2025-11-10T14:59:00Z

docs/configuration/index.md

-|`org.apache.druid.java.util.metrics.CgroupV2DiskMonitor`| **EXPERIMENTAL** Reports disk usage from `io.stat` file. Only applicable to `cgroupv2`.|Any|
-|`org.apache.druid.java.util.metrics.CgroupV2MemoryMonitor`| **EXPERIMENTAL** Reports memory usage from `memory.current` and `memory.max` files. Only applicable to `cgroupv2`.|Any|
+|`org.apache.druid.java.util.metrics.CgroupCpuMonitor`|Reports CPU shares and quotas as per the `cpu` cgroup. Automaticaly switches to `CgroupV2CpuMonitor` in case `cgroupv2` type is detected.|Any|
+|`org.apache.druid.java.util.metrics.CgroupCpuSetMonitor`|Reports CPU core/HT and memory node allocations as per the `cpuset` cgroup. Automaticaly switches to `CgroupV2CpuSetMonitor` in case `cgroupv2` type is detected.|Any|


Suggested change

|`org.apache.druid.java.util.metrics.CgroupCpuSetMonitor`|Reports CPU core/HT and memory node allocations as per the `cpuset` cgroup. Automaticaly switches to `CgroupV2CpuSetMonitor` in case `cgroupv2` type is detected.|Any|

|`org.apache.druid.java.util.metrics.CgroupCpuSetMonitor`|Reports CPU core/HT and memory node allocations as per the `cpuset` cgroup. Automatically switches to `CgroupV2CpuSetMonitor` in case `cgroupv2` type is detected.|Any|

capistrant · 2025-11-10T14:59:13Z

docs/configuration/index.md

-|`org.apache.druid.java.util.metrics.CgroupV2MemoryMonitor`| **EXPERIMENTAL** Reports memory usage from `memory.current` and `memory.max` files. Only applicable to `cgroupv2`.|Any|
+|`org.apache.druid.java.util.metrics.CgroupCpuMonitor`|Reports CPU shares and quotas as per the `cpu` cgroup. Automaticaly switches to `CgroupV2CpuMonitor` in case `cgroupv2` type is detected.|Any|
+|`org.apache.druid.java.util.metrics.CgroupCpuSetMonitor`|Reports CPU core/HT and memory node allocations as per the `cpuset` cgroup. Automaticaly switches to `CgroupV2CpuSetMonitor` in case `cgroupv2` type is detected.|Any|
+|`org.apache.druid.java.util.metrics.CgroupDiskMonitor`|Reports disk statistic as per the blkio cgroup. Automaticaly switches to `CgroupV2DiskMonitor` in case `cgroupv2` type is detected.|Any|


Suggested change

|`org.apache.druid.java.util.metrics.CgroupDiskMonitor`|Reports disk statistic as per the blkio cgroup. Automaticaly switches to `CgroupV2DiskMonitor` in case `cgroupv2` type is detected.|Any|

|`org.apache.druid.java.util.metrics.CgroupDiskMonitor`|Reports disk statistic as per the blkio cgroup. Automatically switches to `CgroupV2DiskMonitor` in case `cgroupv2` type is detected.|Any|

capistrant · 2025-11-10T14:59:25Z

docs/configuration/index.md

+|`org.apache.druid.java.util.metrics.CgroupCpuMonitor`|Reports CPU shares and quotas as per the `cpu` cgroup. Automaticaly switches to `CgroupV2CpuMonitor` in case `cgroupv2` type is detected.|Any|
+|`org.apache.druid.java.util.metrics.CgroupCpuSetMonitor`|Reports CPU core/HT and memory node allocations as per the `cpuset` cgroup. Automaticaly switches to `CgroupV2CpuSetMonitor` in case `cgroupv2` type is detected.|Any|
+|`org.apache.druid.java.util.metrics.CgroupDiskMonitor`|Reports disk statistic as per the blkio cgroup. Automaticaly switches to `CgroupV2DiskMonitor` in case `cgroupv2` type is detected.|Any|
+|`org.apache.druid.java.util.metrics.CgroupMemoryMonitor`|Reports memory statistic as per the memory cgroup. Automaticaly switches to `CgroupV2MemoryMonitor` in case `cgroupv2` type is detected.|Any|


Suggested change

|`org.apache.druid.java.util.metrics.CgroupMemoryMonitor`|Reports memory statistic as per the memory cgroup. Automaticaly switches to `CgroupV2MemoryMonitor` in case `cgroupv2` type is detected.|Any|

|`org.apache.druid.java.util.metrics.CgroupMemoryMonitor`|Reports memory statistic as per the memory cgroup. Automatically switches to `CgroupV2MemoryMonitor` in case `cgroupv2` type is detected.|Any|

cryptoe added 2 commits October 30, 2025 13:38

Adding auto switching monitors

eb7e364

Fixing checkstyle

136bd70

github-actions bot added the Area - Metrics/Event Emitting label Oct 30, 2025

github-advanced-security bot found potential problems Oct 30, 2025

View reviewed changes

processing/src/test/java/org/apache/druid/java/util/metrics/cgroups/CpuSetV2Test.java Show resolved Hide resolved

processing/src/test/java/org/apache/druid/java/util/metrics/cgroups/CpuV2Test.java Fixed Show fixed Hide fixed

Adjusting for static checks

9922cb7

capistrant reviewed Oct 30, 2025

View reviewed changes

Adjusting for review.

54f2ae9

github-actions bot added the Area - Documentation label Nov 10, 2025

cryptoe commented Nov 10, 2025

View reviewed changes

capistrant reviewed Nov 10, 2025

View reviewed changes

Fixing spell check

528365f

capistrant approved these changes Nov 10, 2025

View reviewed changes

cryptoe merged commit 9b6913a into apache:master Nov 11, 2025
57 checks passed


		package org.apache.druid.java.util.metrics.cgroups;

		import org.junit.Assert;

	public void testMicrosecondToJiffiesConversion() throws IOException
	public void testV2doesNotConvertMicrosecondToJiffies() throws IOException

	private static final String
	CPU_MAX_FILE = "cpu.max";
	private static final String CPU_MAX_FILE = "cpu.max";

	\|`org.apache.druid.java.util.metrics.CgroupCpuMonitor`\|Reports CPU shares and quotas as per the `cpu` cgroup. Automaticaly switches to `CgroupV2CpuMonitor` in case `cgroupv2` type is detected.\|Any\|
	\|`org.apache.druid.java.util.metrics.CgroupCpuMonitor`\|Reports CPU shares and quotas as per the `cpu` cgroup. Automatically switches to `CgroupV2CpuMonitor` in case `cgroupv2` type is detected.\|Any\|

Auto switch cgroup monitors from v1 to v2 #18705

Auto switch cgroup monitors from v1 to v2 #18705

Uh oh!

Conversation

cryptoe commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Test Plan

Release notes

Uh oh!

Uh oh!

Uh oh!

capistrant left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

capistrant Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cryptoe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

capistrant left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cryptoe commented Oct 30, 2025 •

edited

Loading

capistrant Oct 30, 2025 •

edited

Loading