-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Add CPU and memory information to web console #18613
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
sql/src/test/java/org/apache/druid/sql/calcite/schema/SystemSchemaTest.java
Fixed
Show fixed
Hide fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds CPU and memory information to the web console by exposing these metrics through the sys.servers table. The new fields help understand resource availability for each server, which is particularly useful for Kubernetes deployments where each server runs in an isolated pod.
Key changes:
- Added
available_processorsandtotal_memoryfields to the sys.servers table and web console - Implemented container-aware resource detection using Java 17's OperatingSystemMXBean
- Updated web console to display CPU processors and total memory with proper formatting
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| processing/src/main/java/org/apache/druid/utils/JvmUtils.java | Added getTotalMemory() method using OperatingSystemMXBean for container-aware memory detection |
| server/src/main/java/org/apache/druid/discovery/DiscoveryDruidNode.java | Added availableProcessors and totalMemory fields with proper serialization and backwards compatibility |
| sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java | Extended sys.servers table schema to include new CPU and memory columns |
| web-console/src/views/services-view/services-view.tsx | Added CPU processors and total memory columns to services view with aggregation support |
| web-console/src/utils/general.tsx | Enhanced formatBytes function to support binary byte formatting |
| sql/src/test/java/org/apache/druid/sql/calcite/schema/SystemSchemaTest.java | Updated tests to include new fields in expected results |
| server/src/test/java/org/apache/druid/discovery/DiscoveryDruidNodeTest.java | Updated serialization tests to include new fields |
| docs/querying/sql-metadata-tables.md | Added documentation for new available_processors and total_memory columns |
| website/.spelling | Added new field names to spelling whitelist |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
server/src/main/java/org/apache/druid/discovery/DiscoveryDruidNode.java
Outdated
Show resolved
Hide resolved
FrankChen021
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
generally looks good, left some minor comments
FrankChen021
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
| public class DiscoveryDruidNode | ||
| { | ||
| private static final Logger LOG = new Logger(DiscoveryDruidNode.class); | ||
| private static final Integer UNKNOWN_VALUE = -1; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| private static final Integer UNKNOWN_VALUE = -1; | |
| private static final int UNKNOWN_VALUE = -1; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated
This PR adds 2 new fields to the sys.servers table which is displayed on the web console. These will be stored on
DiscoveryDruidNodeBoth fields represent the available resource on the machine. If the service is hosted in a container, it will represent the resource available to the container instead. (Container Aware)
The reason for adding this feature is so that we are able to know the cost of running each of the servers. This is useful when deploying on Kubernetes as we have each server running on an isolated pod.
Interestingly, it seems that from Java 17, the
OPERATING_SYSTEM_MX_BEANis container aware which helps to simplify the implementation. It might be possible to simplify some legacy code which can be found in thecgroupsfolder i.e.Cpu.javaandMemory.java. I have not looked into detail but I feel that it is something worth looking into.https://developers.redhat.com/articles/2022/04/19/java-17-whats-new-openjdks-container-awareness#recent_changes_in_openjdk_s_container_awareness_code