Skip to content

Make the node model workload-aware so a job can run once per deployment - #218

Open
karldebisschop wants to merge 3 commits into
rundeck-plugins:mainfrom
karldebisschop:workload-indexing-and-node-naming
Open

Make the node model workload-aware so a job can run once per deployment#218
karldebisschop wants to merge 3 commits into
rundeck-plugins:mainfrom
karldebisschop:workload-indexing-and-node-naming

Conversation

@karldebisschop

Copy link
Copy Markdown
Contributor

The problem

A Deployment with three replicas produces three Rundeck nodes. For most jobs
that is exactly right — you want the command on every pod. But a large class of
maintenance work must run once per deployment, not once per pod, because the
replicas share state: a database migration, a cache rebuild, a data backfill, a
scheduled cleanup, anything that writes to the backend the replicas have in
common.

Running such a job on every pod is wasteful in the best case. In the worst case
it is a correctness bug: three replicas executing the same migration
concurrently against one database is a race that testing on a single-replica
environment will never reveal, and that surfaces in production as partial
writes, duplicate rows, or deadlocks.

Why this cannot be expressed today

The resource model names each node <pod name>-<container name>, and exposes
the pod's labels. Neither gives you a way to select exactly one pod of a
deployment, durably:

  • Pod names are ephemeral. billing-7f4b8bbfc6-2q2hd contains a
    pod-template-hash and a random suffix, both of which change on every rollout.
    A node filter naming a specific pod stops matching the next time you deploy.
  • Labels identify the group, not a member. Filtering labels:app = billing
    selects all three replicas — which is the situation we are trying to avoid.

There is currently no stable identifier for "the deployment this pod belongs
to", and no deterministic way to pick one of its pods.

What this adds

default:workload — the owning controller with the Deployment
pod-template-hash stripped, so billing-7f4b8bbfc6-2q2hd reports billing.
Derived from the pod's controlling ownerReference rather than by parsing the
pod name, falling back to the first reference and then the pod name, so
StatefulSets, DaemonSets, Jobs and bare pods all resolve sensibly. Unlike the
pod name, this survives rollouts.

index — an ordinal within namespace/workload/container, assigned only to
pods that are actually emitted, so a filtered-out Terminated pod does not
consume a number and index == 1 always matches. Because the counter is scoped
per workload, one filter selects one pod from each deployment:

3 deployments x 3 replicas = 9 nodes
nodes matching  index == 1 : 3
     prod / billing   (billing-7f4b8bbfc6-aaaaa)
     prod / search    (search-7f4b8bbfc6-aaaaa)
  staging / billing   (billing-7f4b8bbfc6-aaaaa)

One Node Per Workload (RD_CONFIG_ONE_PER_WORKLOAD, default off) — does the
same thing at the model level for operators who would rather not rely on every
job remembering the filter. The node count then stays stable as replicas scale.

Node Name Format — node names become a ${attribute} template, so they can
be built from the stable identity rather than the ephemeral pod name, for
example ${default:namespace}-${default:workload}. The full pod name remains
available as default:name for traceability.

Custom Mapping sources now resolve against any node attribute rather than the
default:* set alone. That makes a label with an awkward key usable in a
template — service.selector=labels:app.kubernetes.io/name, then ${service}
and fixes mappings from labels: silently producing nothing.

Compatibility

Both new behaviors are opt-in and the defaults reproduce today's output:

  • The default template is ${default:name}-${default:container_name}, which is
    the historical <pod name>-<container name> exactly.
  • An explicit nodename from Custom Mapping or Default attributes still wins
    over the template, so the documented nodename.selector=default:name keeps
    working. Renaming a node makes Rundeck treat it as a different node, so this
    ordering matters.
  • One Node Per Workload defaults to false; the node set is unchanged unless it
    is enabled.
  • default:workload and index are additive attributes.

An unset attribute renders as an empty string rather than the literal None,
so a pending pod with no podIP cannot produce my-app-None — or give several
pending pods the same name.

Testing

The resource-model suite goes from 33 to 49 tests, covering workload derivation
and controller-owner preference, per-workload numbering including the
filtered-pod case, the one-per-workload toggle, template rendering with labels
and unset attributes, explicit-nodename precedence, and mapping from a label.

- workload_name: derive the owning workload from the pod's controlling
  ownerReference with the Deployment pod-template-hash stripped, falling back
  to the first reference then the pod name (StatefulSets, DaemonSets, Jobs,
  bare pods); expose it as the default:workload node attribute
- number pods per namespace/workload/container, so each container name gets its
  own sequence within a workload and a job can target a single replica by
  filtering on index == 1. Numbering runs after the running/terminated filter so
  a pod that is not emitted does not consume an index -- otherwise a workload
  whose first pod is Terminated emits nodes numbered 2 and 3 and that filter
  matches nothing
- RD_CONFIG_ONE_PER_WORKLOAD emits only the first kept pod for each workload
  and container, doing once at the model level what an index filter does in
  every job, and keeping the node count stable as replicas scale; off by
  default
- render nodename from a ${attribute} template that can reference any node
  attribute -- default:*, labels:*, index, Custom Mapping aliases and operator
  defaults. main() renders it once the pod is known to be kept and has its final
  index, so ${index} is meaningful. The default template preserves the
  historical "<pod name>-<container name>"
- an explicit nodename from Custom Mapping ("nodename.selector=default:name",
  the first example in the README) or from Default attributes still wins. The
  template is the default naming strategy, not an override of the operator's;
  overriding it would silently rename every node of an existing job, and
  Rundeck treats a renamed node as a different node
- an attribute that is unset renders as an empty string rather than the literal
  "None": an unscheduled pod has no podIP, and rendering that as "None" would
  put the word in the node name and give every pending pod the same one
- resolve Custom Mapping sources against the assembled node attributes rather
  than the default:* set alone, and after labels are merged, so
  "service.selector=labels:app.kubernetes.io/name" now works and gives an
  awkward label key a short alias usable in the template; an unresolvable
  source is skipped instead of raising KeyError
- plugin.yaml: add Node Name Format and One Node Per Workload properties, note
  that index is counted per namespace, and document label aliases in Custom
  Mapping
- tests: workload derivation and controller-owner preference, per-workload
  container numbering including the filtered-pod case, the one-per-workload
  toggle, template rendering with labels, unset and missing attributes,
  explicit-nodename precedence, and mapping from a label
@karldebisschop
karldebisschop requested review from a team and a lite review from Copilot August 10, 2026 21:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes the Kubernetes pods resource model workload-aware so operators can target exactly one pod per Deployment (or other controller) for “run once per workload” maintenance jobs, while keeping the existing default behavior unchanged unless explicitly enabled.

Changes:

  • Add a stable default:workload attribute derived from the pod’s owning controller (stripping the Deployment pod-template-hash) and an index ordinal per workload/container for deterministic replica selection.
  • Add configurable node naming via a ${attribute} template (nodename_format) and a one_per_workload toggle to collapse replicas at the model level.
  • Expand and update the unit test suite to cover workload derivation, indexing behavior, one-per-workload mode, and template rendering.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
plugin.yaml Adds new plugin config options (nodename_format, one_per_workload) and updates mapping documentation.
contents/pods-resource-model.py Implements workload derivation, per-workload indexing, node name template rendering, and one-per-workload emission behavior.
contents/tests/test_pods_resource_model.py Adds tests covering new workload/index semantics, nodename templates, mapping-from-label behavior, and one-per-workload behavior.
Suppressed comments (1)

plugin.yaml:95

  • The implementation deduplicates on namespace/workload/container (one node per workload per container). The config description currently says “single node per workload”, which can be misleading for multi-container pods where more than one node will still be emitted.
        title: One Node Per Workload?
        description: 'Emit a single node per workload (Deployment/StatefulSet/etc.) instead of one per pod, so a job runs once per deployment without every job needing an index filter. The emitted node keeps the first matching pod name in default:name; default:workload carries the stable workload name.'
        default: "false"

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 177 to 183
for key, value in mapping_array.items():
if ".selector" in key:
attribute = key.replace(".selector", "")
custom_attribute = None
# take the values from default
if "default:" in value:
custom_attribute = default_settings[value]
custom_attribute = data.get(value)

if custom_attribute:
custom_attributes[attribute] = custom_attribute
Comment thread plugin.yaml
Comment on lines +42 to +44
title: 'Node Name Format'
description: 'Template for the node name. Reference any node attribute with ${...}: default:*, labels:*, index, or an alias defined in Custom Mapping. An attribute that is unset renders as an empty string. index counts the emitted pods within a workload in a single namespace, so include ${default:namespace} when scanning all namespaces or two namespaces running the same workload will produce the same node name. Ignored if a nodename is set explicitly through Custom Mapping or Default attributes. The full pod name remains available as default:name for traceability.'
default: '${default:name}-${default:container_name}'

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both descriptions updated

Review findings on the pull request.

A Custom Mapping resolved its source and then applied a truth test, so a label
with an empty value -- legal in Kubernetes -- was dropped even though the source
existed. Test for `is not None` instead; a missing source still resolves to
None and is still skipped.

index counts pods within a namespace/workload/container group, so a template
that omits ${default:container_name} makes the containers of a multi-container
pod render the same node name, exactly as omitting ${default:namespace} does
across namespaces. The Node Name Format description covered only the namespace
case; it now covers both, and One Node Per Workload says that it emits one node
per container rather than one per pod.
@karldebisschop
karldebisschop force-pushed the workload-indexing-and-node-naming branch from dbeba93 to e7e425d Compare August 10, 2026 23:05
A tag selector read its attribute by indexing and appended the value as-is, so
two ordinary configurations took the entire node source down: no JSON, no nodes,
a traceback instead.

No value for this pod. Either the attribute is absent -- a label only some pods
carry -- which raised KeyError, or it is present but unset, which raised
TypeError when the tag list was joined. default:status_message is None on every
healthy pod, so that selector failed on an entirely normal cluster. A cluster
where 99 of 100 pods carry a label produced nothing at all.

A value that is not a string. terminated is a bool, and joining it raised
TypeError even with every pod labelled.

Resolve the attribute with .get() and omit the tag when there is no value; str()
it when there is. A pod that lacks the attribute now carries one tag fewer.
@karldebisschop
karldebisschop force-pushed the workload-indexing-and-node-naming branch from fe4a51a to 7179190 Compare August 11, 2026 11:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants