feat: extend discoverability through CLI by marta-sd · Pull Request #2032 · NVIDIA-NeMo/Gym

marta-sd · 2026-07-15T06:48:14Z

Implementes #1582

copy-pr-bot · 2026-07-15T06:48:17Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

Originally we've used the dataset name. These two are usually identical, but for benchmarks with non-standard config location the output from `gym list command` was not usable as value for `--benchmark` flag in other commands. This commit fixes it. Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

…gwarmstrong

…gwarmstrong conversly to search_dir config field, env var allows to pass extra dirs deep into the code, including rollouts colletion, prompt templates, config paths and env.yaml loading. also it is propagated to any spawned process Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

…ogging too Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

@gwarmstrong

…1264 from @gwarmstrong) Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

gwarmstrong · 2026-07-16T20:13:28Z

+        yield
+        return
+    original = os.environ.get(NEMO_GYM_EXTRA_ROOTS_ENV_VAR_NAME)
+    value = os.pathsep.join(search_dirs)


Should --search-dir add to NEMO_GYM_EXTRA_ROOTS instead of replacing it?

For example:

export NEMO_GYM_EXTRA_ROOTS=/plugins/common gym eval run --search-dir /plugins/experiment ...

During this command, /plugins/common is no longer searched. I expected the order to be /plugins/experiment, /plugins/common, cwd, then the Gym install.

makes sense, I will modify this part

Implemented in 6087209

gwarmstrong · 2026-07-16T20:13:28Z

+    roots into the env (see nemo_gym.cli.main). With no extra roots this is just ``append(PARENT_DIR)``, as
+    before.
+    """
+    for root in [*_extra_roots(), PARENT_DIR]:


Could file lookup and Python imports share the same root-ordering logic?

component_search_roots() gives extra roots priority over PARENT_DIR, but _augment_sys_path() can leave PARENT_DIR before the extra roots because it was already added to sys.path. This means a plugin can win during file resolution but lose during Python import when Gym contains a module with the same import name.

Is there a way for both resolvers to use the same ordered roots, so their precedence cannot diverge over time?

Good catch.
I wonder if I should just ensure that PARENT_DIR is at the end, or in general that the extra roots are added before any other paths. I think the order should be: 1) extra roots, 2) original sys.path excluding PARENT_DIR, 3) PARENT_DIR. What do you think?

done in ee0c74c, let me know if the ordering should be different

gwarmstrong · 2026-07-16T20:13:28Z

+    merged: Dict[str, _T] = {}
+    for entries in per_root:
+        for name, entry in entries.items():
+            merged.setdefault(name, entry)


Could list/search and named selection use the same rule for duplicate names?

This code keeps the first match, but _asset_config_path() reports an error when the same name exists in two roots. A component can therefore appear in gym list but fail when selected by name.

Either rule seems reasonable, but using the same rule in both places would be clearer.

Good point. I think I'll modify _asset_config_path to pick up the first one and maybe print a warning to the user

done in e6c6a88

gwarmstrong · 2026-07-16T20:13:28Z

+    if not benchmarks_dir.is_dir():
+        return []
+    config_paths = [benchmarks_dir / p for p in glob("**/*.yaml", root_dir=benchmarks_dir, recursive=True)]
+    return sorted(p for p in config_paths if "type: benchmark" in p.read_text(errors="ignore"))


This only recognizes the exact text type: benchmark. Valid YAML such as type: "benchmark" or type : benchmark is skipped. I tested both cases.

Could we parse the YAML here, or make the check accept normal YAML formatting differences?

I'll try with yaml parsing and if it doesn't work for any reason - extend the pattern here

done in 97b5c9d I went with yaml and if the loading errors out we treat the config as potential benchmark. then all non-benchmarks fail to load and are filtered with a warning inside _load_benchmarks_from_config_paths

gwarmstrong · 2026-07-16T20:13:28Z

+# `gym search [<type>] <query>`: an optional component type plus the query. The query is surfaced to the
+# chosen listing command as the reserved `query` config key; the type only picks which command to run
+# (see `_search`). A lone positional is the query, defaulting to benchmarks — backward compatible.
+_SEARCHABLE_TYPES = {


Should datasets be included here too?

This one is tricky - datasets are not really an independent concept in Gym, they are tightly coupled with benchmarks, envs and resources servers. But we want to make them separate (#2036) and I think this will be the right moment to add gym list and gym search commands for them

gwarmstrong · 2026-07-16T20:21:07Z

-            _cwd_path = Path.cwd() / _input_path
-            _input_path = _cwd_path if _cwd_path.exists() else PARENT_DIR / _input_path
+        # Search NEMO_GYM_EXTRA_ROOTS, cwd, then the install root.
+        _input_path = _resolve_under_cwd_or_install(config.input_jsonl_fpath)


Could skills.path use this resolver too?

A few lines above, load_skill_directory(config.skills.path) calls _resolve_skills_path(), which only checks cwd and the Gym install. This means --search-dir /plugins/my-plugin can resolve the input path here, but it cannot resolve skills.path=skills from the same plugin.

Good catch, will fix

done in b32c6d0

marta-sd · 2026-07-17T08:14:17Z

This PR got too big so I replaced it with stack. Comments from @gwarmstrong will be addressed in #2061

ritaneves linked an issue Jul 15, 2026 that may be closed by this pull request

[Epic] Search & Discoverability for Environments and Components from the CLI #1582

Closed

7 tasks

ritaneves requested a review from prokotg July 15, 2026 13:53

marta-sd added 13 commits July 16, 2026 14:01

chore: unify metadata and parsing code when listing envs and benchmarks

cf3e16b

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

feat: add --search-dir to gym list and gym search

eb4f974

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

chore: extract repeated code

acb801a

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

feat: add gym list models

de0b55e

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

feat: add gym search <type> <query>

b73fd16

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

feat: add gym list and gym search for resources servers

b6a7149

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

chore: user per-flavor list of resources servers

73fb362

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

feat: add gym list <type> <name> for inspecting

afc9107

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

fix: custom 'did you mean?' for benchmarks

e59aaa6

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

fix: set logging to DEBUG on top of main so it applies to in-script l…

fd2d920

…ogging too Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

fix: use user-provided root dirs to resolve all paths (as did in PR #…

8c19560

…1264 from @gwarmstrong) Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>

marta-sd force-pushed the martas/cli-1582 branch from a4c46b4 to 8c19560 Compare July 16, 2026 12:11

gwarmstrong reviewed Jul 16, 2026

View reviewed changes

marta-sd self-assigned this Jul 17, 2026

marta-sd closed this Jul 17, 2026

marta-sd removed a link to an issue Jul 17, 2026

[Epic] Search & Discoverability for Environments and Components from the CLI #1582

Closed

7 tasks

Uh oh!

Conversation

marta-sd commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Jul 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marta-sd commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

marta-sd commented Jul 15, 2026 •

edited

Loading