feat: extend gym list and gym search commands

[marta-sd]

Extends CLI commands for discovery:

gym list models
gym list resources-servers

gym search environments
gym search agents
gym search models
gym search resources-servers

Part of #1582

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[marta-sd]

/ok to test a52eca1

[gwarmstrong]

Could we make sure gym list models and gym eval run use the same collision rule? list_models()/discover_models() let an earlier root replace a whole model group, while --model-type searches for an exact config in each root. This means a model type can disappear from gym list models even though it is still selectable.

E.g., if we have:

/tmp/my-plugin/
└── responses_api_models/
    └── vllm_model/
        └── configs/
            └── vllm_model.yaml

And Gym includes:

responses_api_models/
└── vllm_model/
    └── configs/
        ├── vllm_model.yaml
        └── vllm_model_for_training.yaml

When running with:

gym list models --search-dir /tmp/my-plugin

vllm_model/vllm_model_for_training is not listed because the plugin's vllm_model entry replaces the whole built-in entry. But this still resolves the built-in config:

gym eval run \
  --search-dir /tmp/my-plugin \
  --model-type vllm_model/vllm_model_for_training \
  ...

Could discovery and selection either both apply precedence to the whole model group, or both apply it to each model type separately?

[marta-sd]

fixed in e531be2 I also added test for the example you provided

[github-actions]

🌿 Preview your docs: https://nvidia-preview-martas-cli-1582-2-list-search.docs.buildwithfern.com/nemo/gym

[prokotg]

gym search <module> with empty QUERY always match benchmark. I would expect it to be equal to gym list <module> . I know gym search <module> "" is. Just highlighting

gym search benchmarks  
                                                                                       Benchmarks matching 'benchmarks' (2)                                                                                       
┏━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Name           ┃ Domain ┃ Description                                                                                                                                     ┃ Agent name           ┃ Num repeats ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ longmt_pg19    │ other  │ Document-level MT verifier for pg19 books using the SEGALE pipeline (ersatz segment → LASER2 embed → vecalign align → COMETKiwi score)          │ longmt_pg19_agent    │ 5           │
│ longmt_wmt24pp │ other  │ Document-level MT verifier for wmt24pp short docs using the SEGALE pipeline (ersatz segment → LASER2 embed → vecalign align → COMETKiwi score). │ longmt_wmt24pp_agent │ 5           │
└────────────────┴────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────┴─────────────┘

It is also unclear what QUERY is really matched against. From the gym search --help I read

QUERY                 Substring to match against component names.

But matching is done wider. For example:

gym search benchmarks agent 
                                                                                   Benchmarks matching 'agent' (7)                                                                                   
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Name                                               ┃ Domain ┃ Description                                              ┃ Agent name                                                 ┃ Num repeats ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ browsecomp                                         │ agent  │                                                          │ browsecomp_benchmark_agent                                 │ 1           │
│ finance_sec_search/config_no_web_search            │ agent  │ SEC EDGAR filing search for financial analysis questions │ finance_sec_search_benchmark_agent                         │ 1           │
│ finance_sec_search/config_web_search               │ agent  │ SEC EDGAR filing search for financial analysis questions │ finance_sec_search_web_search_benchmark_agent              │ 1           │
│ tau2/configs/banking_alltools                      │ agent  │ Tau2 benchmark integration                               │ tau2_banking_knowledge_alltools_agent                      │ 8           │
│ tau2/configs/banking_bm25_grep_artificial_analysis │ agent  │ Tau2 benchmark integration                               │ tau2_banking_knowledge_bm25_grep_artificial_analysis_agent │ 5           │
│ tau2/configs/banking_terminal_use                  │ agent  │ Tau2 benchmark integration                               │ tau2_banking_knowledge_terminal_use_agent                  │ 8           │
│ tau2/configs/tau2                                  │ agent  │ Tau2 benchmark integration                               │ tau2_benchmark_agent                                       │ 8           │
└────────────────────────────────────────────────────┴────────┴──────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┴─────────────┘

Or

gym search agents reflection
                                                 Agents matching 'reflection'                                                  
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ agent           ┃ composition    ┃ variants                                                                   ┃ description ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ langgraph_agent │ composable (A) │ orchestrator_agent, parallel_thinking_agent, reflection_agent, rewoo_agent │             │
└─────────────────┴────────────────┴────────────────────────────────────────────────────────────────────────────┴─────────────┘

But it is not matched against description

gym search benchmarks EDGAR 
No benchmarks match 'EDGAR'.

[prokotg]

Side note: we load a lot of modules to fire up a query. I takes about 10 seconds on a cold-start for me to get results.

[marta-sd]

gym search with empty QUERY always match benchmark.

@prokotg this is expected behavior. module is optional and defaults to benchmarks, so your command is not interpreted as one with empty query, but as one with skipped module.

[marta-sd]

@prokotg I changed the help message and added description to searched fields (commit 54c405b. Additionally I made some imports in eval.py lazy to speed up listing (commit 3aa0f84) but the effect is minimal. I don't have a BW to investigating, I'll create a ticket to track this separately.

[marta-sd]

Ticket for addressing the slowness: #2100 Includes likely cause and proposed solution

[prokotg]

@marta-sd I understand the mechanism but I think it's inconsistent. There is no default for gym list for example

[marta-sd]

@marta-sd I understand the mechanism but I think it's inconsistent. There is no default for gym list for example

Agreed this is inconsistent with gym list, however:

• making module a required arg would be a breaking change
• we shouldn't support empty query regardless. If the module was required and user didn't provide a query we should throw an error.

Given these two, I think it's better to stick to module being optional

[marta-sd]

/ok to test ee0af30