Config loading intermittently fails when ranks share a Hugging Face cache

[janbernloehr]

Describe the bug

Distributed Automodel jobs can intermittently fail during configuration loading when multiple ranks resolve the same model repository through a shared Hugging Face cache:

ValueError: Unrecognized model in <repo-id>. Should have a model_type key in its config.json

The model's config.json contains model_type, and other ranks can load the same model successfully. The failure prevents training from starting.

Steps/Code to reproduce bug

1. Pre-populate a shared Hugging Face cache with a model repository.
2. Point all ranks at that cache using the same HF_HOME and configure Automodel to load the model by repository ID.
3. Start a distributed job so ranks load the configuration concurrently.

The failure is intermittent and affects a subset of ranks. These are the observed triggering conditions; a deterministic minimal reproducer is not yet available.

Expected behavior

All ranks load the cached model configuration successfully and training starts.

Additional context

The affected path is ConfigNode.instantiate() in nemo_automodel/components/config/loader.py, which calls from_pretrained with online resolution enabled by default. Concurrent cache updates can leave another rank's lookup temporarily unable to resolve config.json; Transformers can then report the empty configuration as an unrecognized model.