feat: support JSON config as well as YAML

[patriciapampanelli]

Adds JSON config file support alongside YAML, enabling users to use either format for garak configuration files. Extension-less config lookups (--config fast) now work JSON-only, while YAML configs require explicit .yaml extension.

issue #913

Extension-less Lookup: JSON-Only

• Extension-less lookups (--config fast) only work for JSON files
• YAML files must specify explicit .yaml extension

Behavior

User Input	.json exists	.yaml exists	Behavior
--config fast	Yes	No	Uses fast.json
--config fast	No	Yes	Error: YAML needs explicit .yaml extension
--config fast	Yes	Yes	Warning + uses .json
--config fast.yaml	Any	Yes	Uses fast.yaml (explicit)
--config fast.json	Yes	Any	Uses fast.json (explicit)

Site Config

• Both garak.site.json and garak.site.yaml supported
• Errors if both exist

[jmartin-tech]

Thanks you, some thoughts for consideration.

[jmartin-tech]

Should these also support .yml? This seems to be a somewhat contentious question on the internet. :-)

[jmartin-tech]

this seems inverted based on the assumptions and resolutions in the description:

Suggested change

	if settings_filename.endswith(".json"):
	settings = json.load(settings_file)
	else:
	settings = yaml.safe_load(settings_file)
	if settings_filename.endswith(".yaml"):
	settings = yaml.safe_load(settings_file)
	else:
	settings = json.load(settings_file)

Consider expanding this to have a try block that can be used to retry load with the other format for cases where the filename does not contain a supported extension.

This would account for user provided cli arguments like:

• --config my_config
• --config my_config.file
• --config my_config.yml
• --config my_config.yaml
• --config my_config.json

[leondz]

Would recommend accepting yaml, yml, json extensions in both upper & lower case

-- or we could even have --config_json and --config_yaml options, ignoring extension, and default to --config_yaml

[erickgalinkin]

I like both of these ideas, and we should consider just adding error handling here.

[jmartin-tech]

Consider using a fixture to reload before and after each test.

@pytest.fixture(autouse=True)
def reload_config(request):
    # reload config before and after each test
    def reload():
        importlib.reload(_config)

    reload()
    request.addfinalizer(reload)

Testing is needed to determine how this interacts with the allow_site_config fixture which may require marking tests that needs this vs setting autouse=True.

[leondz]

This looks pretty thorough. Maybe we can re-think how we do switching, and put the burden onto the user to specify their intent explicitly wrt. extension, rather than having to guess it.

Would also like deeper testing of edge cases & loading behaviour from YAML and JSON loading, so we know how to write garak configs in both these formats, and so we know we're treating them both evenly and not e.g. misapplying fixes on YAML loading to results from JSON loading

[leondz]

Would recommend accepting yaml, yml, json extensions in both upper & lower case

-- or we could even have --config_json and --config_yaml options, ignoring extension, and default to --config_yaml

[leondz]

what does this if intent to test for? the literal of a .yaml makes me think it could be re-thought

[leondz]

Would appreciate a more detailed test of YAML and JSON loading - perhaps a test version of garak.core.yaml in JSON, for example - to be sure that we're getting the same results out of both YAML and JSON loaders. YAML->Python has pecularities around Nones, for example, and it's not impossible JSON->Python has its own idiosyncracies.

[jmartin-tech]

garak.core.yaml is not standard user controlled as it should only source from the distributed install location. The core config should not support a json version being found. For other files further more complex examples of each format would be reasonable though I think the current tests cover the required set of expectations.