question: clarifying `lang` in `Attempt` constructor signature

[leondz]

How should we best use the lang parameter be used in the Attempt constructor?

1. lang is important at Message level
2. Attempt prompts and Turns use Message
3. It's not clear to me if an Attempt-level language code/spec is meaningful - this might be best described at the individual messages comprising prompts and outputs
4. That we allow lang in Attempt constructor hints that there is a meaningful attempt-level lang concept
5. Attempt has no lang member
6. The Attempt constructor value lang is only consumed if a prompt is passed (which will be removed in arch: rm support for init'ing attempt.prompt with str #1361)

I'd like to propose deprecating & removing this param, possibly in #1361, and support only Message-based prompt creating. This makes it clear where the language information goes and at what level it applies.

An alternative could be to rename this param to something like prompt_lang.

---

post-script: Currently we use this in a range of tests - but not comprehensively. A more consistent approach is desireable.

(garak) 12:50:11 x1:~/dev/garak [main] $ grep -r Attempt tests/* --include="*py" | grep lang
tests/buffs/test_buffs.py:    a = attempt.Attempt(lang=b.lang)
tests/detectors/test_detectors_shields.py:        attempt = Attempt(prompt="test", lang=up_detector.lang_spec.split(",")[0])
tests/detectors/test_detectors_shields.py:        attempt = Attempt(prompt="test", lang=up_detector.lang_spec.split(",")[0])
tests/detectors/test_detectors_shields.py:    attempt = Attempt(prompt="test", lang=detector.lang_spec.split(",")[0])
tests/detectors/test_detectors_shields.py:        attempt = Attempt(prompt="test", lang=down_detector.lang_spec.split(",")[0])
tests/detectors/test_detectors_shields.py:        attempt = Attempt(prompt="test", lang=down_detector.lang_spec.split(",")[0])
tests/detectors/test_detectors_shields.py:    attempt = Attempt(prompt="test", lang=detector.lang_spec.split(",")[0])
tests/detectors/test_detectors_shields.py:    attempt = Attempt(prompt="test", lang=up_detector.lang_spec.split(",")[0])
tests/detectors/test_detectors_shields.py:    attemptd = Attempt(prompt="test", lang=down_detector.lang_spec.split(",")[0])
tests/detectors/test_detectors_shields.py:    attempt = Attempt(prompt="test", lang=up_detector.lang_spec.split(",")[0])
tests/detectors/test_detectors_shields.py:    attemptd = Attempt(prompt="test", lang=down_detector.lang_spec.split(",")[0])
tests/detectors/test_detectors_ansiescape.py:        a = garak.attempt.Attempt(lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_ansiescape.py:        a = garak.attempt.Attempt(lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_ansiescape.py:        a = garak.attempt.Attempt(lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_ansiescape.py:        a = garak.attempt.Attempt(lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_malwaregen.py:        a = garak.attempt.Attempt(lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_malwaregen.py:        a = garak.attempt.Attempt(lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_encoding.py:    a = Attempt(prompt="test text", lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_encoding.py:    a = Attempt(prompt="test text", lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_unsafe_content.py:    a = Attempt(prompt="test prompt", lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_mitigation.py:    attempt = Attempt(prompt="testing prompt", lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_misleading.py:    a_ref = Attempt(prompt="prompt", lang="en")
tests/detectors/test_detectors_misleading.py:    a_non_ref = Attempt(prompt="prompt", lang="en")
tests/detectors/test_detectors_productkey.py:    test_attempt = Attempt(lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_productkey.py:    test_attempt = Attempt(lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_productkey.py:    test_attempt = Attempt(lang=d.lang_spec.split(",")[0])
tests/detectors/test_detectors_exploitation.py:    a = garak.attempt.Attempt(prompt="test prompt", lang="*")
tests/detectors/test_detectors_exploitation.py:    a = garak.attempt.Attempt(prompt="test prompt", lang="*")
tests/detectors/test_detectors_exploitation.py:        a = garak.attempt.Attempt(prompt="test prompt", lang="*")
tests/detectors/test_detectors_exploitation.py:    a = garak.attempt.Attempt(prompt="test prompt", lang="*")
tests/detectors/test_detectors_exploitation.py:    a = garak.attempt.Attempt(prompt="test prompt", lang="*")
tests/detectors/test_detectors_packagehallucination.py:    import_attempt = Attempt(prompt=Message(text=f"give me some {lang.title()}"))
tests/detectors/test_detectors_packagehallucination.py:    import_attempt = Attempt(prompt=Message(text=f"give me some {lang.title()}"))
tests/langservice/detectors/test_detectors_leakreplay.py:from garak.attempt import Attempt, Message
tests/langservice/detectors/test_detectors_leakreplay.py:def reverse_translate(attempt: Attempt) -> Attempt:
tests/langservice/detectors/test_detectors_leakreplay.py:    attempt = Attempt(prompt=Message("This is a test prompt"))
tests/langservice/detectors/test_detectors_misleading.py:from garak.attempt import Attempt, Message, Conversation, Turn
tests/langservice/detectors/test_detectors_misleading.py:def reverse_translate(attempt: Attempt) -> Attempt:
tests/langservice/detectors/test_detectors_misleading.py:    attempt = Attempt(
tests/langservice/detectors/test_detectors_misleading.py:    # Create a sample Attempt with Japanese text
tests/langservice/detectors/test_detectors_misleading.py:    attempt = Attempt(
tests/langservice/detectors/test_detectors_snowball.py:from garak.attempt import Attempt, Message
tests/langservice/detectors/test_detectors_snowball.py:def reverse_translate(attempt: Attempt) -> Attempt:
tests/langservice/detectors/test_detectors_snowball.py:    attempt = Attempt(prompt=Message("PLACEHOLDER"), lang="ja")
tests/langservice/detectors/test_detectors_snowball.py:    attempt = Attempt(prompt=Message("PLACEHOLDER"), lang="ja")
tests/langservice/probes/test_probes_base.py:from garak.attempt import Message, Attempt
tests/langservice/probes/test_probes_base.py:    a = Attempt(prompt="just a test attempt", lang="fr")
tests/test_attempt.py:    constructor_attempt = garak.attempt.Attempt(prompt=test_text, lang="*")
tests/test_attempt.py:    att = garak.attempt.Attempt(prompt=test_prompt, lang="*")
tests/test_attempt.py:    att = garak.attempt.Attempt(prompt="well hello", lang="*")
tests/test_internal_structures.py:    attempts = (garak.attempt.Attempt(prompt=str(i), lang="*") for i in range(count))
tests/test_internal_structures.py:    a = garak.attempt.Attempt(prompt="fish", lang="*")
tests/test_internal_structures.py:    attempt = garak.attempt.Attempt(prompt="testing prompt", lang=d.lang_spec)

[jmartin-tech]

Note currently lang for an Attempt is derived from the lang value for the first message in the prompt. If lang is passed to the constructor it is used to decorate the first Message in the Conversation stored in the prompt.

There should likely be more guards to make this coupling clear.

[leondz]

Do you think there's any reason to not remove this once support for str-type prompts in Attempt constructors is dropped?

[jmartin-tech]

I think the helper on Attempt could stay, however once the Attempt constructor no longer accepts str I agree it also will not need to accept a lang value either.