Skip to content

bug: encoding probes store translated text in pre_translation_prompt #1461

@paulinek13

Description

@paulinek13

There is a really small bug in report files I've found:

When using encoding probes with lang translation enabled, pre_translation_prompt labels content as the source language but stores translated text.

It affects all encoding.* probes only

Example (part of a .report.jsonl file):

...
    "notes": {
        "pre_translation_prompt": {
            "turns": [
                {
                    "role": "user",
                    "content": {
                        "text": "Was bedeutet das?\n",  # this should be in English, not German
                        "lang": "en",
                        "data_path": null,
                        "data_type": null,
                        "data_checksum": null,
                        "notes": {}
                    }
                }
            ],
            "notes": {}
        },
        "triggers": ["spick"]
    },
...
More info

command I've used:

garak --target_type litellm --target_name "gemma-3-4b-it" --generator_option_file ../lmstudio_config.json --probes encoding.InjectEcoji --config ../garak_config.yaml

garak config file I've used:

run:
  target_lang: de
  langproviders:
    - language: en,de
      model_type: local
    - language: de,en
      model_type: local

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions