Authenticate A2A requests and fix jwt-validation pipeline reload - #46
Merged
kellyaa merged 5 commits intoJul 27, 2026
Merged
Conversation
Wire a Keycloak bearer token through the A2A evaluation path so requests are accepted by the agent's authbridge sidecar, and fix the pipeline merge so jwt-validation reloads instead of failing with "issuer is required". - a2a_client: attach Authorization: Bearer from config.auth_token (A2A_AUTH_TOKEN) to the agent-card fetch and every send_message. - evaluate-benchmark.sh: obtain a rossoctl-realm token via the direct-access-grant flow when A2A_AUTH_TOKEN is unset. - pipeline-merge.py: seed each plugin entry from the operator's rendered config block (deep-merge fragment + overrides on top) instead of replacing the plugin list wholesale, preserving operator-only config such as jwt-validation's issuer and token-exchange's token_url. - jwt-validation.yaml: render issuer/keycloak_url/keycloak_realm from the operator authbridge-config ConfigMap; document the DisallowUnknownFields schema constraint. - apply-pipeline.sh: source JWT_VALIDATION_* from the base ConfigMap and pre-flight fail if the issuer can't be resolved while jwt-validation is active. - deploy-agent.sh: fail fast with a clear message when python3/PyYAML is missing and a plugin selector was supplied. - run-ibac-comparison.sh: new wrapper to run a benchmark with a plugin preset vs a baseline and compare. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Yoav Katz <katz@il.ibm.com>
Surface A2A execution failures as errors, fix trace token accounting to the current OpenTelemetry gen_ai keys, and make comparison experiment names unique per run. - a2a_client: add A2ATaskError and inspect the task's terminal state; a task ending in failed/rejected/canceled is raised as an error even when it returned descriptive text, so the runner marks Agent.Call / Agent.Session spans ERROR instead of counting it as a success. - analyze_traces: read token usage from gen_ai.usage.input_tokens/ output_tokens (the keys current tracers emit) rather than the stale llm.token_count.prompt/completion path. - run-ibac-comparison.sh: append a short random id to experiment names so repeated runs with the same params don't collide; read /dev/urandom via a bounded head to avoid an intermittent SIGPIPE-induced exit under pipefail + set -e. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Yoav Katz <katz@il.ibm.com>
Add an "IBAC Comparison Script" section describing the plugin-vs-baseline comparison wrapper: what it does, the shared random experiment id, the judge env requirements, usage examples, and an options table matching the script's --help defaults. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Yoav Katz <katz@il.ibm.com>
Add an Error Rate (%) row to the --compare/-c report, printed just before Eval Success Rate. Uses the same ERROR-status definition as the per-parallel and per-group tables. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Yoav Katz <katz@il.ibm.com>
The --compare/-c report now includes an "Error Breakdown by Type" table below the main comparison, counting ERROR-status root spans by their verbatim statusMessage per experiment, shown as count (pct%). - Thread statusMessage from the root Agent.Session span into TraceRecord. - Group ERROR traces by first line of statusMessage; blank -> "(no message)". Strip the "A2A task ended in state 'failed':" prefix and truncate labels to 100 chars. - Rows are the union of error types across experiments, sorted by total frequency; the section always prints, showing "No errors" when empty. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Yoav Katz <katz@il.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wire a Keycloak bearer token through the A2A evaluation path so requests are accepted by the agent's authbridge sidecar, and fix the pipeline merge so
jwt-validationreloads instead of failing withissuer is required.Changes
a2a_client.py— attachAuthorization: Bearerfromconfig.auth_token(A2A_AUTH_TOKEN) to the agent-card fetch and everysend_message.evaluate-benchmark.sh— obtain arossoctl-realm token via the direct-access-grant flow whenA2A_AUTH_TOKENis unset (mirrors the deploy flow).pipeline-merge.py— seed each plugin entry from the operator's rendered config block (deep-merging fragment + overrides on top) instead of replacing the plugin list wholesale, preserving operator-only config such asjwt-validation's issuer andtoken-exchange'stoken_url.plugins/jwt-validation.yaml— renderissuer/keycloak_url/keycloak_realmfrom the operatorauthbridge-configConfigMap; document theDisallowUnknownFieldsschema constraint.apply-pipeline.sh— sourceJWT_VALIDATION_*from the base ConfigMap and pre-flight fail if the issuer can't be resolved whilejwt-validationis active.deploy-agent.sh— fail fast with a clear message whenpython3/PyYAML is missing and a plugin selector was supplied.run-ibac-comparison.sh— new wrapper to run a benchmark with a plugin preset vs a baseline and compare.Test plan
issuer is required.evaluate-benchmark.shand confirm A2A requests are authenticated (no HTTP 401).run-ibac-comparison.shend-to-end.🤖 Generated with Claude Code