[observability] Observability Coverage Report - 2026-05-16

[github-actions[bot]]

Executive Summary

This report reviews 17 completed workflow runs from a repository-wide fetch of 30 recent runs covering the last 7 days. Three runs were still in progress and were excluded because their artifacts were not final. Within the analyzed sample, AWF firewall coverage was 93.8% (15/16 firewall-enabled runs with access.log) and MCP telemetry coverage was 100.0% (15/15 MCP-enabled runs with gateway.jsonl or rpc-messages.jsonl). Combined expected-component coverage was 96.8%, and 16/17 analyzed runs had all expected observability artifacts present.

The only critical gap was Smoke CI §25943259958, a firewall-enabled cancelled run that produced aw_info.json but no sandbox/firewall/logs/access.log, leaving no egress trace for debugging. No MCP-enabled run was missing telemetry. All sampled MCP coverage came from the canonical fallback rpc-messages.jsonl; none of the analyzed runs emitted the preferred gateway.jsonl file.

Seven firewall-enabled runs had non-empty access.log files but only allow-path traffic, so deny-path evidence was not exercised in that part of the sample. A separate cancelled Deployment Incident Monitor run §25944972693 stopped before agent setup and only yielded workflow-level logs, so it was treated as N/A for firewall and MCP coverage rather than counted as a component failure.

Key Alerts and Anomalies

🔴 Critical Issues:

• Smoke CI §25943259958: aw_info.json marks the run as firewall-enabled, but sandbox/firewall/logs/access.log is missing. This is a direct observability gap for network debugging.

⚠️ Warnings:

• 7 firewall-enabled runs had parseable access.log files with blocked_requests = 0, which limits deny-path validation even though the files were present.
• All 15 MCP-enabled runs relied on rpc-messages.jsonl; gateway.jsonl was absent across the analyzed sample, so preferred structured gateway telemetry was not exercised.
• Deployment Incident Monitor §25944972693 cancelled after 1s and exposed only workflow-level logs. It was excluded from firewall/MCP denominators rather than treated as a missing-log defect.

Coverage Summary

Component	Runs Analyzed	Logs Present	Coverage	Status
AWF Firewall (access.log)	16	15	93.8%	🔴
MCP Gateway (gateway.jsonl or rpc-messages.jsonl)	15	15	100.0%	✅

📋 Detailed Run Analysis

Firewall-Enabled Runs

Workflow	Run ID	access.log	Entries	Allowed	Blocked	Status
Smoke CI	§25943259958	❌	0	-	-	🔴
Daily Security Red Team Agent	§25946703186	✅	128	17	0	⚠️
PR Sous Chef	§25946449197	✅	99	21	10	✅
AI Moderator	§25945953859	✅	42	10	0	⚠️
Chaos PR Bundle Fuzzer	§25945620172	✅	188	21	20	✅
Design Decision Gate 🏗️	§25945201695	✅	33	9	0	⚠️
Matt Pocock Skills Reviewer	§25945201669	✅	135	13	10	✅
Test Quality Sentinel	§25945201645	✅	234	25	17	✅
Agentic Workflow Audit Agent	§25942489497	✅	514	38	0	⚠️
Daily Regulatory Report Generator	§25942475951	✅	248	15	14	✅
Smoke CI	§25946204648	✅	11	2	0	⚠️
PR Sous Chef	§25945633237	✅	157	35	17	✅
Design Decision Gate 🏗️	§25945168539	✅	33	9	0	⚠️
Matt Pocock Skills Reviewer	§25945168570	✅	211	15	13	✅
Test Quality Sentinel	§25945168572	✅	195	19	13	✅
AI Moderator	§25943471551	✅	41	9	0	⚠️

Missing Firewall Logs (access.log)

Workflow	Run ID	Date	Link
Smoke CI	25943259958	2026-05-15	§25943259958

MCP-Enabled Runs

Workflow	Run ID	Telemetry Source	Entries	Servers	Tool Calls	Errors	Status
Daily Security Red Team Agent	§25946703186	rpc-messages.jsonl	4	1	1	0	✅
PR Sous Chef	§25946449197	rpc-messages.jsonl	4	1	1	0	✅
AI Moderator	§25945953859	rpc-messages.jsonl	18	2	7	0	✅
Chaos PR Bundle Fuzzer	§25945620172	rpc-messages.jsonl	16	2	6	0	✅
Design Decision Gate 🏗️	§25945201695	rpc-messages.jsonl	4	1	1	0	✅
Matt Pocock Skills Reviewer	§25945201669	rpc-messages.jsonl	10	2	3	0	✅
Test Quality Sentinel	§25945201645	rpc-messages.jsonl	6	1	2	0	✅
Agentic Workflow Audit Agent	§25942489497	rpc-messages.jsonl	16	3	5	0	✅
Daily Regulatory Report Generator	§25942475951	rpc-messages.jsonl	20	3	7	0	✅
Smoke CI	§25946204648	rpc-messages.jsonl	6	2	1	0	✅
PR Sous Chef	§25945633237	rpc-messages.jsonl	12	1	5	0	✅
Design Decision Gate 🏗️	§25945168539	rpc-messages.jsonl	4	1	1	0	✅
Matt Pocock Skills Reviewer	§25945168570	rpc-messages.jsonl	16	2	6	0	✅
Test Quality Sentinel	§25945168572	rpc-messages.jsonl	8	1	3	0	✅
AI Moderator	§25943471551	rpc-messages.jsonl	18	2	7	0	✅

Missing MCP Telemetry (gateway.jsonl and rpc-messages.jsonl both absent)

None in the analyzed sample.

🔍 Telemetry Quality Analysis

Firewall Log Quality

• Raw access.log entries analyzed: 2269.
• Summarized firewall requests: 372 total, 258 allowed and 114 blocked (30.6% blocked).
• Domains observed: 9 unique.
• Most accessed destinations: api.githubcopilot.com:443 (161), api.anthropic.com:443 (68), and github.com:443 / api.openai.com:443 (11 each). Blocked requests were summarized under (unknown) in the aggregated firewall report.

Gateway Log Quality

• Telemetry source coverage: rpc-messages.jsonl in 15/15 MCP-enabled runs; gateway.jsonl in 0/15.
• Total MCP telemetry entries analyzed: 162.
• MCP servers used: agenticworkflows, github, mcpscripts, safeoutputs.
• Total tool calls: 56.
• Error rate: 0% (0 errors across 162 entries).
• Average response time: N/A from the sampled rpc-messages.jsonl files because request/response IDs were not pairable for duration calculation.

Healthy Runs Summary

• 8 runs showed both allowed and blocked firewall traffic plus healthy MCP telemetry.
• 7 additional runs had complete files but only allow-path firewall traffic.
• 1 cancelled run missed access.log after firewall enablement.
• 1 cancelled run ended before agent setup and stayed N/A for component coverage.

Recommended Actions

1. Investigate why Smoke CI §25943259958 cancelled after aw_info.json creation but before firewall log export, and ensure access.log is persisted even on early cancellation paths.
2. Add or preserve a minimal deny-path probe in workflows that already use the firewall so sampled runs produce both allowed and blocked entries more consistently.
3. Start emitting mcp-logs/gateway.jsonl alongside rpc-messages.jsonl, or document fallback-only operation as intentional, so structured latency and status metrics are available without extra reconstruction.

📊 Historical Trends

Historical trend data was not derived in this run. This report is a 7-day point-in-time sample built from a single repository-wide fetch, with breadth preserved by excluding 3 in-progress runs and limiting repeated workflows to at most 2 samples each.

References: §25943259958, §25944972693, §25942475951

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

• api.github.com
• github.com

💡 Tip: api.github.com is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding api.github.com to network.allowed, use tools.github.mode: gh-proxy for direct pre-authenticated GitHub CLI access without requiring network access to api.github.com:

tools:
  github:
    mode: gh-proxy

See GitHub Tools for more information on gh-proxy mode.

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.github.com"
    - "github.com"

See Network Configuration for more information.

Generated by 📊 Daily Observability Report for AWF Firewall and MCP Gateway · ◷

• expires on May 17, 2026, 12:16 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Daily Observability Report for AWF Firewall and MCP Gateway.

A newer discussion is available at Discussion #32716.