🪲 ORCA tests report incorrect failures when using certificate-based service principal authentication

[chriskeavey-kkl]

Description

ORCA tests report incorrect failures in Pester/HTML output when using certificate-based service principal authentication, despite the same tests passing when run individually or with interactive authentication.

Environment

• Maester Version: 2.0.0
• PowerShell Version: 7.x
• Authentication Method: Certificate-based with service principal
• Connection: Custom script using Connect-ExchangeOnline, Connect-MgGraph, etc. with certificate thumbprint

Steps to Reproduce

1. Connect using service principal with certificate authentication
2. Run Invoke-Maester -SkipGraphConnect
3. Observe ORCA tests show failures in HTML report
4. Run individual ORCA tests (e.g., Test-ORCA106) - they pass and show correct results
5. Compare with Connect-Maester -Services All (interactive) - ORCA tests pass correctly

Expected Behavior

ORCA tests should report accurate pass/fail results regardless of authentication method

Actual Behavior

• Interactive auth: ORCA tests pass correctly in HTML
• Service principal auth: Majority of ORCA tests report as failures in HTML/Pester output
• Running tests individually with service principal: Tests pass and show correct evaluation
• Manually verified configurations match expected values

Diagnostic Information

• Test-MtConnection returns True for all services (Graph, ExchangeOnline, SecurityCompliance, Azure)
• Verbose output shows: VERBOSE: OrcaCache not set, Get-ORCACollection on every test (cache not persisting)
• Non-ORCA tests (EIDSCA, CIS, CISA, Maester) work correctly with service principal
• All required API permissions and RBAC roles configured

Additional Context

• 99 ORCA tests reported as failed with service principal
• Many tests pass when run with Connect-Maester (interactive)
• Suggests issue in ORCA test framework's handling of service principal context in Pester runs

[chriskeavey-kkl]

All ORCA tests don't display result details in HTML reports when connected with service principal.

A couple don't display result details in HTML reports when run interactively, but most do. ORCA.232 does not show result detail with either method.

[soulemike]

Hey @chriskeavey-kkl thank you for this. I have been doing some testing and see similar results, but I can't figure out why. Have you tested this further?

Here is what I have tried all with certificate based authentication:

• Docker container run noninteractive, ORCA tests fail as shown above
• Same Docker container run interactive, ORCA tests reflect correctly
• Manual run of tests, ORCA tests reflect correctly

My only thought is updating my Docker image to use verbose and try to capture something more or do a new image just with a single ORCA test running verbose. But wanted to follow up with you to see if you have had any luck here.