Skip to content

Improve AI Gateway diagnostics during configure - #453

Merged
david-siqi-liu merged 8 commits into
mainfrom
david/es-2185388-gateway-capabilities
Sep 2, 2026
Merged

Improve AI Gateway diagnostics during configure#453
david-siqi-liu merged 8 commits into
mainfrom
david/es-2185388-gateway-capabilities

Conversation

@david-siqi-liu

@david-siqi-liu david-siqi-liu commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

What did you change, and why?

Change: ucode configure now reports whether it can access a Unity Catalog model service. If no accessible model service is returned, it probes legacy endpoints internally to determine whether Unity AI Gateway is still usable. The user-facing success output remains focused on the model service result.

Why: Today, ucode configure provides little telemetry about the discovery path it can access. Users can eventually see a CSP-related fallback error even when the actionable problem is missing access to system.ai. Reporting the model-service result makes that failure clear, while avoiding an unnecessary legacy probe when model-service discovery already succeeds.

Related: ES-2185388

Expected output by scenario

An accessible model service is returned

✔ Unity AI Gateway detected
  Model service: reachable, accessible model service returned

The legacy endpoint API is not probed.

Model-service discovery is reachable but returns nothing accessible, and the legacy endpoint API is reachable

✔ Unity AI Gateway detected
  Model service: reachable, no accessible model services returned; check USE CATALOG on system, and USE SCHEMA and EXECUTE on system.ai

This applies whether the legacy endpoint listing is empty or contains an accessible endpoint. The fallback is checked internally and is not printed.

Model-service discovery is unavailable or forbidden, and the legacy endpoint API is reachable

✔ Unity AI Gateway detected
  Model service: HTTP 403 Forbidden: ...

The exact model-service reason is printed after replacing any API-version terminology with resource-oriented language. The reachable legacy endpoint fallback is not printed.

Model-service discovery is reachable but empty, and the legacy endpoint API is unavailable

No success summary is printed. Configuration raises an actionable model-service access error with this guidance:

Verify USE CATALOG on `system`, and USE SCHEMA and EXECUTE on `system.ai`.

Model-service discovery is unavailable, and legacy endpoint access is forbidden

No success summary is printed. Configuration raises an actionable legacy endpoint permission error.

Neither discovery path is reachable

No success summary is printed. Configuration reports that Unity AI Gateway is not enabled and includes both resource-oriented failure reasons.

The access token is invalid

No success summary is printed. Configuration stops immediately with databricks auth logout and databricks auth login guidance. If model-service discovery identifies the invalid token, the legacy endpoint API is not probed.

A successful empty model-service response, including HTTP 200 {}, confirms that the API is reachable. It does not prove that the caller can access a model service.

How do you know it works?

  • Added deterministic local scenarios that drive the real probe formatter through configure_shared_state.
  • Verifies that an accessible model service skips the legacy endpoint request.
  • Verifies that legacy endpoint results are used only to validate the fallback and are not printed.
  • Covers empty and forbidden model-service results with reachable, empty, forbidden, and unavailable legacy endpoint discovery.
  • Covers neither discovery path being reachable and invalid access tokens.
  • Verifies that user-facing errors do not expose API-version terminology.
  • Added focused probe coverage for non-empty and empty responses, HTTP failures, and the system.ai permission hint.
  • Isolated publish CLI wiring tests from the real Databricks CLI installer so CI does not depend on network bootstrap.
  • uv run ruff check . and uv run ruff format --check src/ tests/ passed.
  • The suite passed with 2176 passed, 37 skipped, 1 deselected. The deselected live Claude user-agent E2E is blocked in this development environment because system-managed Claude settings override its localhost capture server.

@david-siqi-liu david-siqi-liu changed the title Show AI Gateway versions during configure Report AI Gateway V2 and V3 availability during configure Sep 2, 2026
@david-siqi-liu david-siqi-liu changed the title Report AI Gateway V2 and V3 availability during configure Improve AI Gateway diagnostics during configure Sep 2, 2026
Comment thread src/ucode/databricks.py Outdated
Comment thread src/ucode/databricks.py Outdated
Comment thread src/ucode/databricks.py Outdated
Comment thread src/ucode/databricks.py Outdated
@david-siqi-liu
david-siqi-liu force-pushed the david/es-2185388-gateway-capabilities branch from 0508cb9 to 7a2446b Compare September 2, 2026 17:10
@david-siqi-liu
david-siqi-liu enabled auto-merge (squash) September 2, 2026 17:54
@david-siqi-liu
david-siqi-liu merged commit d631ddc into main Sep 2, 2026
2 checks passed
@david-siqi-liu
david-siqi-liu deleted the david/es-2185388-gateway-capabilities branch September 2, 2026 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants