Skip to content

Fix SSL cert verification failures behind enterprise inspection proxies - #168

Open
edurdevic wants to merge 3 commits into
databricks:mainfrom
edurdevic:fix/ssl-cert-enterprise-proxy
Open

Fix SSL cert verification failures behind enterprise inspection proxies#168
edurdevic wants to merge 3 commits into
databricks:mainfrom
edurdevic:fix/ssl-cert-enterprise-proxy

Conversation

@edurdevic

Copy link
Copy Markdown

Problem

ucode init fails with CERTIFICATE_VERIFY_FAILED: self-signed certificate in certificate chain on networks with a corporate TLS inspection proxy, even though curl to the same AI Gateway endpoint succeeds.

Root cause: Python's urllib.request.urlopen does not load the OS trust store on all platforms. The proxy injects a self-signed CA that curl finds via the system keychain, but Python's default SSL context misses it. The error surfaces as:

Databricks Unity AI Gateway probe failed on this workspace
(network error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
self-signed certificate in certificate chain (ssl.c:1001)).

Fix

Add _make_ssl_context() in databricks.py that creates a default SSL context and additionally loads any CA bundle pointed to by REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, or SSL_CERT_FILE — the same env vars honored by curl and the requests library. All three urlopen call sites (_http_get_json, _http_post_json, discover_sql_warehouse_http_path) now use this context.

Workaround for affected customers (before this ships)

export REQUESTS_CA_BUNDLE=/path/to/enterprise-ca.pem
ucode init

Test plan

  • All existing unit tests pass (uv run pytest)
  • Lint clean (uv run ruff check . + uv run ruff format --check)
  • Manual test on a workspace behind an SSL inspection proxy

This pull request and its description were written by Isaac.

Python's urllib does not load the OS trust store on all platforms, so
workspaces behind a corporate TLS inspection proxy (which injects a
self-signed CA) fail with CERTIFICATE_VERIFY_FAILED even though curl
succeeds. Honor REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, and SSL_CERT_FILE
— the same env vars respected by curl and the requests library — so
customers can point ucode at their enterprise CA bundle.

Co-authored-by: Isaac
Comment thread src/ucode/databricks.py Outdated
ctx.load_verify_locations(cafile=ca_bundle)
except ssl.SSLError:
pass
break

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug — break fires even when load_verify_locations fails (_make_ssl_context, line ~214)

          try:
              ctx.load_verify_locations(cafile=ca_bundle)
          except ssl.SSLError:
              pass
          break   # ← outside try/except, always executes

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated with
else:
break

Comment thread src/ucode/databricks.py
_debug(f"databrickscfg ({cfg_path})", f"read error: {exc}")


def _make_ssl_context() -> ssl.SSLContext:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor — _make_ssl_context() called on every request, should be cached

It hits the env vars + filesystem on every urlopen call. The rest of the file already uses @functools.cache for this pattern. Adding @functools.cache makes it a one-shot per process.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added @functools.cache

Comment thread src/ucode/databricks.py Outdated
try:
ctx.load_verify_locations(cafile=ca_bundle)
except ssl.SSLError:
pass

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OSError (e.g. permission denied) is not caught here — Path.is_file() returns True for files the current user cannot read, but load_verify_locations will then raise OSError: Permission denied, which propagates uncaught and crashes the process.

Suggest broadening the except catch

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

broadened to except OSError

Add @functools.cache to _make_ssl_context so the constructed SSL context
is reused across calls. Fix the CA bundle loading loop so a failure to
load one bundle (now catching OSError, not just SSLError) falls back to
the next candidate instead of breaking out. Update the docstring to
mention SSL_CERT_FILE alongside the other env vars. Add tests covering
caching and fallback behavior.
Keep the custom SSL context on the HTTP helpers added on main, including
retried GETs and byte downloads.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants