Sync GAPIC client construction performs blocking metadata-server DNS + HTTP via google.auth.default(), stalling asyncio callers

[JakeSummers]

Summary

Constructing any sync GAPIC client with no explicit credentials performs blocking network I/O on the calling thread. When that thread is an asyncio event loop, the whole loop stalls. Nothing in the docstrings or docs says construction does I/O, and the recommended-looking usage ("instantiate this client with no arguments") is the one that triggers it.

Versions: google-cloud-pubsub 2.29.0, google-auth 2.39.0, google-api-core 2.24.2, Python 3.12.

Repro

import asyncio, time
from google.cloud import pubsub_v1

async def ticker():
    while True:
        t = time.perf_counter()
        await asyncio.sleep(0.01)
        lag = (time.perf_counter() - t - 0.01) * 1000
        if lag > 5:
            print(f"loop lag {lag:.0f}ms")

async def main():
    asyncio.create_task(ticker())
    await asyncio.sleep(0.2)
    pubsub_v1.PublisherClient()   # blocks the loop

asyncio.run(main())

On GCE/GKE with Workload Identity and no explicit credentials this reports tens to thousands of ms of loop lag, per construction. It reproduces on every sync GAPIC client, not just Pub/Sub.

Where the I/O happens

1. pubsub_v1/publisher/client.py — Client.__init__ → super().__init__(**kwargs)
2. pubsub_v1/services/publisher/transports/base.py:105-107 — elif credentials is None and not self._ignore_credentials: credentials, _ = google.auth.default(...)
3. google/auth/_default.py:650 — runs the ADC checker chain; on GCE the first three checkers miss and it reaches _get_gce_credentials
4. google/auth/_default.py:350 — _metadata.is_on_gce() → ping() → HTTP GET http://169.254.169.254 (_metadata.py:48-51), up to 3 attempts with exponential backoff
5. google/auth/_default.py:353 — _metadata.get_project_id() → HTTP GET http://metadata.google.internal/computeMetadata/v1/project/project-id (_metadata.py:40-45), up to 5 attempts. This one resolves a hostname, so it includes a blocking getaddrinfo.
6. google/auth/transport/_http_client.py:97 — each call builds a fresh http.client.HTTPConnection; no connection reuse, so every construction pays a new resolve and a new TCP connect.

There is no caching at any layer: google.auth.default() is re-run in full for every client constructed.

Worth noting for anyone trying to mitigate this with environment variables: setting GOOGLE_CLOUD_PROJECT does not avoid the project-id fetch. _get_gce_credentials calls get_project_id() unconditionally at _default.py:353, and explicit_project_id is only read at _default.py:635 and applied at _default.py:658 — after both HTTP calls have already happened. Only GCE_METADATA_HOST set to a literal IP removes the getaddrinfo from the path.

Impact (production)

A FastAPI/uvicorn service on GKE with Workload Identity constructed a PublisherClient() per publish — which looked safe, since nothing documents construction as doing I/O.

When cluster DNS began dropping answers, ndots:5 plus six search domains turned each metadata.google.internal lookup into 12+ queries, each dropped one costing a 5s glibc timeout. Every request stalled the single event loop; p99 latency rose roughly an order of magnitude for hours, and a meaningful fraction of requests timed out.

py-spy dump caught MainThread blocked in getaddrinfo inside PublisherClient.__init__ in a significant fraction of samples, and the process accumulated hundreds of threads from leaked clients.

That thread growth is a secondary effect of per-call construction: each fresh client starts its own batch-commit thread (pubsub_v1/publisher/client.py:530-543, _batch/thread.py:231-238) and leaks its gRPC channel.

Measurement: the cost is credential resolution, not gRPC

Constructing the client repeatedly with PUBSUB_EMULATOR_HOST set — which injects AnonymousCredentials at pubsub_v1/publisher/client.py:138 and skips ADC entirely — costs ~0.08 ms after warmup. The same construction with real ADC costs ~460 ms, and google.auth.default() alone accounts for essentially all of it.

gRPC channel creation is lazy-connect and effectively free. Credential resolution is the entire cost.

Why this is worth changing

Blocking network I/O in a constructor is invisible to callers and is a well-known hazard for async code. Two things make it worse here:

• The docstring actively encourages the triggering pattern and never mentions I/O or reuse: "Generally, you can instantiate this client with no arguments, and you get sensible defaults."
• It cannot be caught in testing. With PUBSUB_EMULATOR_HOST set, pubsub_v1/publisher/client.py injects AnonymousCredentials and google.auth.default() is never reached; locally, ADC resolves from a cached file. The metadata-server path only exists in production.

Ask

In order of preference:

1. Defer credential resolution out of __init__ — resolve lazily on first RPC, where a caller can already expect I/O.
2. Or memoize the ADC result process-wide, so repeated construction costs one resolution rather than N.
3. Or, at minimum, document it: state in the PublisherClient / GAPIC client docstrings and in the auth docs that construction performs blocking network I/O (metadata-server HTTP plus a DNS lookup on GCE), and that clients should be constructed once and reused — explicitly off the event loop for asyncio callers.

Even (3) alone would have prevented this incident.

Related

• Publisher thread terminates, forever breaking publication when GCE metadata service blips python-pubsub#1173 — metadata-server blip kills the publisher commit thread. Confirms synchronous metadata calls live in this path; no asyncio angle.
• Async subscriber should raise RuntimeError outside event loop, streaming pull fails python-pubsub#1279 — async client built outside a running loop hangs; the ADC check appears in the debug log but is not identified as the cause.
• auth.default() takes 440ms google-auth-library-python#285 — "auth.default() takes 440ms". Pure perf observation, no blocking framing. Matches the ~460 ms measured above.
• Add asyncio/tornado support google-auth-library-python#321 — "add asyncio support", the general sync-auth-stack-vs-async complaint; does not name client construction.

Filed here rather than against python-pubsub / google-auth-library-python / python-api-core, since those are archived and read-only. The behaviour spans packages/google-auth (where the resolution happens) and packages/google-cloud-pubsub (where it is triggered).

[chalmerlowe]

@JakeSummers Thank you for reporting this. We appreciate the thoroughness of this report.