Skip to content

Slow get() #1893

@t0mstah

Description

@t0mstah

Related issues

Possibly related at the underlying gRPC layer, though not a firebase-functions issue:

[REQUIRED] Version info

node:

v22.22.2

firebase-functions:

7.2.5

firebase-tools:

15.18.0

firebase-admin:

13.10.0

Also observed with:

  • @google-cloud/firestore: 7.11.6
  • @grpc/grpc-js: 1.14.4

[REQUIRED] Test case

I have a Firebase Gen 2 HTTP function using onRequest. The function is public/unauthenticated at the Cloud Run/
Firebase Functions layer because it handles redirect/callback traffic.

Inside the handler, it performs a simple Firestore Admin SDK document read:

let start = performance.now()
let doc = await admin.firestore().collection('users').doc(userId).get()
let userGetMs = performance.now() - start

console.log({
  message: 'timing',
  userId,
  userGetMs
})

The users/{userId} documents are normal-sized documents, usually around 4KB-14KB in the slow samples I checked. The
issue is not limited to a single user document, not obviously related to document size, and not obviously caused by
same-document write contention.

### [REQUIRED] Steps to reproduce

1. Deploy a Gen 2 Firebase onRequest function on Node 22.
2. Use firebase-admin Firestore from inside the function.
3. Keep instances warm with minInstances.
4. Send normal production traffic through the function.
5. Log timing around simple Firestore Admin SDK doc().get() calls.
6. Observe the long-tail latency over time on warm instances.

In my production service, p50/p95 reads are usually normal, but a small percentage of reads intermittently take multiple
seconds. I have observed individual simple users/{userId}.get() calls taking 5s, 10s, and in one recent case 23s.

Revising/flushing the Cloud Run service with a no-code gcloud run services update immediately clears the issue
temporarily, which suggests stale/degraded per-instance client/channel/runtime state rather than Firestore document size
or application-level contention.

### [REQUIRED] Expected behavior

A simple Admin SDK Firestore doc().get() for a small document should not intermittently take 5s-20s+ on otherwise
healthy warm Gen 2 function instances.

If Firestore is temporarily slow, I would expect the issue to be visible as broader service latency, errors, retries, or
some actionable signal. If a client/channel becomes unhealthy, I would expect it to recover or fail clearly rather than
remaining in a degraded state until the Cloud Run instance is replaced.

### [REQUIRED] Actual behavior

After upgrading/moving this workload to Gen 2, I am seeing intermittent very slow Firestore reads inside public
onRequest functions.

The observed pattern:

- Simple admin.firestore().collection('users').doc(userId).get() calls occasionally take 5s-20s+.
- The same collection and similar user reads from onCall functions appear to return normally.
- The issue is concentrated in high-traffic public onRequest redirect/callback handlers.
- CPU and memory are not saturated.
- Cloud Run request concurrency is low.
- Slow reads are not concentrated on one user id.
- Slow user documents are not unusually large.
- Firestore audit logs do not show same-document writes during the slow read windows.
- A no-code Cloud Run revision update/flush clears the latency temporarily.
- Redeploying the function can cause the problem to return later after traffic warms up.

Example recent timing data:

- userGetMs: 23220ms, document size about 10KB, no same-user commit during the read window.
- Multiple other userGetMs values in the 3s-6s range.
- Normal p50/p95 in the same window remained around tens of milliseconds.

This looks like some warm Gen 2 instances may develop degraded Firestore client/channel behavior where individual Admin
SDK Firestore reads stall for seconds, while the process remains otherwise healthy.

### Were you able to successfully deploy your functions?

Yes. Deployment succeeds. There are no deploy errors. The issue is runtime latency after successful deployment.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions