Using Tanstack DB without fetching the entire collection

[scottmessinger]

Hi! I really like the look of TanStack DB, but I'm pretty puzzled as to how you work with a collection where it's not feasible to fetch the entire collection at once. In working with an app like Notion, it isn't performant or feasible to sync all the data the user might access when the user loads the page. I can't figure out how that would work in TanStackDB, given the queryFn is designed to fetch the entire collection

How would I use TanStackDB in a Notion-like app where the data the user has access vastly exceeds what's possible to download into their browser?

[craigsmitham]

You would need apply filtering in the query function to ensure that the data that is fetched/synced from the server is only what is needed to display/work with for the use case. In some cases where data may only be hundreds of records, you can perhaps always fetch everything. In other cases, consider the logical boundaries the user will be working within for a particular use case. This could be the scope of a page, aggregate root, etc.

[scottmessinger]

Thanks! One thing I couldn't discern from the examples is when the initial fetch would happen. All the examples show using a livequery, so I imagined the fetch would happen on page load. Is the queryFn called on every query?

[KyleAMathews]

Right now it's as @craigsmitham says, you need your collections to fetch subsets of data — as large as makes sense. The plan is fairly soon to have APIs where this is fairly automatic — see #315 & #343

[scottmessinger]

@KyleAMathews Thanks! I’m getting this from every single example on the docs site. In every one, the queryFn fetches the entire collection. Not a single example shows the queryFn fetching a document by id from the server. If there’s an example I’m missing, please let me know! It’s possible I’m just blind as a bat and am missing the page that shows how.

If the answer is I can fetch a single document from the server by id (and then fetch another later), how would that be done?

[KyleAMathews]

@craigsmitham explained one way above

[scottmessinger]

For instance:

const todoCollection = createCollection(
  queryCollectionOptions({
    queryKey: ['todos'],
    queryFn: async () => {
      const response = await fetch('/api/todos')
      return response.json()
    }

All the examples fetch the whole collection, hence my initial question and the examples in my examples of apps where this pattern wouldn’t fit.

I think TanStackDB is really neat, but it seems like it requires radically different way of fetching data, specifically over fetching data by default. I’m trying to find out if a more incremental way of fetching data from the server is possible and if so. how.

How would I incrementally fetch a collection by ID from the server? How would I build a blog site that doesn’t require fetching all the blogs on page load and instead fetches blog posts ID by ID as the user navigates pages? If this is possible, I can’t find an example on the docs (but that might be my own fault for not looking at the right place)

[scottmessinger]

I didn’t see your message when I posted that clarification. Sorry for the confusion!

I’ll reread the posts and think about it next week but none of what I’m hearing suggests the traditional data fetching case is straightforward (though it sounds possible).

If it’s easier than it appears, I’d love to see an example on the docs showing how you’d handle a blog site where a user navigates post by post and only bare minimum of blog posts are fetched. I think your point is “this is not as difficult as you’re making it out to be, Scott” which is probably fair. In which case, an example would really help me see it how you do.

[Julien76]

I would love to see an example of how it can be done in the docs (or here). I find the concept of Tanstack DB really interesting and I see the potential to simplify the codebase of some apps I'm working on and which already use Tanstack Query and Router. But reading multiple times the docs, I find it hard to wrap my head around the "blog/$id" pattern in conjunction with Tanstack DB.

[dipiash]

Hello! I recently faced the same task. In the docs, all the examples show queryFn loading an entire collection, so it feels like TanStack DB is only meant for “load everything at once” scenarios. But how you get your data is completely up to you. In the current version, the DB doesn’t support parameterized queries yet and can run a live query only on the whole collection.

I solved it in a different way: I combined React Query and TanStack DB. React Query handles network requests and parameters (page, number of items, filters), and the DB only stores and returns normalized records.

For each new set of parameters I call useQuery, get the required portion of data from the server, and within one collection I clear the store and insert new items —“wipe & insert”, using writeBatch to delete old keys and insert new data.

The collection is synchronized not through queryFn but directly from the React Query cache (queryClient.getQueryData).

This approach lets you store data by ID, perform optimistic updates, and you don’t need to use a factory to create collections for every filter.

And the DB stores only the current set of data, and React Query handles parameterized loading. Not every app needs to keep dozens of pages in memory at once, and separate collections for each filter can be an overhead. A rewritable collection, which you clear and refill when the parameters change, can be enough.

You can find a more detailed explanation with code and optimistic updates here. I hope this pattern will be helpful!

[thruflo]

That's super cool, thanks for sharing!

[JarvisH]

I have the same question when using live queries with Electric collections. As an example, imagine an archive of items which could run into the millions. How to manage that as electric collections?

[KyleAMathews]

We support query-driven sync now! https://tanstack.com/blog/tanstack-db-0.5-query-driven-sync