feat: handle UnixFS directory entries in ipfs-retriever

[BravoNatalie]

Description

Requesting a path that resolves to a UnixFS directory returns 404 Not Found from processIpfsResponse. The check entry.type !== 'file' && entry.type !== 'raw' hard-rejects directory entries with no explanation, making the error indistinguishable from a missing resource.

The blocks needed for listing are already present: the SP (Curio/frisbii) returns a path-scoped CAR with dag-scope=all, which includes the directory block and all immediate child link metadata. The 404 is a missing implementation, not an architectural constraint.

Proposed solution

In processIpfsResponse, detect entry.type === 'directory' and handle it separately instead of rejecting it. In
v13.7.3, use entry.content() (not entry.entries()) to iterate children. By default it fetches each child block and resolves a full entry with type, name, path, cid, and size, no extra SP requests are needed because frisbii's dag-scope=all already includes all child blocks in the CAR stream, and blockReadConcurrency: 1 ensures they're consumed in the same DFS order frisbii wrote them.

Done criteria

• GET ipfsRootCid/ or ipfsRootCid/subdir/ where it resolves to a UnixFS directory returns an HTML or JSON listing of its children
• Response includes correct Content-Type (text/html or application/json)
• A file path that resolves to a file continues to stream raw bytes as before
• Tests cover directory listing response and unchanged file streaming behaviour

Notes

Once you start consuming entry.content() to build the listing, the CAR stream advances past those child blocks. If you later try to serve one of those children's file bytes using the same blockstore, blocksReader.next() will return the wrong block.

[BravoNatalie]

Not sure why directory entries were excluded from the original PR #312. If there's a valid reason not to support them, we should at least return a descriptive error instead of a generic 404. It's also worth discussing whether supporting full directory listings makes sense here.

[rvagg]

@BravoNatalie also while you have your head in here, I'd consider switching all of these requests to dag-scope=entity which gets you just to the terminal content requested rather than everything. If you hit a directory, it's not going to include all of the children of that directory, just the directory listing itself. You just might need to be careful with unixfs-exporter's traversal, when you get to the terminal thing requested (path/to/thing - i.e. when you find you're at thing) and it's a directory, don't traverse further, just list the directory contents and the CAR will be finished. The ideal here is that you get exactly the blocks you need to serve the request and no more. Unless there are caching reasons to fetch more blocks from the server, but I'm not sure in the typical website serving case that there's going to be any benefits to be found. So when the terminal in the requested path is a directory, all can get you too much, entity should be able to limit it and it also works for files too.