piece-retriever returning partial responses

[pyropy]

During the investigation of errors recently reported in our piece-retriever worker, I discovered what appears to be a critical issue. The errors in question occur when the retrieval worker allegedly hangs and is subsequently terminated by the runtime:

Dec 25, 2025 13:11:03 172.71.47.198  {"Outcome":"exception","Logs":[{"Level":"log","Message":["retrieval request","{\"DNS_ROOT\":\".calibration.filbeam.io\",\"url\":\"https://0x253638142cb4ff04053c14ea1cfa3f963021fb3d.calibration.filbeam.io/bafkzcibfztwyqmaxzn5imaj3i5ekhdhvgh757zqfpua3pobp6ngep6sr43w6zweltytq\"}"],"TimestampMs":1766667992100},{"Level":"log","Message":["Looked up 9 retrieval candidates for piece_cid 'bafkzcibfztwyqmaxzn5imaj3i5ekhdhvgh757zqfpua3pobp6ngep6sr43w6zweltytq' and payer '0x253638142cb4ff04053c14ea1cfa3f963021fb3d'","{}"],"TimestampMs":1766667992314},{"Level":"log","Message":["Attempting retrieval via https://pdp-calib.filweb3.com"],"TimestampMs":1766667992317},{"Level":"log","Message":["Retrieval attempt succeeded"],"TimestampMs":1766667993034},{"Level":"log","Message":["First byte received"],"TimestampMs":1766667993034}],"Exceptions":[{"Name":"Error","Message":"Network connection lost.","TimestampMs":1766668117265},{"Name":"Error","Message":"The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: https://developers.cloudflare.com/workers/observability/errors/","TimestampMs":1766668117265}],"ScriptName":"filbeam-piece-retriever-calibration"}

Summary

It appears that our worker occasionally returns only a partial response. When this occurs, the worker does not read the full stream and fails to log the retrieval.

I have not yet identified the root cause of this issue. Storage providers do not appear to be at fault, as the same providers return proper responses in other requests.

Methodology

I performed these experiments using a local Cloudflare worker connected to the remote database. For each experiment, I hardcoded a wallet address and CID for the piece to be retrieved, then observed both local logs and retrieval logs persisted in the remote database.

Findings

After observing inconsistencies in the first experiment, I conducted several additional tests. In the majority of cases, the full response is returned and results are correctly persisted to the database. However, in some cases the results are absent—indicating the partial response issue occurred.

Experiment 1

CID: bafkzcibfr3v3iliy2nk4t2htpyvo5l77grkocbxiqzr3mnkq6iequhrr3wfa2oc3p4rq
Wallet: 0x4fc1f5bbf39a46842e5704d7b925233863a1f004

Request #	Downloaded Size	Cache Status	Logged to console	SP Used	Notes
1	74.3 MB	—	No	—	Did not resolve successfully
2	152.5 MB	—	No	warp.lotus.dedyn.io	Did not resolve successfully
3	437.4 MB	—	Yes	pdp.laughstorage.com	Successfully resolved, entered ctx.waitUntil block
4–6	437.4 MB	—	No	—	No logging
7	268.6 MB	—	No	—	Partial download
Restart
1	437.4 MB (full)	—	Yes	pdp.topblocks.io	—
2	437.4 MB (full)	—	No	—	—

DB Summary: 8 logged requests, 1 cache-hit, all with egress_bytes = 437,439,090

---

Experiment 2

CID: bafkzcibf3xj2oqqyopdghhdq7zz33xkmmzkyjvwbc6u2xswgpl445y7ooqgs2mi4fqza
Wallet: 0x253638142cb4ff04053c14ea1cfa3f963021fb3d
Fresh start with worker restart and new hardcoded CID/wallet pair

Request #	Downloaded Size	Cache Status	Logged to DB	Logged To Console	SP Used
1	393.6 MB	Miss	Yes	Yes	pdp.laughstorage.com
2	393.6 MB	Miss	Yes	Yes	calib2.ezpdpz.net
3	393.6 MB	Hit	Yes	Yes	calib2.ezpdpz.net
4	393.6 MB	Hit	Yes	Yes	pdp.laughstorage.com
5	393.6 MB	Hit	Yes	Yes	pdp.laughstorage.com
6	393.6 MB	Miss	Yes	Yes	calib.ezpdpz.net
7	393.6 MB	Miss	Yes	Yes	caliberation-pdp.infrafolio.com
8	393.6 MB	Hit	Yes	Yes	calib2.ezpdpz.net
9	393.6 MB	Miss	Yes	Yes	warp.lotus.dedyn.io
10	393.6 MB	Hit	Yes	Yes	calib2.ezpdpz.net

DB Summary: Logged requests show egress_bytes = 393,614,883

---

Experiment 3

CID: bafkzcibftg72eaqvxeoqzcvaneukkn6zblycpe6remqmgr55j6jkaupjffce5cxvea3q
Wallet: 0x253638142cb4ff04053c14ea1cfa3f963021fb3d

Request #	Downloaded Size	Cache Status	Logged to DB	Logged To Console	SP Used
1	61.8	Miss	Yes	Yes	warp.lotus.dedyn.io
2	61.8	Miss	Yes	Yes	calib2.ezpdpz.net
3	61.8	Miss	Yes	Yes	caliberation-pdp.infrafolio.com
4	61.8	Hit	Yes	Yes	caliberation-pdp.infrafolio.com
5	61.8	Miss	Yes	Yes	calib.ezpdpz.net

DB Summary: Logged requests show egress_bytes = 61,825,127

NOTE: I have kept this experiment shorted due to smaller piece size

---

Experiment 4

CID: bafkzcibfssly65yz33v4hm4nzzlgfx2rlcbxjvftnmsgzzb5uwc7ug3aijmqcspome4q
Wallet: 0x4fc1f5bbf39a46842e5704d7b925233863a1f004

Request #	Downloaded Size	Cache Status	Logged to DB	Logged To Console	SP Used
1	815.5	Miss	Yes	Yes	pdp.laughstorage.com
2	815.5	Miss	Yes	Yes	pdp-calib.filweb3.com
3	815.5	Miss	Yes	Yes	pdp.laughstorage.com
4	815.5	Miss	Yes	No	warp.lotus.dedyn.io
5	686.4	-	No	No	-
5	815.5	Miss	Yes	No	pdp.laughstorage.com
6	815.5	Miss	Yes	No	calib2.ezpdpz.net
7	815.5	Yes	Yes	Yes	pdp.660688.xyz:8443
8	815.5	Yes	Yes	Yes	pdp.660688.xyz:8443

Limitations of Performed Experiments

During these experiments, I did not observe any alerts, nor did I retrieve pieces directly from the storage provider before performing retrievals through the worker. This limits our ability to compare expected file size and contents to the response generated by retrieval worker.

Next steps

I recommend repeating these experiments with the following approach:

• First, fetch the piece directly from the storage provider reported in the error
• Perform a retrieval through the remote piece-retriever worker (rather than local)
• Compare the two responses

Following this, we should repeat the retrieval multiple times through the remote piece-retriever worker while monitoring for errors. This would help identify which storage provider the worker used for each retrieval, allowing us to rule out whether a specific storage provider is the root cause rather than a runtime issue.

If the storage provider is not causing this fault, the issue most likely lies in how we read the response through the transformation stream.

[juliangruber]

I have seen similar issues with content validation (?validate=true), where, if the worker ran out of CPU time, the response simply ended. A client might be able to figure out that it was a premature end, but the headers indicate otherwise.

Interesting to see that this also happens without content validation.

[juliangruber]

Hopefully #537 helped with this already

[pyropy]

Looks like #537 did not resolve the issue as alerts are still present

Jan 13, 2026 13:24:13 172.70.71.78  {"Outcome":"exception","Logs":[{"Level":"log","Message":["retrieval request","{\"DNS_ROOT\":\".[calibration.filbeam.io](http://calibration.filbeam.io/)\",\"url\":\"[https://0x4fc1f5bbf39a46842e5704d7b925233863a1f004.calibration.filbeam.io/bafkzcibfxxh424izvan2hsimkeypr6vgn2rkywvjdiee3vk5c5l7hknf3n2potzoiqyq\](https://0x4fc1f5bbf39a46842e5704d7b925233863a1f004.calibration.filbeam.io/bafkzcibfxxh424izvan2hsimkeypr6vgn2rkywvjdiee3vk5c5l7hknf3n2potzoiqyq/)"}"],"TimestampMs":1768310477860},{"Level":"log","Message":["Looked up 8 retrieval candidates for piece_cid 'bafkzcibfxxh424izvan2hsimkeypr6vgn2rkywvjdiee3vk5c5l7hknf3n2potzoiqyq' and payer '0x4fc1f5bbf39a46842e5704d7b925233863a1f004'","{}"],"TimestampMs":1768310477987},{"Level":"log","Message":["Attempting retrieval via https://warp.lotus.dedyn.io/"],"TimestampMs":1768310477987},{"Level":"log","Message":["Cache miss response validation is disabled"],"TimestampMs":1768310478399},{"Level":"log","Message":["Retrieval attempt succeeded (cache miss)"],"TimestampMs":1768310478399},{"Level":"log","Message":["First byte received"],"TimestampMs":1768310478399}],"Exceptions":[{"Name":"Error","Message":"Network connection lost.","TimestampMs":1768310503457},{"Name":"Error","Message":"The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: [https://developers.cloudflare.com/workers/observability/errors/","TimestampMs":1768310503457}],"ScriptName":"filbeam-piece-retriever-calibration"}](https://developers.cloudflare.com/workers/observability/errors/%22,%22TimestampMs%22:1768310503457%7D],%22ScriptName%22:%22filbeam-piece-retriever-calibration%22%7D)

[juliangruber]

Unfortunate! I assume the worker runs out of CPU time. I also assume this only happens on cache misses, is that correct?

[pyropy]

Unfortunate! I assume the worker runs out of CPU time. I also assume this only happens on cache misses, is that correct?

Unfortunately in cases when this timeout happens retrieval logs are not persisted to the database. I can't accurately tell if it only happens on cache-miss but I would assume so. We would have to retry the experiment, maybe with more logging this time, to observe if that really is the case.

[juliangruber]

One potential next step is to move the retrieval log writing to a tail worker, which will always be executed

[juliangruber]

I have a wip for this in #473, wdyt about this strategy?

[pyropy]

One potential next step is to move the retrieval log writing to a tail worker, which will always be executed
I have a wip for this in #473, wdyt about this strategy?

I think it's a perfect fit for what we need, and not coincidentally as documentation states that's exactly what the tail workers should be used for. It should definitely help us out with the debugging.

[FAIZ1409]

Hi 👋
I’d like to work on this issue if it’s still available.
Please let me know if I can proceed or if there are any specific guidelines to follow. Thanks!

[bajtos]

I am afraid this issue may be too complex for a first-time contributor. Having said that, if you have an idea how to approach this, @FAIZ1409, then please leave a comment explaining how you envision fixing the problem.

[juliangruber]

Tackling this one as part of incidents duty this week