Skip to content

Corrupted responses (bad Content-Length/Content-Encoding) #13353

Description

@ant5

This is a hard to identify bug. From the end-user view it appeared in browser console as net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK).
In ATS access log it may (*) appear as ERR_READ_ERROR/200

The problematic URL sporadically changes. Site can show 0-3 errors of some URLs per page loading. When go to another page of the same site the situation repeat itself.

There is a one catched problematic transaction client <->ATS assembled from the wire:

sample 1

GET /wp-content/themes/epoxyworks/dist/quote-block-overlay-daFJl7Sw.svg HTTP/1.1

Host: www.epoxyworks.com
Pragma: no-cache
sec-ch-ua-platform: "Windows"
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36
sec-ch-ua: "Google Chrome";v="149", "Chromium";v="149", "Not)A;Brand";v="24"
sec-ch-ua-mobile: ?0
Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: image
Referer: https://www.epoxyworks.com/
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9,ru;q=0.8,pl;q=0.7
Cookie: cookieyes-consent=consentid:MmxnZDMwaG5tdWJ4SVpmQVRPZjhGRzhoMXZMMUhwRWM,consent:no,action:,necessary:yes,functional:no,analytics:no,performance:no,advertisement:no; _ga=GA1.1.2040193634.1778861584; _clck=v0s5jz%5E2%5Eg62%5E0%5E2326; _ga_8ZFKGQ7DBS=GS2.1.s1781793747$o3$g1$t1781793790$j17$l0$h0
Via: 1.1 z101.orgs.link (squid)
Cache-Control: no-cache
Connection: close
X-Forwarded-Proto: https

--

HTTP/1.1 200 OK

Date: Fri, 26 Jun 2026 10:15:42 GMT
Content-Type: image/svg+xml
Content-Length: 32578
Last-Modified: Fri, 29 May 2026 20:24:08 GMT
Accept-Ranges: bytes
Cache-Control: public, must-revalidate, proxy-revalidate, immutable, max-age=31536000, stale-while-revalidate=86400, stale-if-error=604800
Expires: Sat, 24 Oct 2026 10:15:41 GMT
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
Server: ATS/10.2.0
nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
cf-cache-status: MISS
speculation-rules: "/cdn-cgi/speculation"
report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=OPavTOqmnCcehb%2BByumlVeIAypgiLckUg3aRv1qg%2Bwh5nd6SKj1zthsQ30Msq87bgdws2nLBfAvOzYne0o5KkOkmEaqJMws0h3IeETdvPLKwFSsY7Yb68xuWZXWGJdqACD7dd40%3D"}]}
cf-ray: a11b73c6abbb8055-ARN
alt-svc: h3=":443"; ma=86400
Age: 0
Connection: Keep-Alive
Via: http/1.1 traffic_server (ApacheTrafficServer/10.2.0)


<svg width="1287" height="380" viewBox="0 0 1287 380" fill="none" xmlns="http://www.w3.org/2000/svg">
...

Do you see the problem?

Response contain "Content-Encoding: gzip" while the body is a plain svg. I assume that "Content-Length: 32578" is wrong and dedicated to gzip'ed variant of a content.

I have another case but it is not so funny and more frustrating:

sample 2

GET /wp-content/uploads/2017/05/IIT-Architecture-768x512.jpg.webp HTTP/1.1
Host: www.epoxyworks.com
Pragma: no-cache
sec-ch-ua-platform: "Windows"
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36
sec-ch-ua: "Google Chrome";v="149", "Chromium";v="149", "Not)A;Brand";v="24"
sec-ch-ua-mobile: ?0
Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: image
Referer: https://www.epoxyworks.com/category/home-garden/architecture/
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9,ru;q=0.8,pl;q=0.7
Cookie: cookieyes-consent=consentid:MmxnZDMwaG5tdWJ4SVpmQVRPZjhGRzhoMXZMMUhwRWM,consent:no,action:,necessary:yes,functional:no,analytics:no,performance:no,advertisement:no; _ga=GA1.1.2040193634.1778861584; _clck=v0s5jz%5E2%5Eg62%5E0%5E2326; _ga_8ZFKGQ7DBS=GS2.1.s1781793747$o3$g1$t1781793790$j17$l0$h0; swpext86386=5fa1b333ba5fb75515384e19da19d4fc; PHPSESSID=bf7a9f0cb68ab7e1ab1dfaaf7d5b09f5
Via: 1.1 z101.orgs.link (squid)
Cache-Control: no-cache
Connection: close
X-Forwarded-Proto: https


HTTP/1.1 200 OK
Date: Wed, 01 Jul 2026 14:28:56 GMT
Content-Type: image/webp
Content-Length: 46806
Last-Modified: Fri, 06 Jun 2025 19:44:44 GMT
Accept-Ranges: bytes
Cache-Control: public, must-revalidate, proxy-revalidate, immutable, max-age=31536000, stale-while-revalidate=86400, stale-if-error=604800
Expires: Thu, 29 Oct 2026 14:28:56 GMT
Vary: Accept-Encoding
Server: ATS/10.2.0
nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
cf-cache-status: MISS
speculation-rules: "/cdn-cgi/speculation"
report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=7K0aITjElycFvh2agtTOgfruUFfXGeP9L4Bm6KZQjbjV580%2BrLaMUTcLqo9GbDU8nZw3N6JxCdO3rAAZCQUaQlradCvLn75q0BM5QlTz1tiIXvaGf6aRvMfB%2BjRu%2FreMN3QnAZ0%3D"}]}
cf-ray: a146199ac9588291-ARN
alt-svc: h3=":443"; ma=86400
Age: 0
Connection: Keep-Alive
Via: https/2 traffic_server (ApacheTrafficServer/10.2.0)

RIFF....WEBPVP8 .
...

Wireshark say that all this data is about 34Kb while "Content-Length: 46806" say it must be larger.
This case appear in access log as
1782916131.021 180 fd05:562e:5a23::212:2e01 ERR_READ_ERROR/200 952 GET https://www.epoxyworks.com/wp-content/uploads/2017/05/IIT-Architecture-768x512.jpg.webp - DIRECT/www.epoxyworks.com image/webp

The ATS<->origin interaction contain no RST or something abnormal. Morever there is no TCP "dialog" for less than 44KB. So I don't now where "952" from.

sample 3

The same as sample 2. Bytes on wire is less than content-length.

GET /gtm.js?id=GTM-KLQLNPW8 HTTP/1.1
Host: www.googletagmanager.com
Pragma: no-cache
sec-ch-ua-platform: "Windows"
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36
sec-ch-ua: "Google Chrome";v="149", "Chromium";v="149", "Not)A;Brand";v="24"
sec-ch-ua-mobile: ?0
Accept: */*
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: script
Sec-Fetch-Storage-Access: active
Referer: https://www.epoxyworks.com/
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9,ru;q=0.8,pl;q=0.7
Via: 1.1 z101.orgs.link (squid)
Cache-Control: no-cache
Connection: close
X-Forwarded-Proto: https

---
HTTP/1.1 200 OK
Content-Type: application/javascript; charset=UTF-8
access-control-allow-origin: *
access-control-allow-credentials: true
access-control-allow-headers: Cache-Control
Content-Encoding: gzip
Vary: Accept-Encoding
Date: Wed, 01 Jul 2026 15:52:41 GMT
Expires: Wed, 01 Jul 2026 15:52:41 GMT
Cache-Control: private, max-age=900
Last-Modified: Wed, 01 Jul 2026 15:00:00 GMT
Strict-Transport-Security: max-age=31536000; includeSubDomains
cross-origin-resource-policy: cross-origin
Server: ATS/10.2.0
Content-Length: 127909
x-xss-protection: 0
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Age: 0
Connection: Keep-Alive
Via: https/2 traffic_server (ApacheTrafficServer/10.2.0)

............k[...(.._....K.....F..**...qC...4!
	&....o?OUw'....s.}.~....$}........,..(;..%p.^........N.*-.*(.~_..P	hH.Gj....|w.Y..{y.ua....M"...........Uk......4.].Sk.../(

ERR_READ_ERROR/200 in access log.

conclusion

May be the problem is in update object in cache logic. It seems that objects in cache replaced by the object from response from origin regardless of it's Content-Encoding header. When the fresh object has arrived it replace previous object in a cache without updating Content-Length and Content-Encoding header.

Another thread is in rather complex Vary header processing. In case of Vary in response and when an object from this response is cached, further cache may hit only if apropriate headers (typical Accept-Encoding and User-agent) in request is strictly matching corresponding headers of object in a cache. There may be (?) various form of object in cache that match a criteria. There may be gzip'ed svg and plain svg for the same request. So the issue may be related to building reply which contain headers from one object and body from another.

May be I'm totally wrong.

P.S. ATS 10.2.X, FreeBSD 14-STABLE


    • a few weeks ago when I began investigation I don't find errors in access log related to problematic URLs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions