Skip to content

fix(gumroad, girlslife): force Accept-Encoding: identity#338

Merged
kaifcodec merged 1 commit into
kaifcodec:mainfrom
sturq:fix/missing-content-decoding-deps
May 17, 2026
Merged

fix(gumroad, girlslife): force Accept-Encoding: identity#338
kaifcodec merged 1 commit into
kaifcodec:mainfrom
sturq:fix/missing-content-decoding-deps

Conversation

@sturq
Copy link
Copy Markdown
Contributor

@sturq sturq commented May 16, 2026

Both modules advertised Accept-Encoding: gzip, deflate, br, zstd, but httpx ships only gzip/deflate decoders by default. When servers honored the header and replied with brotli or zstd, the response body came back as raw compressed bytes, breaking the nonce regex in girlslife and response.json() in gumroad:

  • Girlslife: Failed to parse response
  • Gumroad: unexpected exception:

Switching both to Accept-Encoding: identity keeps the response uncompressed, no extra deps required.

Verified locally with brotli/zstandard NOT installed (httpx decoders limited to identity, gzip, deflate): both modules return Registered / Not Registered cleanly.

Per your comment on the previous revision, scoped this to only the two confirmed-broken modules. Same pattern exists in a dozen other places (orchestrator, both github.py, foxnews, ama, dribbble, linktree, producthunt, launchpad, battlenet, snapchat, threads), but those are currently working since their upstream servers are falling back to gzip. Happy to follow up with a separate PR if any of those start failing.

@kaifcodec
Copy link
Copy Markdown
Owner

@sturq Thank you for the heads up, you’re correct about this.

Some modules can indeed break on different environments because of the advertised Accept-Encoding values. However, instead of introducing new dependencies for Brotli and Zstandard support, we can likely fix most of these cases by forcing Accept-Encoding: identity on the affected modules.

You can try this yourself: if a module is failing because of compressed responses, replace the current Accept-Encoding value with just identity and test it again. If it resolves the issue, feel free to open a PR for those specific modules.

In general, using identity is the safest option. Some modules were left with default/browser-like encodings simply because they worked at the time, but it’s possible that certain servers have since changed their behavior and are now returning br or zstd responses, which would require explicitly forcing identity.

Both modules advertised "gzip, deflate, br, zstd" in Accept-Encoding,
but httpx ships only gzip/deflate decoders out of the box. When the
upstream server honored the header and replied with brotli or zstd,
response bodies came back as raw compressed bytes, so the nonce regex
in girlslife and response.json() in gumroad fell over with parse and
exception errors.

Switching to identity keeps the response uncompressed, no extra deps
required.

Confirmed locally with brotli/zstandard NOT installed (httpx decoders
limited to identity, gzip, deflate): both modules return Registered /
Not Registered cleanly.
@sturq sturq force-pushed the fix/missing-content-decoding-deps branch from 22c5e80 to 7ac981c Compare May 17, 2026 02:32
@sturq sturq changed the title Declare brotli + zstandard so httpx can decode br/zstd response bodies fix(gumroad, girlslife): force Accept-Encoding: identity May 17, 2026
@sturq
Copy link
Copy Markdown
Contributor Author

sturq commented May 17, 2026

Makes sense, agree on avoiding new deps. I've reset the branch and force-pushed a new commit that just switches gumroad and girlslife to Accept-Encoding: identity instead. Both confirmed working locally with brotli/zstandard not installed (httpx falls back to identity/gzip/deflate only) and the modules return real results. Scoped this PR to those two since they're the ones throwing visible errors. Left the rest alone for now since their servers seem to be falling back to gzip, can follow up if any of them start regressing.

Copy link
Copy Markdown
Owner

@kaifcodec kaifcodec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sturq This is the correct approach, It keeps us safe from future changes.

@kaifcodec kaifcodec merged commit c1e0f2d into kaifcodec:main May 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants