Skip to content

fix: handle 403 Forbidden in download_url_to_file#1283

Open
GoldenSection0618 wants to merge 1 commit intoMouseLand:mainfrom
GoldenSection0618:enhance/download-url-robustness
Open

fix: handle 403 Forbidden in download_url_to_file#1283
GoldenSection0618 wants to merge 1 commit intoMouseLand:mainfrom
GoldenSection0618:enhance/download-url-robustness

Conversation

@GoldenSection0618
Copy link

What changed?

  1. Extended HTTP headers to fully simulate a browser request (added Accept, Accept-Language, etc.), solving 403 Forbidden caused by incomplete headers.
  2. Added retry logic for HTTPError: If the server rejects Accept-Encoding, automatically retry without that header.
  3. Ensured the target directory exists before saving files, avoiding path-related errors.

Why?

I encountered 403 Forbidden errors when downloading files—some servers block requests with incomplete headers. These changes make the download function more robust and compatible, fixing the access issue while improving error handling.

Tested on

  • URLs that previously returned 403 (now download successfully).
  • Paths with non-existent directories (auto-created and saved correctly).

@GoldenSection0618
Copy link
Author

The 403s were happening because the old request lacked browser-grade headers (Accept, Accept-Language, etc.). Servers often block “script” traffic. I added Chrome-like headers and a fallback that strips gzip when the server chokes on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant