Help with dealing with Flickr timeouts

[jeanmichel-nwsb]

Bonjour @alexis-mignon , Hello @beaufour

First let me thank you a lot for this library and to restart the work on it.

I've been using it actively since a little more than a year to publish batches of photos (~200-250 per batch) and I have been fighting with Flickr limitations.
It may not look very much but because of our current infrastructure, I don't get a list of all photos to upload. We have an event driven system (AWS queue + AWS lambda) and I get 1 event per photo. So for each photo we have to redo it all again:

• login
• gather the full list of Photoset in the account,
• scan the list until I find the correct one by title
• If it doesn't exist, create it
• upload the photo using flickr_api.upload(...)
  And we can run up to 5 such flow simultaneously

Last week-end, we had to upload 140 photos.
It went well for 120 photos, but for 20 of them it failed
I already had implemented a retry mecansim around flickr_api.upload(...), with 1 retry after 30 seconds but for the 20 that failed, they all failed both the original attempt and the retry with

ReadTimeoutError("HTTPSConnectionPool(host='api.flickr.com', port=443): Read timed out. (read timeout=10)")

I was about to increase the number of retries and the delay between retries when I noticed the recent improvements.

Could you please kindly advise if and how the newly added methods could help me tackle this problem ?

refering to:

Implement opt-in rate limiting that throttles requests to respect
Flickr's documented 3600 requests/hour limit. This is proactive
throttling (sleeping before requests) as opposed to the existing
reactive retry handling (retrying after 429 responses).

New public functions:
- set_rate_limit(requests_per_hour) - Enable/disable rate limiting
- get_rate_limit() - Get current configuration
- get_rate_limit_status() - Get detailed status including interval

Internal function _maybe_wait_for_rate_limit() is called at the start
of _make_request_with_retry() to sleep if necessary before making
each request.

Thanks in advance

[beaufour]

Hi @jeanmichel-nwsb, thanks for the detailed description of your use case!

Good timing — we just merged #159 which directly addresses your ReadTimeoutError problem. The library now automatically retries on timeouts, connection errors, and server errors (5xx) for both API calls and uploads, with exponential backoff.

Here's what I'd recommend for your setup with ~200-250 photos and 5 concurrent flows:

import flickr_api

# Increase timeout - the default 10s is often too short for uploads
flickr_api.set_timeout(30)

# More retries with longer backoff for batch uploads
# Default is 3 retries, 1s base delay, 60s max delay
flickr_api.set_retry_config(max_retries=5, base_delay=2.0, max_delay=120.0)

# Proactive rate limiting to avoid hitting Flickr's 3600 req/hr limit
# With 5 concurrent flows, this helps prevent 429 errors
flickr_api.set_rate_limit(requests_per_hour=3600)

What each setting does:

• set_timeout(30) — Gives uploads 30 seconds instead of 10 before timing out. This alone might fix most of your failures.
• set_retry_config(max_retries=5, base_delay=2.0, max_delay=120.0) — If a request does time out, it retries up to 5 times with exponential backoff (2s, 4s, 8s, 16s, 32s). This replaces your manual retry logic — you can remove it and let the library handle retries.
• set_rate_limit(requests_per_hour=3600) — Proactively throttles requests to stay within Flickr's documented limit. With 5 concurrent lambdas each doing login + list photosets + upload, the requests add up quickly.

One note on concurrency: The rate limiting is per-process, so each of your 5 Lambda invocations tracks its own rate independently. If you find you're still hitting 429 errors, you could lower the limit per Lambda (e.g., set_rate_limit(requests_per_hour=720) to split the budget 5 ways).

These features are all available on the master branch now (#156, #157, #159). They'll be included in the next PyPI release.

[jeanmichel-nwsb]

Hi @beaufour

This is AWESOME

Thanks for the improvement and the detailled guide
I will test that this week

[jeanmichel-nwsb]

Sorry for pinging Alexis instead of Allan