Skip to content

perf: make Hacktoberfest prep tracker fast + verbose (async fan-out, progress) - #15226

Merged
cclauss merged 1 commit into
TheAlgorithms:masterfrom
priya-sundaram-dev:hacktober-prep-async
Sep 8, 2026
Merged

perf: make Hacktoberfest prep tracker fast + verbose (async fan-out, progress)#15226
cclauss merged 1 commit into
TheAlgorithms:masterfrom
priya-sundaram-dev:hacktober-prep-async

Conversation

@priya-sundaram-dev

Copy link
Copy Markdown
Contributor

Follow-up to #15225, addressing @cclauss's feedback on #15081: the daily hacktoberfest_prep run took ~16 minutes with no output.

What was slow

scripts/hacktoberfest_prep_update.py looks up the changed files of every open awaiting reviews PR to rank the busiest directories — up to a few hundred GET /pulls/{n}/files calls, previously issued one at a time. That serial chain, not the total request count, is what ate the 16 minutes.

Changes

  • Async fan-out. Those lookups are independent, so they now run concurrently through a single httpx2.AsyncClient (per the async guide) bounded by a small asyncio.Semaphore (CONCURRENCY = 8) to stay well under GitHub's secondary rate limits. Tracked-row PR-state checks and the three issue/PR counts are gathered concurrently too. Runtime drops from ~16 min of serial round-trips to well under a minute.
  • Progress reporting. Each phase logs to stderr (flushed, so it streams live in the Actions log): row-resolution, count collection, and a ...scanned N/total tick every 25 PRs. The final line now reports elapsed seconds.
  • File I/O stays in the sync main(); only the network work is async (keeps ASYNC230 happy).

Rate-limit/5xx retry behaviour is unchanged, just moved into the async helper. Stdlib asyncio + the already-installed httpx2, so no workflow change is needed.

On the two links you shared

  • httpx2 async — yes, this is the win, applied above.
  • pydantic-ai eval retry strategies — that one is for pydantic-ai evaluation runs (retrying flaky LLM eval cases), so it doesn't map onto a plain GitHub REST poller. The reliability lever here is the concurrency cap + the existing X-RateLimit-Reset/5xx backoff, which I kept.

Testing

Ran the whole flow against a stubbed client (checkbox ticking for merged/closed vs still-open rows, stats block, top-3 directory ranking with correct singular/plural, progress lines, exit code). ruff check/ruff format clean.

The daily prep job spent ~16 minutes because it looked up the changed
files of every open 'awaiting reviews' PR one request at a time. Those
lookups are independent, so fire them concurrently through a single
httpx2.AsyncClient bounded by a small semaphore, and likewise resolve the
tracked-row PR states and the issue/PR counts concurrently. Report
progress to stderr (flushed) so a human watching the Actions log can see
the job is alive. Runtime drops from ~16 min of serial round-trips to
well under a minute.
@algorithms-keeper algorithms-keeper Bot added the awaiting reviews This PR is ready to be reviewed label Sep 8, 2026
@algorithms-keeper

Copy link
Copy Markdown

Closing this pull request as invalid

@priya-sundaram-dev, this pull request is being closed as none of the checkboxes have been marked. It is important that you go through the checklist and mark the ones relevant to this pull request. Please read the Contributing guidelines.

If you're facing any problem on how to mark a checkbox, please read the following instructions:

  • Read a point one at a time and think if it is relevant to the pull request or not.
  • If it is, then mark it by putting a x between the square bracket like so: [x]

NOTE: Only [x] is supported so if you have put any other letter or symbol between the brackets, that will be marked as invalid. If that is the case then please open a new pull request with the appropriate changes.

@algorithms-keeper algorithms-keeper Bot closed this Sep 8, 2026
@algorithms-keeper algorithms-keeper Bot removed the awaiting reviews This PR is ready to be reviewed label Sep 8, 2026
@cclauss cclauss reopened this Sep 8, 2026
@algorithms-keeper algorithms-keeper Bot added the enhancement This PR modified some existing files label Sep 8, 2026
@cclauss
cclauss enabled auto-merge (squash) September 8, 2026 07:50

@cclauss cclauss left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RSLGTM

@cclauss
cclauss merged commit fb906d8 into TheAlgorithms:master Sep 8, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement This PR modified some existing files invalid

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants