Conversation
- Switch gunicorn to --worker-class gthread --workers 2 --threads 8 so I/O-bound requests run concurrently instead of head-of-line blocking behind a single sync worker (was blocking the whole site on 7.7s funnel-aggregate recomputes) - Remove --log-level debug in prod (noisy, slow) - Add ENV PATH="/app/venv/bin:$PATH" to Dockerfile so git-fame CLI is found without calling it by absolute path (P2-6) - Add lock=threading.Lock() to every @cached(TTLCache(...)) decorator across services/, api/, and common/ — required for gthread correctness (concurrent threads hitting an unguarded cachetools dict can corrupt it) - Add docs/perf-reliability-plan-2026-06.md (execution plan) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t 500s
Fixes 6 crash patterns surfaced in Sentry (133 + 21 + 11 + 10 + 6 + 3 events):
- feedback_service.save_feedback: guard slack_user is None → return None;
view (messages_views) now returns 404 instead of crashing on vars(None)
- giveaway_service: guard slack_user is None in save_giveaway and
get_user_giveaway; guard get_user_by_id_old None in get_all_giveaways
- messages_service.save_helping_status_old: guard slack_user is None (in-place fix)
- messages_service.get_single_problem_statement_old: guard doc_to_json None return
- users_views: save_volunteering_time returns 404 (not None) when service
returns None; get_volunteering_time guards tuple unpack from None
- users_service.get_volunteering_time: return None (not implicit) on missing user
- users_service.get_profile_metadata: guard db_id is None after save_user()
with warning log (was AttributeError → 500)
- hackathons_service.update_*_by_event_id: use doc.to_dict().get('slack_user_id')
instead of doc.get() which raises KeyError; gate all Slack calls on non-None
slack_user_id so the update still succeeds and returns 200
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
perf(P0-1/P2-6): gthread workers, TTLCache thread-safety, venv PATH fix
- leaderboard_service: Redis 5-min cache on get_github_leaderboard; hackathon fetched once and passed to all helpers; event_id alias normalization (season-YYYY → YYYY_season); fixed broken org existence check (was testing DocumentReference truthiness, now uses .get().exists); downgraded spurious logger.error → logger.warning/info for missing org/hackathon/achievement; collect_mentor_panel_opportunities cached 5 min (TTLCache+lock), short- circuits before any Firestore reads when event is outside its live window; mentor panel opportunities now include open flags + stale-touch teams - hackathons_service get_hackathon_funnel_aggregate: replaced sequential N reads with db.get_all() batch for funnel/summary subcollections + single global volunteers Counter query; Redis 6-hour cache via get_cached/set_cached; clear_cache() updated to use delete_cached() (no longer @cached) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dening, log sweep
P1-3 — PropelAuth OAuth lookup cache (users_service.py):
- Two-tier in-process cache: 10-min TTL for positive results (512 entries),
5-min TTL for negative/None results — eliminates repeated HTTP to PropelAuth
for the same user across a request storm; intentionally NOT Redis (payload
contains OAuth access tokens)
- 5 call sites downgraded error → warning ("Could not get OAuth user from
PropelAuth" is an expected client-state, not an internal error)
P2-1 — NPO N+1 fix (nonprofits_service.py):
- get_npos_by_hackathon_id: replaced per-ref sequential .get() loop with a
single db.get_all(npo_refs) batch read; falls back to serial loop on error
P2-2 — Slack hardening:
- get_slack_user_by_email: added 24h/1h two-tier cache (hit/miss) + @sleep_and_retry
+ @limits(20/min); users.lookupByEmail is tightly rate-limited (28 Sentry failures)
- send_slack: replaced assert + re-raise on SlackApiError with warning log — a
Slack failure must not raise out of business logic
- is_channel_id: downgraded conversations.info failure from error → debug
(falling back to name lookup is the designed path)
- volunteers_service send_slack_volunteer_notification: truncate availableDays /
skills / experience fields to ≤500 chars; split section block if joined text
>2900 chars (Slack cap 3000, invalid_blocks Sentry issue)
P2-7 — log sweep (expected-absence conditions):
- teams_service: "Could not resolve Slack/database user" error → warning (×4)
- users_service: "User not found" error → warning (×3)
- judging_service: "Judge not found" (404 path) error → info
- messages_service: "Document not found in users" (404 path) error → info
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix(P0-2): guard None returns from PropelAuth/Slack lookups — eliminate 500s
…-hardening [Perf] P1-3, P2-1, P2-2, P2-7: PropelAuth cache, NPO batch read, Slack hardening, log sweep
- Keep `from collections import Counter` (used by funnel aggregate batch) - Keep Redis TTL constants over TTLCache decorator (function uses get_cached/set_cached) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…oard [Perf] P1-2+P1-3: leaderboard Redis cache + funnel aggregate batch reads
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.