Skip to content

feat(provider): analytics day sync and the Analytics page - #384

Open
jorgecuesta wants to merge 1 commit into
stagingfrom
feat/fleet-day-sync
Open

jorgecuesta wants to merge 1 commit into
stagingfrom
feat/fleet-day-sync

Conversation

@jorgecuesta

@jorgecuesta jorgecuesta commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Credits

The Analytics page reimplements, inside the provider app, the analysis that @mykyal built: the metrics, the formulas (actual earnings Y, equal share B, F = Y / B, presence, the 95% bootstrap range, the breakdowns, the session view and the "What if your fleet changed?" selection model) and the page structure. Credit for the analysis, the metrics and the formulas goes to @mykyal. This PR ports that work to Igniter's data and scope; where it differs, the list under "Differences from @mykyal's fleet analysis" below gives what changed and why.

Summary

  • Admin > Analytics (owner-only). A new provider page shows what the provider's suppliers earned in the sessions they were selected for, against an equal share of each session's pool. It follows the structure of @mykyal's fleet analysis.
    • Summary: earnings vs equal share, with the 95% range; how often you were selected; earnings through the period; and per-service difference bars.
    • Breakdown: compare by service, supplier, owner, week, application, selected count or session value. A table under each chart has CSV export.
    • Sessions: a list with a phase filter, and a session drawer.
    • The page reads only the provider DB.
  • "What if your fleet changed?" A Summary card under "One provider. Different services.", ported from @mykyal's tool (Ramiro asked for it). It answers: with a different number of eligible suppliers, how likely is a session to select at least k of yours, and how many of yours does it select on average?
    • Controls: "Selection model for" a service, since selection is calculated separately per service. "Select at least" 1, 2, 3 or 5 of yours, or a custom count. "Your eligible suppliers" as a number input and a slider (0 to the largest of the count for 99%, twice the recorded count and the scenario, rounded up to 25). "Find a fleet size for" 90/95/99% or a custom target with "Find count". A chart of the chance against your eligible count. "Model assumptions" and Reset.
    • Model (whatif.ts#L59-L117): a session draws min(cap, N) of the N eligible suppliers uniformly without replacement, so the number of yours is Hypergeometric(N, n, K). The expected selected is K × n / N. "Find count" is a doubling then bisection search on the monotone tail, capped at 100,000. It is a port of @mykyal's math with the same operations in the same order, and the unit tests assert equality with outputs of their own code.
    • Reference: per service, the latest closed day of the period, using its latest session start. N is that session's candidates, K is the session param num_suppliers_per_session at that height, and n is how many of the day's verified suppliers were eligible there.
    • Own eligible is a new count-only indexer read (fetchSelectionCounts): supplierServiceConfigs totalCount over our ids (at most 200 per request, at most 20 aliases), with pocketdex's candidate rule (live at the start or closed by the start block, activated at or below the start, or not activated yet). It returns counts, never a supplier id.
    • Sync: at the end of each AnalyticsSync, syncAnalyticsSelectionReferences writes the pending closed days, newest first, best-effort (activities/analytics.ts#L370-L422). Each row carries the day's rollup version, so a supplier the backfill adds later, or a key leaving the verified states, makes the day count again. A day that fails is skipped for the rest of the run. A service counted above its candidates is not stored.
  • Day sync (provider-workflows).
    • AnalyticsSync is a new schedule, every 10 min (SCHEDULE_ANALYTICS_SYNC_INTERVAL), forward only: today and every day still open.
    • AnalyticsBackfill is manual, started from the page's "Fill missing days" button. It fills the days of the retention window that were never synced.
    • Both run one AnalyticsSyncDay per day under a single workflow id, so the schedule and the backfill never sync a day at once. Each day fans out to AnalyticsSyncBatch children that read the pocketdex indexer's getSupplierSessionFunnelJson and getSupplierSessionListJson (the _json twins are generated in settlement/functions.ts).
    • After its batches, a day rebuilds its breakdown rollups: analytics_owner_day and analytics_session.own_*, versioned per day in analytics_day_rollup. A day that is syncing, or stale, is read from the member rows instead.
  • Data scope. Only the provider's verified suppliers are read and stored. Every indexer call passes at most 200 supplier ids and never a domain, owner or operator. Any returned row of another supplier is dropped before writing.
  • Shared code.
    • postGraphql moves to @igniter/graphql, now built for ./client and ./analytics.
    • Every CSV export of every app now starts with a UTF-8 byte order mark, so spreadsheet apps read "÷", "–" and other non-ASCII text correctly (packages/ui/src/lib/csv.ts#L51-L61). No app in the repo re-reads its own exports.

How the sync keeps the stored rows exact

  • Deletes keep the answer, never a time cut. After writing a day's pages, a batch deletes its member rows that are not in the indexer's answer, by their (session, supplier) keys, and the sessions left with no member (lib/dal/analytics.ts#L142-L161). The funnel rows work the same way, by (supplier, service) (L83-L117). An earlier version deleted rows older than the run's start. Local testing lost a page of member rows that way: the test machine's clock stepped back by about 0.8 s, so a fresh page looked older than the run.
  • A cancelled attempt stops writing. Each write first checks the activity's cancellation signal, so an attempt that Temporal already timed out does not write after its retry (activities/analytics.ts#L99-L108).
  • A day closes only when its rows are complete. A (day, supplier) also needs its stored member rows to equal its funnel seats (seats + active). A mismatch keeps it open, logs both counts, and the next sync fetches it again (lib/analytics.ts#L45-L85, activities/analytics.ts#L279-L333).
  • The close query's plan does not depend on statistics. It runs while its day is still being written, before any ANALYZE has seen it. It counts the batch's member rows with a primary-key lookup of each session (lib/dal/analytics.ts#L163-L210). The earlier shape took 19 s instead of 0.12 s on a 245k-member day, and twelve of those held the worker's whole DB pool.
  • The sync's DB work is capped at 6 connections in a worker. A day runs 12 batches at a time, and each holds at most one connection. The cap leaves 4 of the default 10 to the rest of the worker; a batch waits in order instead of failing on the pool's connection timeout (activities/analytics.ts#L54-L62, lib/semaphore.ts).

Differences from @mykyal's fleet analysis

Compared with @mykyal's implementation as of 13 Oct 2026. Their code is cited as file:line without a link. Igniter links are pinned to this PR's head (4a6303357faf9a6f842a8b285ee283ef1e42c600), and pocketdex links to 5ec4b0a on main.

  1. Data source and how it is fetched.

    • Theirs: an offline collector builds one saved month (6 Sep – 5 Oct 2026) at a fixed indexer cutoff (height 955836); the page reads that snapshot and makes no API calls (docs/data-collection.md:L3–L21). Payments come from eventClaimSettleds and modToAcctTransfers on data.pocket.network (collector/settlement_collect.py:L41–L52). Each session's roster is rebuilt from archive REST at its end height (collector/settlement_collect.py:L250–L283).
    • Igniter: provider-workflows reads the live indexer configured as indexerApiUrl, one day and at most 200 suppliers per call, through getSupplierSessionFunnelJson (per day, service and supplier) and getSupplierSessionListJson (one row per session, 1,000 per page) (packages/graphql/src/analytics.ts#L3-L43, activities/analytics.ts#L134-L277). pocketdex builds the rosters, V, Y and B when it indexes (the session_rosters table and the session_supplier_daily rollup).
    • Why: the page has to stay current for one provider without a manual collection run. The indexer already holds every roster, so nothing is sampled or rebuilt from archive blocks.
    • Effect: the e2e matched on the same month and services (Testing). Igniter re-reads a day until it is final (point 8), so a past day can still change if the indexer fills a gap.
  2. Session day axis: start, not end.

    • Theirs: a settled session is dated by the block time of its end height, endedAt (src/settlements.js:L45–L53, L130–L137).
    • Igniter: a session belongs to the UTC day of its start, as pocketdex buckets it (lib/analytics.ts#L3-L6, funnel doc).
    • Why: the indexer's daily rollups, its range filters and the list keyset are all by session start, and a session's start is known while it is still active.
    • Effect: sessions that start before midnight and end after it fall on different days. Over the e2e month, actual and equal-share totals were within 0.05%. This is the only known source of that gap, but it was not isolated by a separate run.
  3. Scope: the provider's verified keys only.

    • Theirs: a scope is chosen by domain, hostname, owner, fleet, supplier or the whole network. Membership follows each session's historical owner and service endpoints (src/engine.js:L85–L94, src/settlements.js:L78–L87).
    • Igniter: the scope is always the provider's own suppliers: the keys in a staked-or-later state (schema/analytics.ts#L22-L34). There is no domain selector. The queries accept only supplier ids, an empty list is refused (graphql/src/analytics.ts#L261-L267), and any returned row of another supplier is dropped (activities/analytics.ts#L147-L150, L226-L229). Every page query filters on those keys (dal/analytics.ts#L46-L52).
    • Why: holding the key is the proof of control. A provider should see only its own suppliers, and a domain can be claimed by anyone who points an endpoint at it.
    • Effect: for a provider whose keys are the same set as a domain's suppliers, the numbers match (the e2e). A domain shared with other operators gives different numbers.
  4. How V, K, Y and B are computed, and the per-seat columns.

    • Theirs: per session, in POKT floats: Y = the payouts of the in-scope members, B = pool × X / K, where X = in-scope members and K = selected count (src/settlements.js:L129–L136; sampled frames src/engine.js:L189–L197).
    • Igniter: pocketdex gives B per seat as V / K, rounded to upokt (list; the daily rollup sums it, and the funnel rounds the group sum). Igniter sums Y and B per supplier row as BigInt upokt, and divides only for F, rounding half away from zero before any float (math.ts#L1-L38, L174-L176). The CSV's ratio and % columns are exact decimal strings from BigInt, rounded half up (csv.ts#L19-L30). The funnel's sessions_with_selected, pool_upokt and roster_seats are per supplier: two of your suppliers in one session each carry that session's full V and K. So they are never summed across suppliers. Distinct sessions, V and K come from analytics_session rows with at least one member in scope (schema/analytics.ts#L36-L40, dal/analytics.ts#L9-L16, L159-L186).
    • Why: integer upokt end to end keeps the totals exact. Reading V and K once per session is what makes them correct when several of your suppliers share a session.
    • Effect: the rounding is under 1 upokt per seat. On the e2e data it was not visible at the precision shown.
  5. Presence: closed days only, clamped, and a different denominator.

    • Theirs: presence = sampled sessions with at least one in-scope member ÷ sampled sessions with a complete roster. The sample is four observation times a day per application and service (src/engine.js:L261–L280, docs/data-collection.md:L13). An expected presence from a hypergeometric model is shown beside it.
    • Igniter: presence = your distinct sessions ÷ the network's settled sessions of the same services (pocketdex network_sessions), over closed days only, and capped at 100% (math.ts#L139-L173, L217-L218, actions/Analytics.ts#L155-L171). No expected presence is computed.
    • Why: every session is in the indexer, so no sample is needed. On an open day, the session list is newer than the funnel's network_sessions, so your sessions can exceed it. A closed day above 100% is a data error. It is logged, listed on the page and shown capped.
    • Effect: the e2e gave 100.0% on both sides. Days still open do not count toward presence until they close.
  6. The 95% range (bootstrap).

    • Theirs: 2,000 resamples of days with replacement, from mulberry32 seeded with a hash of the query, so each scope and filter has its own seed. The quantiles are interpolated. Days with no equal share stay in the draw. No range is shown with fewer than 7 informative days or 24 sessions with an equal share, or when under 95% of draws are defined (src/math.js:L104–L124, src/engine.js:L304, L476–L490). Every breakdown row gets its own range.
    • Igniter: the same gate: a range only with at least 7 days with an equal share (B > 0) and 24 own sessions with one (V > 0) in the period (math.ts#L76-L85, L204-L214). Below it, the page shows "Limited history" with both counts instead of a range. The range always carries a line that it reflects variation across the recorded days in the period, not a forecast of future earnings (SummaryTab.tsx#L190-L206). The resampling: 2,000 resamples, the same mulberry32, one fixed seed (20261025). Only days with B > 0 are drawn, so every draw is defined. The quantile is the nearest rank (math.ts#L64-L119). The range is shown for the Summary only, not per breakdown row.
    • Why: a fixed seed gives the same range on every render and every server. Without the gate, two similar days gave a narrow range that looked more certain than the data allows.
    • Effect: e2e upper bound −33.2% here vs −33.1% there; the lower bound matched. The seed and quantile method are the likely cause; this was not measured separately. A short or thin period now shows "Limited history" where it used to show a range.
  7. Weeks.

    • Theirs: weeks are 7-day blocks counted from the snapshot's first day, for both the trend chart and the Week breakdown (src/engine.js:L153–L163, src/settlements.js:L93).
    • Igniter: weeks are 7-day blocks counted from the period's first day, and the last block can be shorter. That holds for both the earnings chart (coverage.ts#L100-L127, SummaryTab.tsx#L299-L308) and the Week breakdown, which groups in SQL (dal/analytics.ts#L240-L244, L327-L342). The chart shows single days for periods up to 14 days. The labels and the CSV file name use the period the server answered with, not the one in the page header.
    • Why: the period is any range ending today, so it is its own snapshot. Counting from its first day matches their weeks on the same month.
    • Effect: for the same 30 days, both the chart and the Week breakdown give the same five blocks as theirs (6–12 Sep … 4–5 Oct).
  8. When a day is final: the close rule and money_missing.

    • Theirs: the snapshot is frozen at one cutoff. A session either has verified payouts, whose sum must equal the pool, or is "unknown". Unknown sessions add to neither Y nor B but stay in the coverage count (src/settlements.js:L61–L66).

    • Igniter: a (day, supplier) closes only when all of these hold (lib/analytics.ts#L11-L12, L45-L85):

      • the indexer's coverage spans the whole day plus 2 h;
      • the range has no settlement gaps;
      • no row came back for a supplier outside the batch;
      • none of its seats is still active;
      • every one of its sessions is settled and not money_missing;
      • its stored member rows equal its funnel seats.

      A closed day reopens if the indexer's coverage moves back (L87-L100). Until a money_missing session's money is filled, pocketdex counts its V as 0, so it adds 0 to Y and B, and its per-member B is null. It is fetched again until it settles.

    • Why: live data keeps changing. A day has to be marked final before presence uses it (point 5), and it has to be fetched again until it is final.

    • Effect: Y, B and F include open days, and the page shows how many days are still open. Presence uses closed days only.

  9. Retention and periods.

    • Theirs: one fixed month; views of the last 7, 14 or 30 days of it (src/engine.js:L106–L111).
    • Igniter: a rolling window of analytics_day_retention_days (1–365, default 90), ending today (UTC). Periods are 7, 30 and 90 days, capped by the retention, or a custom range (coverage.ts#L11-L53). Older days are deleted on every AnalyticsSync run (activities/analytics.ts#L424-L431, lib/dal/analytics.ts#L392-L410).
    • Why: the member table holds one row per own seat (about 11.6M rows for 51 days in the e2e), so storage has to be bounded.
    • Effect: none on the numbers of a day that is kept.
  10. The session list and its phases.

    • Theirs: no list. Sessions are inspected as points in Diagnostics > Session earnings, and each opens a dialog. Every session in the snapshot is settled, and its payout status is verified or unknown (src/app.js:L459, L674–L684).
    • Igniter: a paged list, newest first. Each row has its pocketdex phase (active … settled). The filter has three groups that make up all sessions: settled (with its money), in progress (any phase before settled), and money missing (dal/analytics.ts#L417-L462). The drawer lists only your members of the session (L466-L497).
    • Why: with live data, an operator needs to see which sessions are not final yet and why.
    • Effect: none on the totals.
  11. Owner attribution.

    • Theirs: a supplier counts under the owner it had at that session, taken from the historical roster (src/engine.js:L3–L5, src/settlements.js:L29–L35).
    • Igniter: the breakdown is labeled "Current owner" in the selector, the table and the CSV. It uses the key's current owner, which is its stake owner, or the owner it was delivered to when that is not known yet, and the page says that history is grouped by each supplier's current owner, not the owner it had at the time (BreakdownTab.tsx#L26, schema/analytics.ts#L139-L140, dal/analytics.ts#L314-L326).
    • Why: the provider's own keys table is the source of ownership for its suppliers, and the page reads only the provider DB.
    • Effect: if a key changed owner inside the period, all its sessions in the period show under the new owner. The label and the note make that visible.
  12. Breakdown groups.

    • Theirs: selected count in bands 0, 1, 2, 3–5 and 6+; session value in fixed bands (under 250, 250–500, over 500 POKT); also hostname, domain, time of day and fleet groupings (src/engine.js:L141–L147, L358–L361, src/app.js:L16–L20).
    • Igniter: selected count by its exact value, from 1 (only sessions with one of your suppliers). Session value in quartiles of V over your sessions, with each band's V range shown on the page and exported exactly in the CSV (Min V and Max V, in POKT). A settled session with no V is "No payout"; a session not settled yet is "Not settled yet". Groupings are service, supplier, owner, week, application, selected and value (dal/analytics.ts#L223, L356-L364).
    • Why: fixed POKT bands depend on the service and the period, while quartiles always split your sessions into four equal parts. Hostname and domain do not apply to a scope of verified keys.
    • Effect: the Selected and Value groups are not comparable row by row with theirs. Y and B overall are unchanged.
  13. No availability labels.

    • Theirs: a comparison is labeled "partial" when under 90% of sessions have a known outcome. It is labeled "limited" with under 7 informative days, under 24 sessions with an equal share, or when one day holds over 40% of B. Partial rows get no rank (src/engine.js:L476–L492, docs/data-collection.md:L55).
    • Igniter: F is shown whenever B > 0. The page shows the period's open days and missing days instead (coverage.ts#L55-L85).
    • Why: the labels deal with sampled sessions and unknown payouts. Here every session is read, and an unsettled one is shown as open, not unknown.
    • Effect: a thin scope, for example one service over a few days, still shows an F. Read it with the session and day counts beside it.
  14. Left out.

    • The "Past performance" test of an earlier estimate ("Test an estimate"; src/engine.js:L426–L461).
    • The selection diagnostic against the uniform-selection model, and the expected presence (src/engine.js:L261–L280, L393–L424).
    • The Explore view that ranks domains across the network.
    • Gateway grouping (src/app.js:L16–L20).

    These either model the network beyond the provider's own suppliers or need data per session that this sync does not store. Each can be a follow-up.

  15. The What-if selection model.

    • Theirs: the scenario takes the service's latest sampled session population overall: its eligible suppliers, yours among them, and the 50-seat cap (src/engine.js:L223–L231), with the model in src/math.js:L26–L102 and the card in src/app.js:L267–L322.
    • Igniter: the same model and card. The reference is the latest closed day inside the selected period, per service, from the stored session's candidates, the cap param at that height and our verified suppliers' eligible count (Summary). The card's state stays in the card and resets on a new period or service filter.
    • Why: the page reads only stored data, and the reference should belong to the period the page shows.
    • Effect: on their period (6 Sep – 5 Oct 2026) both use 5 Oct, and the numbers match (Testing). On another period, the reference moves with it.

Migration 0024

  • New tables: analytics_supplier_day, analytics_service_day, analytics_session, analytics_session_member, analytics_supplier_day_state, analytics_owner_day and analytics_day_rollup, with their indexes.
  • New column application_settings.analytics_day_retention_days, an integer, default 90.
  • Additive only.

Migration 0025

  • New tables: analytics_selection_reference (per closed day and service: height, candidates, cap, own eligible and the day's version) and analytics_selection_reference_day (the record that a day was counted under a version, also when it stored no row).
  • Retention deletes both with the other analytics tables.
  • Additive only.

Settings

Settings > Analytics history sets analytics_day_retention_days (1–365). AnalyticsSync deletes the days before it on every run, with a plain delete.

CSV exports

  • File names say what they hold: analytics-summary-services_<from>_<to> and analytics-by-<dimension>_<from>_<to>, plus _<n>svc when a services filter is set, then the export's date and time.
  • Amounts are exact POKT with six decimals. The ratio and % columns are exact and rounded half up.
  • The Session value export has the band key as its first column, then Min V and Max V in POKT.

After the walkthrough review

  • One scroll per page. DataTable has a new bounded prop. It defaults to true, so every other table keeps its viewport-height box. The three Analytics tables (Summary services, Breakdown details, Sessions) pass bounded={false}, so the page scrolls as one and the tables only scroll sideways.
  • "All services" has no count. The selector lists every service with data in the retention window, so the list stays the same when the period changes. The header counts only the period's services. Two different counts side by side read as an error, so the selector no longer shows one.
  • Shorter DB debug log. safeQueryParams now logs an array param longer than 50 items as [N items: first 3 …]. The analytics queries pass up to 2,385 supplier ids. At debug level, each of those queries made logtape warn that redaction exceeded maxProperties. With this change, the e2e logs no such warning (measured).
  • Not changed: a hydration warning on the admin pages. It shows in next dev (Turbopack) on about 7–10% of same-tab reloads, as a Radix useId mismatch. It never showed in 60 reloads on next start (measured). Removing single layout providers did not remove it. So it looks like a dev-toolchain effect, not app code (inferred, not proven).

Testing

  • Local e2e with the 2,385 nodefleet.net suppliers that @mykyal's fleet analysis groups under that domain, backfilled from scratch from the mainnet pocketdex indexer (the same GraphQL roots are served publicly at data.pocket.network). For 6 Sep – 5 Oct 2026 and the same 8 services:

    @mykyal's analysis This PR
    vs equal share −35.7% −35.7%
    Actual ÷ equal share 0.64 0.64
    95% range −38.8% to −33.1% −38.8% to −33.2%
    Presence 100.0% 100.0%

    Actual and equal-share amounts are within 0.05% (3,633,438.4 and 5,650,096.4 POKT here), and every per-service result is within 0.1%. The remaining difference is consistent with the day axis (difference 2): @mykyal's analysis keys sessions by end, this PR by start.

  • The backfill of that e2e (51 days, 2,385 suppliers):

    • It took 21 min 25 s, about 25 s per day.
    • It had no pool timeouts and no failed activities. At most 7 DB connections were busy at once, and the longest sync transaction took 4.5 s.
    • On all 50 closed days, every (day, supplier) has exactly as many member rows as funnel seats: 119,250 of 119,250.
  • CSV exports. A script exported all eight CSVs of that view from the page (Summary services and the seven breakdowns). It checked each one against the page and against SQL written from the table definitions: rows and groups, every amount, the exact ratio and %, the week blocks, the value bands with their Min V and Max V, the BOM and the file name. All eight pass.

  • Postgres tests, run only with MIGRATION_TEST_DATABASE_URL, like middleman's pg tests, against the real migrations:

    • analytics.rollup.pg.test.ts: stored = live; key owner and verified-state changes; a day with no state rows; the sync/rebuild race, including two-connection cases; an abandoned sync; the per-batch refresh.
    • analytics.sessions.pg.test.ts: a late page write of an abandoned attempt that lands before the next attempt's delete (it lost rows with the earlier time-based delete); a re-sync that drops rows the indexer no longer returns, and only those; the close counts, including null funnel values.

    On the e2e data, every breakdown also returned the same numbers from the rollups as from the member rows (checked before the final week and value changes; not re-run since).

  • Timings, 30 days on the e2e DB (about 11.6M member rows over the 51-day history). These were measured before the final changes to the week and value queries, which were not timed again:

    Breakdown Time
    Summary 0.45 s
    Supplier 1.3 s
    Owner 1.2 s
    Service, week 0.9 s
    Application, selected, value 0.3–0.4 s

    The page's first view is read on the server, and the summary appears with the HTML.

  • What-if on that e2e (retention 54, so 6 Sep is kept), base on 5 Oct, against @mykyal's tool:

    @mykyal's tool This PR
    Eligible (yours + other) 2,342 + 1,010 = 3,352 2,342 + 1,010 = 3,352
    Seats per draw 50 50
    Chance of at least 1 of yours >99.9% >99.9%
    Expected selected per session 34.93 34.93
    Slider range 0 – 4,700 0 – 4,700
    • Every other step of a full walkthrough, recorded locally, matched @mykyal's own math on the stored references: bsc 29.08; k = 35 gives 56.1%; 500 suppliers give 16.56; fleet sizes for 90 / 95 / 99 / 99.9% are 47 / 61 / 96 / 146; at least 50 at 90% is unreachable.
    • The own count was checked a second way: listing our own ids' configs on data.pocket.network and counting distinct suppliers client-side gave base 2,342 and solana 1,384, the same as stored. The candidate rule with no supplier ids gave exactly pocketdex's candidates on 21 of 21 (service, day) pairs.
    • The e2e found that the indexer refuses more than 20 aliases in one request ("Alias limit exceeded"); the fetcher splits its requests for that. All 53 closed days got a reference, with no failed day, at about 14 s per day.
    • analytics.reference.pg.test.ts: which days are pending; a recount when the day's version changes (a supplier added, a key leaving the verified states); no write when the day changed during the count; a day done with no rows; the inputs (the latest session per service, verified suppliers only).
  • Unit tests for the planning and close rules, the single-flight day id, the DAL SQL and scoping, cancellation, the DB slot limit, the actions, the page math (including the 95% range gate), the exact rounding and the CSV content; and for the selection model against @mykyal's outputs, the selection-count fetcher (our ids only, one cap per height, the 20-alias split) and the reference activity (batches of 200, the over-count guard, the failing-day skip, the empty day).

  • test, check-types (provider-workflows), lint and build pass.

Operator notes

  • The backfill loads the pocketdex indexer. A day is about 12 batches of 200 suppliers, run 12 at a time (ANALYTICS_BATCH_CONCURRENCY = 12). Each batch is one funnel call plus pages of 1,000 sessions. In the e2e that took about 25 s per day, so about 38 min for 90 days. The button asks for confirmation before it starts.
  • The sync uses at most 6 of the worker's DB connections (ANALYTICS_DB_SLOTS; the pool is PG_POOL_MAX, 10 by default).
  • A day being synced counts as stale until its rebuild finishes. A syncing mark older than 2 h counts as abandoned, and AnalyticsSync then rebuilds that day.
  • The selection references read the indexer once per closed day: one request per 200 suppliers and per 20 services. After a deploy, the first AnalyticsSync runs fill the retention window, newest first; until a day has one, the What-if card says it appears once a day closes.

@jorgecuesta jorgecuesta changed the title feat(provider): fleet day sync and the Analytics page feat(provider): analytics day sync and the Analytics page Oct 26, 2026
@jorgecuesta jorgecuesta self-assigned this Oct 26, 2026
@jorgecuesta jorgecuesta added the enhancement New feature or request label Oct 26, 2026
@jorgecuesta jorgecuesta assigned RawthiL and unassigned RawthiL Oct 26, 2026
@jorgecuesta
jorgecuesta requested a review from RawthiL October 26, 2026 19:31
@jorgecuesta
jorgecuesta force-pushed the feat/fleet-day-sync branch 2 times, most recently from a7a3530 to 337a314 Compare October 27, 2026 19:54
Adds the analytics_* day tables, analytics_owner_day and application_settings.analytics_day_retention_days (migration 0024, with a settings field); the AnalyticsSync schedule (forward only) and the manual AnalyticsBackfill, which run one AnalyticsSyncDay per day under a single workflow id, so a day is never synced by both at once, fanning out to AnalyticsSyncBatch children that read getSupplierSessionFunnelJson and getSupplierSessionListJson for the provider's verified suppliers only, keep exactly the rows of each answer, close a (day, supplier) only when its member rows match its funnel seats, and then rebuild the day's breakdown rollups (analytics_owner_day and analytics_session.own_*, versioned per day in analytics_day_rollup); the owner-only Admin > Analytics page (summary, breakdown, sessions, CSV exports, backfill button) over all verified suppliers, reading only those tables; and the "What if your fleet changed?" card, a port of @mykyal's selection model over analytics_selection_reference (migration 0025), which AnalyticsSync fills per closed day and service from a count-only supplierServiceConfigs query over our ids. postGraphql moves to @igniter/graphql, built for ./client and ./analytics, and every CSV export now starts with a UTF-8 BOM.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants