fix(framework): first-ever converter run should full-sync, not silent… - #9045
fix(framework): first-ever converter run should full-sync, not silent…#9045bramhanandlingala wants to merge 1 commit into
Conversation
|
The diagnosis looks legitimate to me, but the fix does not; it effectively disables the incremental state completely. I think a better way is to migrate to |
|
Hi @klesh, thanks so much for reviewing this! Just to clarify the scope of the change — the diff only removes the two lines that copied collectorState.LatestSuccessStart into PrevStartedAt. Everything else stays the same, so incremental mode still works normally after the first run: Close() saves PrevStartedAt, and the next run picks it up and goes incremental as before. It's only the very first run that behaves differently now — it does a proper full sync instead of being mistaken for a run that already happened. On the _tool_github_runs idea — I think we might run into the same issue there. The extractor writes to that table just before the converter runs, in the same pipeline, so on a genuine first run it would already have data too, and we could end up dropping the backlog again the same way. Let me know your thoughts, and happy to adjust if I'm missing something on your end! |
Summary
Fixes a bug where a subtask running for the very first time (fresh install, or
a newly-added converter subtask) is silently bootstrapped into incremental
mode, causing it to filter out the entire already-collected backlog and
produce zero rows in the domain layer — even though the tool-layer table is
fully populated.
Repro: fresh DB → GitHub connection + CICD scope → run blueprint once →
_tool_github_runsis populated, butcicd_pipelinesreturns 0 rows.Root Cause
bootstrapStateFromCollectorStateIfNeeded(inbackend/helpers/pluginhelper/api/subtask_state_manager.go) copied thesibling collector's
LatestSuccessStartinto the subtask's ownPrevStartedAtwhenever the subtask had no prior state of its own.PrevStartedAt == nilis the framework's signal that a subtask has never runbefore, and it's what triggers a full sync in
calculateStateManagerIncrementalMode. On a fresh install, the collectorfinishes moments before the converter runs — in the same pipeline execution —
so
LatestSuccessStartis effectively "now". BootstrappingPrevStartedAtfrom that value made the converter's genuinely-first-ever run look like a
returning run, flipping it into incremental mode and filtering rows by
sinceagainst historical timestamps (e.g.github_updated_at), whichsilently dropped the entire backlog.
Fix
Stop bootstrapping
PrevStartedAtfrom the collector state. OnlyTimeAfteris still inherited from the collector, since it's just a fallback lower bound
and has no effect on the incremental-vs-full-sync decision.
This is a shared framework fix (
SubtaskStateManager, used byStatefulDataConverter/StatefulDataExtractor), so it applies to everyplugin using stateful subtasks — not just GitHub's CICD converters.
Closes #9016