feat(ci): auto-bump recall floors after stable releases by carlos-alm · Pull Request #2168 · optave/ops-codegraph-tool

carlos-alm · 2026-07-29T01:04:58Z

Summary

Adds a workflow that triggers after a successful stable Publish run, downloads the pre-publish benchmark artifact, and raises any per-language precision/recall floor that improved
Floors are a one-way ratchet — never lowered — keeping the resolution benchmark gate honest as engine accuracy improves over time
Opens a PR automatically only when at least one floor actually changed

This replaces #2162, reverted in #2166. Fixes the 3 issues Greptile found on #2162 (Confidence Score 2/5):

Trigger scope. Restricted to release/workflow_dispatch events — the daily schedule trigger and push (dev builds) don't produce a benchmark-results-json artifact and would fail the download step.
Version/artifact binding. Derives the release version via git describe on the triggering workflow_run.head_sha, not on whatever main's tip happens to be when the job runs — so a workflow_dispatch retry that finishes after a newer release lands can't mislabel and force-push its (older) metrics onto the newer release's branch.
Regex injection. Escapes the language key before interpolating it into the threshold-matching regular expression, so a future key containing regex metacharacters (e.g. c++) matches literally instead of failing to match or matching unintended text.

Test plan

node --check scripts/update-recall-floors.mjs
Verified the regex-escape fix: escapeRegExp('c++') → c\+\+, matches c++ literally
Validated workflow YAML syntax
npm run lint clean
Confirm the workflow runs correctly on the next stable Publish

Adds a workflow that downloads the pre-publish benchmark artifact after a successful stable Publish run and raises any per-language precision/recall floor that improved, opening a PR. Floors are a one-way ratchet — they are never lowered. Restricts the trigger to release/workflow_dispatch events (scheduled and push-triggered dev builds don't produce a benchmark artifact and would fail), derives the release version from the triggering run's own commit rather than main's tip at execution time (so a late-finishing retry can't mislabel its metrics onto a newer release), and escapes language keys before interpolating them into a regular expression. Impact: 2 functions changed, 2 affected

greptile-apps · 2026-07-29T01:09:54Z

+  const pattern = new RegExp(
+    `(  ${escapeRegExp(lang)}: \\{ precision: )([\\d.]+)(, recall: )([\\d.]+)( \\})`,
+  );


Quoted language keys are skipped

When results improve for a hyphenated language such as pts-javascript, this pattern searches for an unquoted property while the threshold table uses a quoted key, causing the updater to silently skip all seven current quoted entries.

Suggested change

const pattern = new RegExp(

`( ${escapeRegExp(lang)}: \\{ precision: )([\\d.]+)(, recall: )([\\d.]+)( \\})`,

);

const escapedLang = escapeRegExp(lang);

const pattern = new RegExp(

`( (?:${escapedLang}|['"]${escapedLang}['"]): \\{ precision: )([\\d.]+)(, recall: )([\\d.]+)( \\})`,

);

Fixed — the pattern now optionally matches a wrapping single quote around the key ('?${escapeRegExp(lang)}'?), so hyphenated keys like 'pts-javascript' match alongside bare identifier keys.

greptile-apps · 2026-07-29T01:09:54Z

+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - uses: actions/setup-node@v6


Privileged actions use mutable tags

The write-capable job executes actions/checkout@v6 and actions/setup-node@v6 through mutable tags, so an upstream tag compromise would run unreviewed code with repository-content and pull-request write authority. Pin both actions to full commit SHAs. How this was verified: The job grants write permissions and explicitly passes its write-scoped token to the mutable checkout action.

Valid concern, but it applies to every workflow in this repo (all use mutable major-version tags — checkout, setup-node, cache, upload/download-artifact, etc.), not something specific to this PR's diff. Filed as a repo-wide issue rather than pinning just this one file inconsistently: #2170

Reopened and fixed for real this time — Greptile's round-3 re-review kept flagging this same gap, and since this workflow file is entirely new in this PR (not a pre-existing file with unrelated mutable tags elsewhere), pinning just its own two actions in scope makes sense rather than continuing to defer. actions/checkout@v6 and actions/setup-node@v6 are now pinned to their commit SHAs (with a version comment for readability). The repo-wide cleanup for all other workflows remains tracked in #2170.

greptile-apps · 2026-07-29T01:09:54Z

+          git config user.email "github-actions[bot]@users.noreply.github.com"
+
+          git fetch --tags
+          VERSION=$(git describe --tags --match "v*" --abbrev=0 "$HEAD_SHA" 2>/dev/null | sed 's/^v//' || echo "unknown")


Missing tags produce shared unknown branch

When HEAD_SHA has no reachable v* tag, the suppressed git describe failure assigns unknown, causing unrelated runs to force-push the same chore/recall-floors-vunknown branch and create a misleading release PR. Fail the workflow instead of converting an unresolved version into a valid branch name.

Suggested change

VERSION=$(git describe --tags --match "v*" --abbrev=0 "$HEAD_SHA" 2>/dev/null | sed 's/^v//' || echo "unknown")

VERSION=$(git describe --tags --match "v*" --abbrev=0 "$HEAD_SHA" | sed 's/^v//')

Fixed — the workflow now exits with an error when no tag resolves, instead of falling back to a shared 'unknown' version/branch name.

greptile-apps · 2026-07-29T01:09:58Z

Greptile Summary

Adds an automated post-publish floor-ratchet workflow.

Downloads benchmark results only from successful release or manually dispatched Publish runs.
Validates the artifact-bound release version and skips prerelease metrics.
Raises improved per-language precision and recall floors without lowering existing values.
Uses release-specific branches to open or update a pull request only when floors change.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
.github/workflows/update-recall-floors.yml	Adds the post-Publish workflow, artifact-bound version validation, stable-release gating, SHA-pinned actions, and conditional PR creation; the previously reported workflow issues are resolved.
scripts/update-recall-floors.mjs	Adds the one-way threshold updater with literal language-key matching and support for both quoted and bare threshold keys; the previously reported quoted-key failure is resolved.

Sequence Diagram

sequenceDiagram
  participant Publish as Publish workflow
  participant Artifact as Benchmark artifact
  participant Ratchet as Update Recall Floors
  participant Script as Floor updater
  participant PR as GitHub PR
  Publish->>Artifact: Upload benchmark-result.json
  Publish-->>Ratchet: Successful workflow_run
  Ratchet->>Artifact: Download triggering run artifact
  Ratchet->>Ratchet: Validate version and stable status
  alt Stable release
    Ratchet->>Script: Update precision/recall floors
    alt Floors improved
      Script-->>Ratchet: Modified threshold table
      Ratchet->>PR: Push release branch and open PR
    else No improvement
      Script-->>Ratchet: No changes
    end
  else Prerelease
    Ratchet-->>Ratchet: Skip floor update
  end

_{Reviews (10): Last reviewed commit: "fix: skip recall-floors ratchet for prer..." | Re-trigger Greptile}

github-actions · 2026-07-29T01:13:05Z

Codegraph Impact Analysis

2 functions changed → 2 callers affected across 1 files

fmt in scripts/update-recall-floors.mjs:40 (1 transitive callers)
escapeRegExp in scripts/update-recall-floors.mjs:46 (1 transitive callers)

carlos-alm · 2026-07-29T01:30:28Z

@greptileai

greptile-apps · 2026-07-29T01:35:06Z

+          git config user.email "github-actions[bot]@users.noreply.github.com"
+
+          git fetch --tags
+          VERSION=$(git describe --tags --match "v*" --abbrev=0 "$HEAD_SHA" 2>/dev/null | sed 's/^v//')


Tag lookup selects prior release

When main advances after a workflow_dispatch run starts, Publish creates the new release tag on the newer main tip, while this lookup searches backward from the older workflow_run.head_sha. It therefore selects the preceding release tag, causing the workflow to mislabel the PR and force-push its changes to the preceding release's floor-update branch.

Fixed — the version is now read directly from the downloaded benchmark artifact's version field (written by compute-version at the start of this same Publish run and later used verbatim to build the release tag), instead of re-deriving it via git describe on head_sha. This removes the race entirely rather than just re-scoping the git describe search, since the artifact's version can never disagree with the tag the publish job will create.

…d_sha tag lookup The Publish workflow's release tag is pushed by its publish job against whatever main HEAD is at that point in the pipeline, not against the triggering run's head_sha. If main advances mid-run (e.g. a slow workflow_dispatch retry), the new tag lands on a commit outside head_sha's ancestry, so `git describe --tags $HEAD_SHA` silently resolves to the previous release's tag instead of failing. Read the version directly from the downloaded benchmark artifact instead — its version field is written from compute-version's output at the start of the same Publish run and is the exact string later used to build the release tag, so it can't race. Also drops the now- unneeded full-history checkout (fetch-depth: 0) that existed only to support the git describe/tag lookup.

carlos-alm · 2026-07-29T01:56:37Z

@greptileai

…workflow This job runs with contents:write + pull-requests:write and passes its write-scoped token straight into the checkout step. Greptile flagged the mutable v6 tags as a supply-chain risk in round 1; deferred at the time to a repo-wide tracking issue (#2170) since every workflow in the repo has the same gap. Round 3's re-review kept the confidence score capped on this same concern, and this file is entirely new in this PR (not a pre-existing file whose unrelated lines would be touched), so pinning just these two refs here is in scope. The remaining workflows stay tracked under #2170 as a separate, dedicated cleanup.

carlos-alm · 2026-07-29T05:39:44Z

Round 3's confidence gap (mutable action tags in this workflow — Files Needing Attention) is fixed: actions/checkout@v6 and actions/setup-node@v6 are now pinned to commit SHAs. Since this whole file is new in this PR, pinning its own two refs is in scope, unlike the repo-wide cleanup tracked separately in #2170.

carlos-alm · 2026-07-29T06:10:43Z

@greptileai

greptile-apps · 2026-07-29T06:13:28Z

+
+  // Match lines of the form `  lang: { precision: X, recall: Y },` (bare identifier
+  // keys) or `  'lang-with-hyphens': { precision: X, recall: Y },` (keys that aren't
+  // valid bare identifiers, e.g. "pts-javascript", are quoted in the source).


Short keys match quoted suffixes

When metrics improve for a bare key such as java, the independently optional quotes let this pattern match the java suffix in the earlier 'dynamic-java' entry. The updater then compares against or replaces the dynamic-Java thresholds, leaving Java unchanged or assigning Java's metrics to the wrong language.

Not a bug — verified against the real file. The claimed collision doesn't occur because the required two leading spaces must immediately precede the (optionally quoted) key; 'dynamic-java' has dynamic- between the indent and java, so the adjacency check fails there:

const pattern = new RegExp(`( '?java'?: \\{ precision: )([\\d.]+)(, recall: )([\\d.]+)( \\})`); pattern.test(" 'dynamic-java': { precision: 1.0, recall: 1.0 },") // -> false

Matching against the actual threshold table, the java pattern correctly finds the real bare java: { precision: 0.7, recall: 0.3 } entry (line 177) and never touches 'dynamic-java' (line 150). I couldn't reproduce a collision anywhere in the real 34-language table. Leaving the regex as-is.

You're right — I was wrong. The two-space indent anchor immediately precedes '?, so 'dynamic-java' never matches: after consuming ', the engine expects java next but finds dynamic-. Your demo confirms it, and the logic holds for any *-java-style key. I'll retract this comment.

greptile-apps · 2026-07-29T06:13:28Z

+          # run and is the exact string later used to build that tag, so it's
+          # immune to the race.
+          VERSION=$(node -p "require('./benchmark-results-json/benchmark-result.json').version")
+          if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$'; then


Published versions fail validation

When a stable release uses a version with SemVer build metadata such as v2.1.0+build.1, Publish accepts that tag and records 2.1.0+build.1 in the benchmark artifact, but this narrower regex rejects the +. The workflow exits before committing the improved floors or opening their pull request.

Fixed — the release-event path in compute-version (publish.yml) takes github.event.release.tag_name verbatim with no format validation, so a real GitHub Release tagged with legitimate SemVer build metadata (e.g. v2.1.0+build.1) would indeed sail through Publish and land in the benchmark artifact's version field, then get rejected here. Extended the regex to accept an optional +[a-zA-Z0-9.]+ suffix alongside the existing prerelease suffix, and verified it against plain/prerelease/build-metadata/prerelease+build samples in both node and bash grep -E before pushing.

compute-version's release-event path takes github.event.release.tag_name verbatim with no format validation, so a real GitHub Release tagged with legitimate SemVer build metadata (e.g. v2.1.0+build.1) sails through Publish untouched but was being rejected by this workflow's stricter version regex, aborting before the floor-bump PR could open. Extend the regex to accept an optional "+[a-zA-Z0-9.]+" suffix alongside the existing prerelease suffix.

greptile-apps · 2026-07-29T06:23:23Z

+          # path takes github.event.release.tag_name verbatim with no format
+          # validation, so a real GitHub Release tagged with legitimate SemVer
+          # build metadata must not be rejected here.
+          if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?(\+[a-zA-Z0-9.]+)?$'; then


Hyphenated versions fail validation

When a stable GitHub Release uses a valid SemVer tag such as v2.1.0-rc-hotfix or v2.1.0+build-hotfix, Publish records that version verbatim in the benchmark artifact, but this validator rejects the identifier hyphen, causing the workflow to exit without committing the improved floors or opening a pull request.

Suggested change

if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?(\+[a-zA-Z0-9.]+)?$'; then

if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.-]+)?(\+[a-zA-Z0-9.-]+)?$'; then

Fixed — you're right, SemVer alphanumeric identifiers can contain internal hyphens (e.g. "rc-hotfix"), and my previous character classes only allowed [a-zA-Z0-9.], missing that case. Added - to both the prerelease and build-metadata character classes and verified against plain/prerelease/build-metadata/hyphenated-identifier samples in both node and bash grep -E before pushing.

SemVer alphanumeric identifiers (e.g. "rc-hotfix" in v2.1.0-rc-hotfix, or "build-hotfix" in v2.1.0+build-hotfix) may themselves contain internal hyphens per the spec. The previous character classes only allowed [a-zA-Z0-9.], rejecting valid versions Publish would accept verbatim from a real GitHub Release tag. Add "-" to both suffix character classes.

carlos-alm · 2026-07-29T06:44:17Z

@greptileai

greptile-apps · 2026-07-29T06:46:52Z

+    if: >
+      github.event.workflow_run.conclusion == 'success' &&
+      (github.event.workflow_run.event == 'release' ||
+       github.event.workflow_run.event == 'workflow_dispatch')


Prereleases ratchet stable floors

When a GitHub prerelease such as v2.1.0-rc.1 is published, this condition accepts its successful Publish run as a stable release, and the artifact version passes the later validation. The workflow then raises the stable benchmark thresholds using prerelease metrics and opens a floor-update PR even though no stable release occurred.

Fixed — added a new Extract and validate release version step that runs right after downloading the artifact. It parses the SemVer prerelease identifier (the -xyz component before any +build metadata) from the artifact's own version field and sets stable=false when one is present, since that's the only reliable stable-vs-prerelease signal available here (workflow_run.event == 'release' fires for GitHub prereleases too, and Publish's own compute-version step doesn't check the prerelease flag either). The Update recall floors step is now gated on steps.version.outputs.stable == 'true', so a prerelease's benchmark numbers never touch the threshold file or open a PR.

carlos-alm · 2026-07-29T06:59:19Z

@greptileai

greptile-apps Bot reviewed Jul 29, 2026

View reviewed changes

carlos-alm mentioned this pull request Jul 29, 2026

Pin GitHub Actions to commit SHAs instead of mutable version tags repo-wide #2170

Open

carlos-alm added 2 commits July 28, 2026 18:30

fix: match quoted hyphenated language keys in threshold table (#2168)

5489789

fix: fail instead of falling back to a shared vunknown branch (#2168)

b87265a

greptile-apps Bot reviewed Jul 29, 2026

View reviewed changes

fix: skip recall-floors ratchet for prerelease versions

36c0af7

carlos-alm merged commit cc0ae2b into main Jul 29, 2026
29 checks passed

carlos-alm deleted the feat/recall-floors-v2 branch July 29, 2026 07:45

github-actions Bot locked and limited conversation to collaborators Jul 29, 2026

	VERSION=$(git describe --tags --match "v*" --abbrev=0 "$HEAD_SHA" 2>/dev/null \| sed 's/^v//' \|\| echo "unknown")
	VERSION=$(git describe --tags --match "v*" --abbrev=0 "$HEAD_SHA" \| sed 's/^v//')

	if ! echo "$VERSION" \| grep -qE '^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?(\+[a-zA-Z0-9.]+)?$'; then
	if ! echo "$VERSION" \| grep -qE '^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.-]+)?(\+[a-zA-Z0-9.-]+)?$'; then

Uh oh!

Conversation

carlos-alm commented Jul 29, 2026

Summary

Test plan

Uh oh!

greptile-apps Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

github-actions Bot commented Jul 29, 2026

Codegraph Impact Analysis

Uh oh!

carlos-alm commented Jul 29, 2026

Uh oh!

greptile-apps Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlos-alm commented Jul 29, 2026

Uh oh!

carlos-alm commented Jul 29, 2026

Uh oh!

carlos-alm commented Jul 29, 2026

Uh oh!

greptile-apps Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

carlos-alm Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlos-alm commented Jul 29, 2026

Uh oh!

greptile-apps Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlos-alm commented Jul 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

greptile-apps Bot commented Jul 29, 2026 •

edited

Loading

carlos-alm Jul 29, 2026 •

edited

Loading