Skip to content

feat(query): find code by the words in its body - #5052

Open
rohit-jsfreaky wants to merge 2 commits into
Graphify-Labs:v8from
rohit-jsfreaky:feat/query-body-words
Open

rohit-jsfreaky wants to merge 2 commits into
Graphify-Labs:v8from
rohit-jsfreaky:feat/query-body-words

Conversation

@rohit-jsfreaky

@rohit-jsfreaky rohit-jsfreaky commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Closes #5051

What does this PR do?

query now also looks at the words inside a node's body. This is the follow-up to #4982, built on top of the existing scoring tiers.

Extraction (Python only, to keep the diff small). File, class and function nodes carry body_terms: the unique words of their own source lines. That means identifier parts, string words and comment words, 3-40 characters long. Secret-like text is redacted first, the same way as for rationale labels. A nested def keeps its words in its outer function. AST cache schema 47 -> 48.

Query only. path and explain do not change.

  1. Body tier: scored below the exact and substring name tiers. For any one word, a name match still beats a body match, and the per-term seed guarantee still holds.
  2. Body ranking: a separate body ranking (BM25 over body_terms) adds 3 extra seeds after the name seeds. The answer's file order fuses the name ranking and the body ranking.
  3. Vendored copies rank lower: vendor/, node_modules/, a nested packages/, jquery-3.5.1.js-style names and .min.js.

Cost in a new process. Building a full word index made the first CLI query 2 s slower on Django. Instead, all body words sit in one string and each query word is found with a substring search. A long-running MCP server switches to a full index after 32 distinct words. Both paths return the same nodes, and a test checks that.

Diff: graphify/serve.py +350 / -21, graphify/extract.py +78, one docs line, 10 tests in tests/test_query_body_terms.py.

One trade-off to flag

Step 2 changes the answer's file order: a file that ranks first by name can now be shown after a file that is strong in both rankings. Seed choice and the per-term guarantee are not affected.

I also measured a strict version, the body tier alone with no fused order. It only reaches 66% (vs 75%). In 52 of 54 Django misses, one exact name match took every seed slot and kept the gold file out of the top 5. If you want names to always decide the order, I can cut this PR down to the strict version.

Numbers

SWE-bench Lite + Verified, 293 questions. The gold file is the one the fix touched. Every setting was chosen on the 114 Django questions only and frozen at one commit. The rest ran once, through graphify benchmark --questions (#4342).

top-1 top-3 top-5 median answer tokens
179 non-Django questions, 0.9.102 31% 53% 61% 2,202
BM25 over whole files 30% 56% 67% 46,541 (to read 5 files)
this PR 45% 70% 75% 2,237
116 Verified questions (never used for any choice), 0.9.102 65%
this PR 78%

Paired top-5 on the 179: this PR vs 0.9.102, 29 questions gained and 4 lost (McNemar p = 1e-5). This PR vs BM25-files, 26 gained and 11 lost (p = 0.02).

Per repo top-5, 0.9.102 -> this PR:

repo 0.9.102 this PR
astropy 75% 92%
pytest 53% 74%
requests 69% 92%
sphinx 60% 72%
pylint 33% 53%
xarray 69% 77%
flask 100% 100%
seaborn 67% 50% (one question of 6)

Cost

0.9.102 this PR
graph.json size +11% to +18%; requests +35% (bundled urllib3, long docstrings)
build time (Django) 93.9 s 94.2 s
cold CLI graphify query, Django (median of 20) 3.63 s 3.86 s
warm query, Django (median of 114) 249 ms 282 ms

Limitations

  • Python only. Other languages behave exactly as before. If the approach is OK, other extractors can add body_terms the same way.
  • The 179 questions are not fully fresh. Earlier rounds of this study looked at the 63 Lite questions' misses, and the vendored-file rule came from one of those loss lists. That is why I also show the 116 Verified questions, which were never used for any choice.
  • Answers are about 35 tokens larger at the median.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Tests or CI
  • Refactor
  • Security fix

Verification & Invariants

Invariants:

  • With the body tier off, query gives exactly 0.9.102's answers.

  • path and explain are unchanged.

  • The two word-lookup paths return the same nodes.

  • The new process gives byte-identical answers to the full-index version on all 293 questions, at depths 2 and 3 with a fixed hash seed. With a random seed, only the EDGE line order varies, the same way it already does on v8.

  • Read the CONTRIBUTING.md guide.

  • Reproduced the issue and identified the invariant.

  • Made the smallest fix necessary.

  • Added a regression test (if bug fix) or isolated boundary test.

  • Kept the PR description synchronized with the final implementation.

  • Documented any limitations / unsupported cases explicitly.

How was this tested?

uv run --frozen pytest tests/test_query_body_terms.py -q        # 10 passed; 7 fail with v8's serve.py/extract.py (the other 3 check the two lookup paths agree)
uv run --frozen pytest tests -q                                  # 10001 passed, 0 failed (v8: 9991 passed, 0 failed)
uv run --frozen pytest tests/test_query*.py tests/test_serve*.py tests/test_cache.py -q   # 465 passed on Windows, plus 1 symlink test that also fails on v8 on Windows
uv run --frozen ruff check .                                     # clean
uv run --frozen pyright graphify/serve.py graphify/extract.py    # no new errors
graphify benchmark --questions on 293 SWE-bench Lite + Verified questions   # numbers above
graphify query "credit history transactions" on the issue's repro            # now finds Ledger.entries() and its caller

The scripts and every result file are on my fork, branch claude/query-study, folder research/query/pr2/, with a README that says how each number was made.

Graphify-specific checklist

  • I updated generated skill artifacts (uv run python -m tools.skillgen --bless) when changing their source fragments. (Not needed: no skill fragments changed.)
  • I confirmed that AST/structural extraction remains deterministic (no ambient state dependencies like ENV variables).
  • I reviewed changes for security implications (no unsafe interpolation into shell/Python).
  • I confirmed no API keys or local-only graph data are included.
  • (If applicable) I disclosed AI authorship in my commit messages.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Thanks for the pull request, @rohit-jsfreaky. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.

Not checked: tests were not run; no formal proof of the changed code.


Graphify review — findings

Adds an optional body_terms field to Python file, class and function nodes, holding the unique identifier parts, string words and comment words from each node's own source lines, with credential-shaped text redacted and words limited to 3–40 characters. query uses it as a body tier below the name tiers and as a body ranking that adds start nodes after the name seeds and feeds the file ordering, while path and explain ignore it. On nine Python repos graph.json grows 11–35% with no measurable build-time cost, and _attach_python_body_terms decides which node owns each line span, so decorator lines are where that ownership can go wrong.

Worth a look

  • Decorator text is assigned to the parent node instead of the decorated definition — graphify/extract.py:1859 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review

Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 5274 functions depend on the 700 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: _query_graph_text() — 55 callers, 10 callees
  • worse: _score_query() — 23 callers, 9 callees
  • worse: collect_files() — 22 callers, 8 callees
  • worse: _query_terms() — 32 callers, 3 callees
  • new: walk() — 3 callers, 4 callees
  • worse: _extract_python_rationale() — 1 callers, 8 callees
  • worse: _xaml_csharp_class_nodes() — 1 callers, 8 callees

Verification — 5274 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 4636 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

527 of 527 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety (noncode-change, uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • tests/pi.test.ts — full-run-safety
  • tests/test_4874_jvm_resolution.py — impact, full-run-safety
  • tests/test_4909_reference_binding_scale.py — full-run-safety
  • tests/test_4978_reconcile_cost.py — impact, full-run-safety
  • tests/test_4992_ts_import_types.py — impact, full-run-safety
  • tests/test_affected_cli.py — impact, full-run-safety
  • tests/test_affected_defines.py — full-run-safety
  • tests/test_affected_member_seed.py — full-run-safety
  • tests/test_affected_overrides.py — impact, full-run-safety
  • tests/test_affected_path_file_node.py — impact, full-run-safety
  • tests/test_affected_seed_reference_forms.py — impact, full-run-safety
  • tests/test_agents_platform.py — full-run-safety
  • tests/test_alias_folding.py — impact, full-run-safety
  • tests/test_analyze.py — impact, full-run-safety
  • tests/test_anthropic_custom_endpoint.py — full-run-safety
  • tests/test_antigravity_install.py — full-run-safety
  • tests/test_apm_fallback_version.py — full-run-safety
  • tests/test_architecture_doc.py — full-run-safety
  • tests/test_astro_extraction.py — impact, full-run-safety
  • tests/test_astro_import_ids.py — impact, full-run-safety
  • tests/test_atomic_canvas_export.py — full-run-safety
  • tests/test_atomic_version_stamp.py — full-run-safety
  • tests/test_atomic_writes.py — full-run-safety
  • tests/test_backend_env_isolation.py — full-run-safety
  • tests/test_backend_extras.py — full-run-safety
  • tests/test_backend_opt_in.py — full-run-safety
  • tests/test_benchmark.py — impact, full-run-safety
  • tests/test_benchmark_raw_graph.py — impact, full-run-safety
  • tests/test_benchmark_real_query.py — impact, full-run-safety
  • tests/test_binary_media_words.py — impact, full-run-safety
  • tests/test_blade_extractor.py — impact, full-run-safety
  • tests/test_build.py — impact, full-run-safety
  • tests/test_build_located_semantic_identity.py — impact, full-run-safety
  • tests/test_build_merge_dedup_scope.py — impact, full-run-safety
  • tests/test_build_merge_ghost_fold_4955.py — impact, full-run-safety
  • tests/test_build_merge_hyperedges_and_prune.py — impact, full-run-safety
  • tests/test_build_merge_shrink_guard.py — impact, full-run-safety
  • tests/test_builtin_global_type_refs.py — impact, full-run-safety
  • tests/test_c_include_resolution.py — impact, full-run-safety
  • tests/test_cache.py — impact, full-run-safety
  • tests/test_cache_long_paths.py — impact, full-run-safety
  • tests/test_cache_stale_import_target.py — impact, full-run-safety
  • tests/test_cache_windows_case_4925.py — impact, full-run-safety
  • tests/test_callflow_html.py — full-run-safety
  • tests/test_cargo_introspect.py — full-run-safety
  • tests/test_cargo_missing_manifest.py — full-run-safety
  • tests/test_carried_hyperedge_remap.py — impact, full-run-safety
  • tests/test_case_only_decl_ids_4904.py — impact, full-run-safety
  • tests/test_case_sensitive_resolution.py — impact, full-run-safety
  • tests/test_charmap_encoding.py — impact, full-run-safety
  • … and 477 more

non-code file(s) changed (docs/how-it-works.md) → running the full suite for safety (a code graph can't see config/fixture/data deps)

changed code file(s) with no mapped test (docs/how-it-works.md) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Docs that may be stale (advisory)

…and 10 more.

· 3 grounded finding(s) anchored inline below; 4 more finding(s) on lines outside this diff (see the check run).

Comment thread graphify/extract.py
if target is not None and words:
target["body_terms"] = " ".join(sorted(words))

def walk(node, owner_nid: str, is_function: bool) -> None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — walk()

high coupling complexity (Ca·Ce = 12).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/extract.py
walk(root, file_nid, False)


def _extract_python_rationale(path: Path, result: dict) -> None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — _extract_python_rationale()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread graphify/serve.py
return _score_query(G, terms, collect_per_term_seeds=False).ranked


def _score_query(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — _score_query()

fans out to 9 callees (efferent coupling); 23 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

@rohit-jsfreaky

Copy link
Copy Markdown
Contributor Author

The bot's decorator finding was right, thanks. A decorated def was cut at its def / class line, so @app.route("/invoices/export") and @audit_trail("ledger") landed in the words of the file or the enclosing class instead of the function they decorate.

Fixed in 642cff0: a decorated_definition is now one span owned by the definition it wraps. That covers functions, methods with stacked decorators, and decorated classes. The new test test_decorator_words_belong_to_the_decorated_definition fails before the fix and passes after it. The query, serve, Python and rationale tests pass (686), and ruff is clean.

This moves only decorator words between nodes, so I haven't re-run the 293-question benchmark for it. I'll post updated numbers here if they change.

The other inline notes are coupling metrics on functions this PR extends (_score_query, walk), not findings.

@rohit-jsfreaky

Copy link
Copy Markdown
Contributor Author

Updated numbers with the decorator fix (642cff0). I rebuilt all 125 graphs from scratch and re-ran graphify benchmark --questions on all 293 questions:

split top-5 before fix top-5 after fix top-1 / top-3 after
Django (114) 85 (75%) 85 (75%) 46% / 66%
other 179 135 (75%) 135 (75%) 45% / 69%
Verified (116) 90 (78%) 90 (78%) 49% / 73%

No question entered or left the top 5. 8 questions moved by one or two places: 2 up and 6 down by one. I rebuilt those graphs from the same base without the fix to separate the two effects. All 8 moves come from the fix, and the newer upstream base only changed answer sizes. So the decorator change is neutral for ranking. I'm keeping it because the words now sit on the right node. The headline numbers in the description still hold.

The data is on my fork, branch claude/query-study, folder research/query/pr2/decorator-fix/.

@rohit-jsfreaky

Copy link
Copy Markdown
Contributor Author

Rebased onto current v8 (after 0.9.104). The only conflict was _AST_CACHE_SCHEMA: this PR now bumps it 40 -> 41. Query, cache and CLI tests: 231 passed, 1 skipped (the Windows symlink case). ruff is clean.

@rohit-jsfreaky

Copy link
Copy Markdown
Contributor Author

Rebased again onto v8 0.9.105, which is already on schema 42. This PR now bumps _AST_CACHE_SCHEMA 42 -> 43. Query, cache and CLI tests pass (164), and ruff is clean. #5113 also bumps to 43; whichever lands second takes 44.

@rohit-jsfreaky

Copy link
Copy Markdown
Contributor Author

Rebased onto v8 0.9.106. Two conflicts: _AST_CACHE_SCHEMA (now 43 -> 44) and serve.py, where v8 added the demote_docs option to _query_path_weight and _score_query. I kept both: demote_docs still feeds keep_docs, and the body tier passes body=True next to it. Query, serve, MCP and cache tests: 521 passed. ruff is clean, and pyright reports the same 45 errors in serve.py as on v8.

@rohit-jsfreaky

Copy link
Copy Markdown
Contributor Author

Rebased onto v8 0.9.107. The only conflict was _AST_CACHE_SCHEMA, now 44 -> 45 (#5152 took 44). Query, serve, MCP, cache and read-path tests: 528 passed. ruff is clean.

@rohit-jsfreaky

Copy link
Copy Markdown
Contributor Author

Rebased onto v8 0.9.108. The only conflict was _AST_CACHE_SCHEMA, now 45 -> 46. Query, serve, MCP and cache tests pass, and ruff is clean.

rohit-jsfreaky and others added 2 commits November 17, 2026 00:06
`query` only matched node names and labels, so a question worded the way
an issue is written ("credit history transactions endpoint") ranked tests,
docs and look-alike names above the function whose body does the work.

Python file, class and function nodes now carry `body_terms`: the unique
words of their own source lines (identifier parts, string and comment
words, 3-40 chars, secret-like text redacted first, nested defs keep their
own words). In `query` only (`path` and `explain` are unchanged):

- a body tier scores below the exact/substring name tiers, so per word a
  name match still beats a body match and the per-term seed guarantee holds;
- a separate body ranking adds 3 extra seeds after the name seeds, and the
  answer's file order fuses the name and body rankings;
- vendored copies (vendor/, node_modules/, nested packages/, jquery-x.y.z.js,
  .min.js) rank lower.

In a new process the body words are searched as one string instead of
building a full word index, so a cold CLI query stays within 0.25 s of
before. AST cache schema 47 -> 48 (Python output changes).

SWE-bench Lite + Verified, settings chosen on the 114 Django questions only:
top-5 file hit on the 179 other questions 61% -> 75% (v8 vs this,
McNemar p=1e-5), on the 116 Verified questions never used for any choice
65% -> 78%. graph.json +11-35%, build time unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
`body_terms` cut a decorated def at the `def`/`class` line, so its
decorator lines (`@app.route("/invoices/export")`) were counted in the
parent's words (the file or the class) instead of the definition they
decorate. Treat a `decorated_definition` as one span owned by the def.

Co-Authored-By: Claude <noreply@anthropic.com>
@rohit-jsfreaky

Copy link
Copy Markdown
Contributor Author

Rebased onto v8 0.9.111. The only conflict was _AST_CACHE_SCHEMA, now 47 -> 48. Query, serve, MCP and cache tests: 527 passed, 1 skipped. ruff is clean.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor Pull request from a returning community contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: query only matches names, so code whose body has the question's words is never found

1 participant