Skip to content

fix(csharp): escape contextual-keyword names the grammar rejects (#5043) - #5262

Open
ThatAblaze wants to merge 1 commit into
Graphify-Labs:v8from
ThatAblaze:ntextual-keyword-names
Open

ThatAblaze wants to merge 1 commit into
Graphify-Labs:v8from
ThatAblaze:ntextual-keyword-names

Conversation

@ThatAblaze

Copy link
Copy Markdown

What does this PR do?

tree-sitter-c-sharp 0.23.5 reads partial, required, async, nint, nuint and with as their keywords when C# reads a plain name: a statement that starts with one (partial |= x;, required.Add(1);), async in most expressions, nint = 1;, var with = x;. The statement becomes an ERROR, and when it is the last statement of an if block that an else block follows, the recovery swallows the rest of the file and only the file node is extracted.

extract_csharp now rewrites such a name to @name before parsing, which is the same identifier to the compiler and a name to the grammar. The @ takes the place of an adjacent blank, so the rewrite is byte-length preserving and every reported line and column still matches the file (#2876). This runs only on a file that fails to parse, escapes every occurrence of a word together so a local's declaration and its uses keep one spelling (a call on the local still binds through its type), restores a word the parse does not need, and is dropped unless the file then parses cleanly.

The AST cache schema moves from 47 to 48 so cached C# extractions re-run.

Fixes #5043.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Tests or CI
  • Refactor
  • Security fix

Verification & Invariants

Invariants: Line numbers stay true, untouched files stay untouched, keywords stay keywords, and the cache version bump makes cached C# results rebuild. Each has a test that proves it.

  • Read the CONTRIBUTING.md guide.
  • Reproduced the issue and identified the invariant.
  • Made the smallest fix necessary.
  • Added a regression test (if bug fix) or isolated boundary test.
  • Kept the PR description synchronized with the final implementation.
  • Documented any limitations / unsupported cases explicitly.

How was this tested?

Windows 10, Python 3.13.1, after uv sync --all-extras.

  • uv run --no-sync pytest tests/test_csharp_contextual_keyword_names.py -q: 17 passed. The cases cover the whole-file loss, unshifted source locations, byte-length preservation, each word in a form the grammar rejects, receiver binding through an escaped local, keyword uses left untouched, a word the parse does not need keeping its spelling, comments and strings, a healthy file and a still-broken file left alone, and the cache bump.
  • The same file against v8's extract.py and cache.py: collection fails, since the function does not exist there. With a small pytest plugin stubbing the function to return None, 15 fail and 2 pass. The 2 are the guards that a healthy file and a still-broken file are left alone.
  • python -m pytest tests -q in the synced venv: 11,301 passed, 269 skipped, 1 xfailed, 5 failed. The same 5 fail on a clean v8 on this machine:
    • test_code_only_update_all_hosts.py::test_code_only_update_follows_its_paragraph_and_merges_the_new_function[aider] and [devin]
    • test_language_override_lifecycle.py::test_watch_relative_roots_receive_config_events_without_rebasing_source_paths[src]
    • test_resolve_memoization.py::test_absolute_path_lookup_skips_getcwd
    • test_source_key_memoization.py::test_absolute_path_lookup_skips_getcwd
  • ruff check .: all checks passed.
  • uv run --no-sync pyright: 770 errors on Windows, the same set as on v8 except that v8 also flags the new test's import of the missing function. None is in the changed code.
  • Two private Unity C# codebases, 529 files, extracted with v8 and with this branch: only the file that hit the bug changes. It goes from 11 to 25 symbols and from 8 to 22 methods, and its syntax-error warning is gone. One call edge in another file now resolves, because the broken file no longer leaves a placeholder for a type the call needed. No other node or edge changes, and no label gains an @.

Limitations

  • An occurrence with no adjacent blank cannot be escaped without moving bytes (Use(async), -async, (required.Add(1))). If the file still fails, the rewrite is dropped and the file extracts as today.
  • nint and nuint read as a value (return nint;) are not covered: there they are followed by the same tokens as the type keyword.
  • A lambda parameter named async (async => async + 1) is not covered, since => also follows property names that must keep their spelling.
  • A file that needs both this rewrite and the verbatim-interpolation mask stays as today, because each rewrite applies only when it alone clears the file. That is the existing composition and is unchanged here.
  • In a file that needs a word escaped, every candidate occurrence of that word carries the @, so a field or enum member of that name in the same file is labelled @name. No such label appeared in the two codebases above.
  • The real fix belongs in the grammar: fix: read partial, async and required as identifiers at statement start tree-sitter/tree-sitter-c-sharp#449 reserves partial, async and required as identifiers. Once a release with it is pinned, this rewrite finds nothing to do for those words and returns early.

Material AI assistance: Claude Opus 5.5 helped write the change and the tests; the commit carries a Co-Authored-By trailer.

Graphify-specific checklist

  • I updated generated skill artifacts (uv run python -m tools.skillgen --bless) when changing their source fragments. Not applicable: no skill fragments changed.
  • I confirmed that AST/structural extraction remains deterministic (no ambient state dependencies like ENV variables).
  • I reviewed changes for security implications (no unsafe interpolation into shell/Python).
  • I confirmed no API keys or local-only graph data are included.
  • (If applicable) I disclosed AI authorship in my commit messages.

…phify-Labs#5043)

tree-sitter-c-sharp 0.23.5 reads `partial`, `required`, `async`, `nint`,
`nuint` and `with` as their keywords where C# reads a plain name: a
statement that starts with one (`partial |= x;`, `required.Add(1);`),
`async` in most expressions, `nint = 1;`, `var with = x;`. The statement
becomes an ERROR, and when it is the last statement of an `if` block that
an `else` block follows, the recovery swallows the rest of the file and only
the file node is extracted.

extract_csharp now rewrites such a name to `@name` before parsing, which is
the same identifier to the compiler and a name to the grammar. The `@` takes
the place of an adjacent blank, so the rewrite is byte-length preserving and
every reported line and column still matches the file (Graphify-Labs#2876). It runs only
on a file that fails to parse, escapes every occurrence of a word together so
a local's declaration and its uses keep one spelling (a call on the local
still binds through its type), restores a word the parse does not need, and
is dropped unless the file then parses cleanly.

The AST cache schema moves from 47 to 48 so cached C# extractions re-run.

Fixes Graphify-Labs#5043.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Thanks for the pull request, @ThatAblaze. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.11 and 3.13, and that the change keeps extraction deterministic.

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

No issues found by static checks — no coupling regressions or blocking issues in the code graph. That is not the same as safe to merge: see what was not checked.

Not checked: tests were not run; formal verification proved 0 of 1 changed function(s) (1 not verified).

Formal verification. PR-changed functions: 0/1 verified (0 proven, 0 may-equivalent, 0 distinguished) · 1 not verified (1 vacuous).

Not verified on this run: extract\_csharp (vacuous: never exercised).


Graphify review — findings

Adds a pre-parse pass for C# that escapes contextual-keyword identifiers (a member, type or variable named like a contextual keyword) so tree-sitter-c-sharp no longer misparses them. The aim is that these declarations and calls aren't dropped from the graph. A string-literal scanner keeps the rewrite out of string contents. Bumps the AST cache schema so C# extractions cached before this change are re-run instead of reused, and a new test covers that. New tests also cover the keyword-named extraction itself.

Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 5276 functions depend on the 536 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: extract() — 1259 callers, 78 callees
  • worse: extract_csharp() — 24 callers, 5 callees

Verification — 5276 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 4574 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

366 of 582 test file(s) selected (63%) via static blast radius.

  • tests/test_4874_jvm_resolution.py — impact
  • tests/test_4978_reconcile_cost.py — impact
  • tests/test_4992_ts_import_types.py — impact
  • tests/test_affected_overrides.py — impact
  • tests/test_alias_folding.py — impact
  • tests/test_analyze.py — impact
  • tests/test_astro_extraction.py — impact
  • tests/test_astro_import_ids.py — impact
  • tests/test_binary_media_words.py — impact
  • tests/test_blade_extractor.py — impact
  • tests/test_build.py — impact
  • tests/test_build_located_semantic_identity.py — impact
  • tests/test_build_merge_dedup_scope.py — impact
  • tests/test_build_merge_ghost_fold_4955.py — impact
  • tests/test_build_merge_hyperedges_and_prune.py — impact
  • tests/test_build_merge_shrink_guard.py — impact
  • tests/test_builtin_global_type_refs.py — impact
  • tests/test_c_include_resolution.py — impact
  • tests/test_cache.py — impact
  • tests/test_cache_long_paths.py — impact
  • tests/test_cache_stale_import_target.py — impact
  • tests/test_cache_windows_case_4925.py — impact
  • tests/test_carried_hyperedge_remap.py — impact
  • tests/test_case_only_decl_ids_4904.py — impact
  • tests/test_case_sensitive_resolution.py — impact
  • tests/test_charmap_encoding.py — impact
  • tests/test_chunking.py — impact
  • tests/test_cjs_module_extension.py — impact
  • tests/test_cli_export.py — impact
  • tests/test_cluster.py — impact
  • tests/test_cobol_extractor.py — impact
  • tests/test_code_only_scope_4923.py — impact
  • tests/test_confidence.py — impact
  • tests/test_confidence_score_output.py — impact
  • tests/test_corrupt_graph_json.py — impact
  • tests/test_cpp_alias_member_resolution.py — impact
  • tests/test_cpp_anonymous_namespace_receivers.py — impact
  • tests/test_cpp_conditional_initializers.py — impact
  • tests/test_cpp_forward_declarations.py — impact
  • tests/test_cpp_forward_receiver_completion.py — impact
  • tests/test_cpp_inherited_member_resolution.py — impact
  • tests/test_cpp_local_callable_shadowing.py — impact
  • tests/test_cpp_member_call_by_receiver_type.py — impact
  • tests/test_cpp_member_declaration_identity.py — impact
  • tests/test_cpp_method_declarations.py — impact
  • tests/test_cpp_nested_and_cli.py — impact
  • tests/test_cpp_objc_cross_file_calls.py — impact
  • tests/test_cpp_receiver_chains.py — impact
  • tests/test_cpp_receiver_parameters.py — impact
  • tests/test_cpp_split_header_owner.py — impact
  • … and 316 more

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Formal verification

Could not verify: Could not verify extract\_csharp.

The verifier did not have enough to check extract\_csharp, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: no capturable inputs from the test suite; property tier: not verifiable: all 9 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly NameError — names the real obstacle, not a sampling gap)

· 2 more finding(s) on lines outside this diff (see the check run).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: C#: a local named required / async / partial / nint / nuint / with gives a syntax-parse error from tree-sitter-c-sharp 0.23.5

1 participant