Progress towards tree-sitter feature by saniyafatima07 · Pull Request #3102 · mandiant/capa

saniyafatima07 · 2026-06-10T18:39:59Z

Addresses PR #1080 and #2931

This PR updates capa’s Tree-sitter support for script analysis and fixes the related integration issues so the feature works correctly in both source and packaged builds.

Changes include:

Updated few outdated tree-sitter queries
Ensured that all the CI tests pass
Fixed few import errors
Added Python test scripts
Added PyInstaller packaging support so the Tree-sitter signature JSON files are included in bundled builds.

Checklist

CHANGELOG updated

Few tests added

No documentation update needed currently.

This submission includes AI-generated code and I have provided details in the description.

Parts of this implementation were assisted using AI tools (Codex and ChatGPT).

AI was used for:

writing the test script for python
fixing few errors

All code was reviewed, modified and tested manually before submission.

Testing:
I built it using PyInstaller and tested the capabilities on script inputs. It functions as intended.

gemini-code-assist

Code Review

This pull request introduces Tree-Sitter Script Analysis to capa, enabling feature extraction from script languages such as C#, Python, HTML, and ASPX templates. It adds a new Tree-Sitter-based feature extractor, auto-detection capabilities, and signature-based tools, along with comprehensive tests and updated dependencies. The code review feedback primarily addresses compatibility issues with the upgraded tree-sitter library (version 0.25.0), specifically pointing out that QueryCursor has been removed and the Parser instantiation has changed in tree-sitter versions >= 0.21.0. The feedback provides actionable suggestions to execute queries directly and update parser usage. Additionally, it identifies a bug in integer suffix parsing and recommends replacing a deprecated importlib.resources API.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

… sigs)

…heels

Revives and supersedes old PR mandiant#1080. Resolves merge conflicts and brings up to date with current master.

saniyafatima07 · 2026-06-23T18:44:44Z

@mike-hunhoff @larchchen @Maijin The first version of the script analysis feature is ready for review.
Could you please take a look when you have a chance?

mr-tz

Initial review of the first few files

mr-tz · 2026-06-24T17:22:21Z

+
+def get_language_from_ext(path: str) -> str:
+    if path.endswith(EXT_ASPX):
+        return LANG_TEM


Embedded template is not immediately clear to me, some doc on that would be good (maybe it is further down).

I haven’t added documentation yet, but I’ll add a section describing the script analysis feature in the near future.

mr-tz · 2026-06-24T17:23:17Z

+            buf = f.read()
+        return get_language_ts(buf)
+    except ValueError:
+        return get_language_from_ext(str(path))


I guess the order is debatable here, so flagging for further discussion

My thinking here was to prefer Tree-sitter’s language detection when possible, since it can inspect the file contents rather than relying only on the extension. The extension fallback is mainly for cases where content-based detection fails.

That said, I agree that the order is debatable. Happy to discuss more on this!

What's the performance difference if we switch the order? My concern is that Tree-sitter's language detection could be slow for large or heavily obfuscated scripts, while simply checking the extension won't be bogged down by script size or complexity.

I ran a small benchmark test with simple Python scripts of different file sizes. There is actually a huge difference in the time taken by content-based detection compared to extension-based detection.

Extension-based: 0.0005 ms

Content-based: 7.3845 ms

So, it would be optimal for us to have extension-based detection first. Thank you for pointing it out @mr-tz and @mike-hunhoff.

I will correct it in the next commit.

Maijin · 2026-06-25T08:22:00Z

Kudo for making this huge pr work 👍, I have two high-level comments mainly:

Since automated detection may struggle with heavily obfuscated scripts, it might be beneficial to allow users to force specific language detection on the CLI. This would not only improve user experience but also serve as a useful tool for regression testing. I suggest considering this for the next PR.
Regarding the cs.json, py.json etc. signature files, what is the intended process for updating these as new malicious script patterns emerge? Since these are separated from capa-rules, we should document how rule authors can contribute new signatures. It may also be worth having an architectural discussion regarding the long-term management of these files tagging @mike-hunhoff for visibility on this.

Maijin · 2026-06-25T08:23:01Z

@@ -0,0 +1,95 @@
+{
+    "classes" : [
+        "System.Data.SqlClient.SqlCommand",


Would recommend sorting all of these signatures everywhere. This will ease contributions.

Maijin · 2026-06-25T08:23:13Z

@@ -0,0 +1,47 @@
+{
+    "classes": [
+        "socket.socket",


Same here - sort everywhere

saniyafatima07 · 2026-06-26T06:44:45Z

Kudo for making this huge pr work 👍, I have two high-level comments mainly:

* Since automated detection may struggle with heavily obfuscated scripts, it might be beneficial to allow users to force specific language detection on the CLI. This would not only improve user experience but also serve as a useful tool for regression testing. I suggest considering this for the next PR.

* Regarding the `cs.json`, `py.json` etc. signature files, what is the intended process for updating these as new malicious script patterns emerge? Since these are separated from `capa-rules`, we should document how rule authors can contribute new signatures. It may also be worth having an architectural discussion regarding the long-term management of these files tagging @mike-hunhoff for visibility on this.

Sure, Maxime. We can discuss about the architecture in the upcoming meets.

mr-tz

nice progress overall, thanks for all the work on getting this feature finally integrated :)

mike-hunhoff

Thank you @saniyafatima07 , I know this is a large PR to work with, so thank you for all you've done so far. I've left comments for your review. Mostly it appears that the PR still contains unrelated changes, likely from some rebase conflicts?

mike-hunhoff · 2026-07-16T21:54:56Z

+            buf = f.read()
+        return get_language_ts(buf)
+    except ValueError:
+        return get_language_from_ext(str(path))


What's the performance difference if we switch the order? My concern is that Tree-sitter's language detection could be slow for large or heavily obfuscated scripts, while simply checking the extension won't be bogged down by script size or complexity.

mike-hunhoff · 2026-07-16T22:46:57Z


 ### New Features

+- Tree-Sitter Script Analysis


TODO: expand this description to include contributions from previous developers.

saniyafatima07 · 2026-07-21T21:00:26Z

Thank you @saniyafatima07 , I know this is a large PR to work with, so thank you for all you've done so far. I've left comments for your review. Mostly it appears that the PR still contains unrelated changes, likely from some rebase conflicts?

Thank you for the review, Mike.

I looked into all the comments. It seems like I messed up a few things during the rebase, which caused a lot of unrelated changes.

I will carefully look into all the unrelated changes and resolve them.

gemini-code-assist Bot reviewed Jun 10, 2026

View reviewed changes

saniyafatima07 marked this pull request as ready for review June 10, 2026 18:58

saniyafatima07 marked this pull request as draft June 10, 2026 18:58

saniyafatima07 force-pushed the script-feature branch 2 times, most recently from e71ca37 to 5c19383 Compare June 17, 2026 16:41

mr-tz reviewed Jun 18, 2026

View reviewed changes

Comment thread capa/capabilities/common.py

saniyafatima07 force-pushed the script-feature branch 7 times, most recently from 2e75cd1 to 050ab84 Compare June 22, 2026 20:54

saniyafatima07 and others added 13 commits June 23, 2026 02:32

Fix issues

7bd2acd

fix: address PR review feedback (regex optimization, typos, duplicate…

f8d3848

… sigs)

Update CHANGELOG.md

bcfe22e

Fix outdated code

d86a44e

Fix outdated code

3ebf4bd

Modernize Tree-sitter: Replace local C-compilation with native PyPI w…

9b36aac

…heels

feat: revive script analysis ts feature

ddb701c

Revives and supersedes old PR mandiant#1080. Resolves merge conflicts and brings up to date with current master.

Fix: CI builds

772e55d

Fix import errors

fcca9ad

Fix CI: trial-1

88a98f4

Fix tree-sitter packaging

18342bb

Address valid gemini reviews

14610dc

Minor fixes

8057248

saniyafatima07 force-pushed the script-feature branch 4 times, most recently from 46a6446 to 19308f7 Compare June 22, 2026 22:31

saniyafatima07 requested a review from mike-hunhoff June 23, 2026 14:16

mr-tz reviewed Jun 24, 2026

View reviewed changes

Address comments

652b6cf

Maijin reviewed Jun 25, 2026

View reviewed changes

Sort json files

af4fd99