Skip to content

FIX Ignore Perl version declarations in PackageHallucinationScorer - #3481

Merged
jbolor21 merged 2 commits into
microsoft:mainfrom
ShivamSharma43:fix/perl-version-declarations
Oct 23, 2026
Merged

jbolor21 merged 2 commits into
microsoft:mainfrom
ShivamSharma43:fix/perl-version-declarations

Conversation

@ShivamSharma43

Copy link
Copy Markdown
Contributor

Description

Fixes #3480.

PackageHallucinationScorer with PackageEcosystem.PERL matched use statements with (?:{3}|^)use\s+([A-Za-z0-9_:]+)\b. That pattern stops at the dot, so the version declarations idiomatic Perl starts with, use 5.010;anduse v5.36;, were captured as 5andv5. Neither is in any registry, so a correct Perl answer scored Trueand thePackageHallucinationscenario'sperl` technique counted it as a successful attack.

Changes:

  • _extract_package_references drops Perl references that are only a version, matching v?\d[\d_]* in full: 5, 5_010 and v5. This is what the pattern captures from use 5;, use 5.010_001;, use 5.36.0;, use v5; and use v5.36;.
  • The pattern is a class constant next to the extraction patterns. The Raku branch already drops use v6.d; the same way.

Module names are unaffected, including ones that contain digits such as Digest::SHA1, and the other ecosystems are unchanged.

Tests and Documentation

  • New tests in tests/unit/score/test_package_hallucination_scorer.py:
    • use 5;, use 5.010;, use 5.010_001;, use 5.36.0;, use v5;, use v5.36;, and a declaration right after a code fence are not extracted.
    • Digest::SHA1 and Fake::Module2 are still extracted next to a version declaration.
    • A fenced Perl answer with use 5.010;, use v5.36; and a known module scores False with no hallucinated packages.
  • On main all 9 new tests fail (5 or v5 is extracted and the answer scores True); with this change they pass.
  • pytest tests/unit/score tests/unit/scenario: 6664 passed, 122 skipped.
  • ruff check, ruff format and ty check pass on the changed files.
  • No documentation changes.

🤖 Generated with Claude Code

The Perl pattern captured 5 from use 5.010; and v5 from use v5.36;,
so a Perl answer that declared its required Perl version was flagged
as referencing a hallucinated package. Version declarations are now
dropped, as the Raku branch already does for use v6.d;.

Fixes microsoft#3480

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 23, 2026 11:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@jbolor21 jbolor21 self-assigned this Oct 23, 2026
@jbolor21
jbolor21 requested a balanced review from Copilot October 23, 2026 22:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The narrowly scoped filter addresses the reported false positives, includes focused regression coverage, and has no unresolved findings.

No open issues 🎉

Review effort: Balanced

@jbolor21
jbolor21 added this pull request to the merge queue Oct 23, 2026
Merged via the queue into microsoft:main with commit a380d44 Oct 23, 2026
97 of 100 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: PackageHallucinationScorer reports Perl version declarations such as use v5.36 as hallucinated packages

3 participants