Skip to content

fix: accept UTF-8 BOM in SKILL.md frontmatter - #99

Open
mimran-khan wants to merge 11 commits into
NVIDIA:mainfrom
mimran-khan:fix/utf8-bom-frontmatter
Open

fix: accept UTF-8 BOM in SKILL.md frontmatter#99
mimran-khan wants to merge 11 commits into
NVIDIA:mainfrom
mimran-khan:fix/utf8-bom-frontmatter

Conversation

@mimran-khan

Copy link
Copy Markdown
Contributor

Summary

A valid SKILL.md that only differs by a UTF-8 BOM failed schema with HIGH frontmatter_format, while the unicode check called that BOM benign.

I read manifests with utf-8-sig (same as plugin JSON) and allow an optional BOM before the opening ---. Fixes #91.

Verification

  • I am familiar with the Contributing Guidelines
  • Added or updated focused tests
  • Updated documentation for user-visible changes
  • Ran make lint
  • Ran make test
  • Ran make build
  • Did not add credentials, private datasets, or proprietary benchmark content

Release Impact

  • Updated CHANGELOG.md

Schema read utf-8 and anchored frontmatter at ^---, so a Notepad BOM
made a valid skill fail HIGH while unicode called the same BOM benign.
Read utf-8-sig and allow an optional BOM before the opening fence.

Fixes NVIDIA#91

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The focused parser/schema suites pass (46 tests), with Ruff and diff checks clean. One Tier 1 consumer still fails to parse the newly accepted BOM form, leaving inconsistent results; details are inline. The shared Gitleaks failure is unrelated branch history.

Comment thread src/skillevaluator/validators/frontmatter_parser.py
Schema already accepted a BOM-prefixed SKILL.md, but quality still
matched ^--- and skipped XML-tag checks. Read utf-8-sig and reuse the
shared FRONTMATTER_PATTERN.

Fixes NVIDIA#91

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
# Conflicts:
#	CHANGELOG.md
@mimran-khan

Copy link
Copy Markdown
Contributor Author

Merged main. BOM handling in the quality parser should be covered now. Ready for re-review.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the current head. BOM-prefixed frontmatter is now handled consistently by schema and quality parsing, so the original quality-gate mismatch is fixed. The focused frontmatter, schema, and quality suite passed (181 tests), with Ruff and diff checks clean. Approved. GitHub currently reports a CHANGELOG-only merge conflict, so the branch still needs an update before it can merge.


try:
content = file_path.read_text(encoding="utf-8")
content = file_path.read_text(encoding="utf-8-sig")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Make security scanning honor BOM-aware frontmatter

I took another final review pass and found one remaining issue that should be fixed before merge. This utf-8-sig read makes BOM-prefixed frontmatter valid here, but _scan_file_for_pii() still reads the raw file with UTF-8 and _frontmatter_author_emails() requires the first raw line to equal ---. In a fresh real-CLI reproduction, byte-identical plain files exit 0 while BOM-prefixed files exit 1 with a HIGH finding for the valid metadata.author email, for both LF and CRLF. Can we make the security path BOM-aware too and add regression coverage for both line endings before we merge?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Frontmatter and quality parsing accept BOM now, but the security scan path still reads the raw first line in _frontmatter_author_emails(). I will wire that through the same BOM-aware read and add LF/CRLF regression coverage next.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at c4e249d. _scan_file_for_pii now reads with utf-8-sig and _is_frontmatter_delimiter treats a BOM-prefixed opener as valid, so metadata.author emails stay exempt. Added LF and CRLF regression coverage in test_security.

@mimran-khan

Copy link
Copy Markdown
Contributor Author

Pushed the security-scan BOM fix at c4e249d. Ready for re-review.

@chrisknvidia

Copy link
Copy Markdown
Collaborator

Thanks for the update. The current DCO check is failing because these two commits do not contain Signed-off-by trailers:

  • c4e249d — fix: make security PII scan honor UTF-8 BOM frontmatter
  • 5ce9931 — docs: note security BOM handling in CHANGELOG

Please rewrite or amend both commits with your own Signed-off-by line, then force-push the updated branch using --force-with-lease. The DCO check and CI can rerun on the new head.

Read SKILL manifests with utf-8-sig and treat BOM-prefixed --- fences as valid
frontmatter so metadata.author emails are exempt in LF and CRLF files.

Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
@mimran-khan
mimran-khan force-pushed the fix/utf8-bom-frontmatter branch from 5ce9931 to 1cf459b Compare September 3, 2026 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: UTF-8 BOM makes a valid SKILL.md fail schema while unicode calls the BOM benign

3 participants