Add iocs.files[] to database_specific for file indicators (per #1345 discussion) by akyroslabs · Pull Request #1394 · ossf/malicious-packages

akyroslabs · 2026-07-21T08:42:20Z

This implements the structured file-IoC shape @calebbrown proposed in the review of #1345, so that file indicators (hashes etc.) have a documented home — today iocs only defines domains/ips/urls, and file hashes have nowhere schema-defined to go.

iocs.files[] records each file as { path, note, source, digests:{sha256,tlsh} }. The source enum (package-archive / downloaded / generated) keeps package artifacts separate from files dropped or generated at run time, as suggested. path is optional when a file is identified only by digest; each entry must carry a path or at least one digest.

Includes the Indicators struct + validation (source enum, SHA-256/TLSH format, length limits), unit tests, and docs/schema_additions.md.

Flagged as a proposal — happy to adjust the shape (e.g. digest algorithms, required fields) per contributor consensus, as noted in #1345.

kam193 · 2026-07-21T20:21:01Z

+#### iocs.files[].digests field
+
+The `digests` field is an optional JSON object mapping a hash algorithm name to
+the file's digest under that algorithm. Recognised algorithms are:


Maybe let's extend it to hashes recognized by VirustTotal (MD5, SHA1, SHA256, TLSH, SSDEEP) and add that it's prefered to add at least SHA256?

Agreed. "md5", "sha1", "sha256", "tlsh", "ssdeep" should all be acceptable. "sha256" is preferred and recommended at a minimum.

kam193 · 2026-07-21T20:23:47Z

+
+- `package-archive` — the file is part of the published package artifact.
+- `downloaded` — the file was retrieved at run time (e.g. from a C2).
+- `generated` — the file was produced at run time from code in the archive or


Maybe dropped instead of the generated, and add memory as an option for memory-only artifacts? Though I'm not sure if we want to include memory artifacts under files

I also wonder if it's better to keep the list closed or open. No hard opinion.

I'd prefer to not have unconstrained entries, or at least have a short conversation if a new value was desired.

As for "dropped" instead of "generated" how would that differ with "downloaded", as these are technically dropped as well?

I'd also be okay with just having "package-archive", "dropped" and perhaps "in-memory" as well to keep it simpler.

Yeah, my thoughts were also going in the similar direction, so I'd be for the simplified version with dropped and in-memory. Additional information could be hold in the note

kam193 · 2026-07-21T20:25:09Z

+
+#### iocs.files[].path field
+
+The `path` field is an optional string giving the file name as observed, either


I think I'd suggest renaming it as paths and typing to the list - the same file can exist in multiple places.

This sounds reasonable.

kam193 · 2026-07-21T20:29:24Z

+- `sha256` — a lowercase hex-encoded SHA-256 digest (64 hex characters).
+- `tlsh` — a hex-encoded [TLSH](https://tlsh.org/) fuzzy hash (70 hex


I'd suggest accepting any valid form of the hash (lower/upper/mixedcase for all, prefixed or not for TLSH) but performing normalization in the CI.

My preference is that they are all hex-encoded lowercase. Although I am okay if they are mixed or uppercase as that is an easy conversion.

The CI can transform them to lowercase hex as needed.

kam193 · 2026-07-21T20:31:37Z

+Files are recorded separately from the network indicators above, and each entry
+carries its own `source` so that consumers can distinguish package artifacts
+(the published tarball/wheel, which may only ever be processed as a stream) from
+files that are dropped or generated at run time (for example a second-stage
+payload downloaded from a C2, or content decoded from data embedded in the
+archive). Keeping these apart avoids mixing, e.g., ten package tarballs and two
+dropped files into a single flat list.


I think this paragraph could be omited.

The paragraph could definitely be improved, although I am still keen to keep package artifacts (i.e. the archives downloaded from registries) separate from other files associated with the malware.

My idea was that this is explained later when describing the source field - or did I not understand something?

calebbrown

cc @awsactran @KunalSin9h

It would be great to get your input here as well. Thanks!

calebbrown · 2026-07-22T04:19:22Z

+Files are recorded separately from the network indicators above, and each entry
+carries its own `source` so that consumers can distinguish package artifacts
+(the published tarball/wheel, which may only ever be processed as a stream) from
+files that are dropped or generated at run time (for example a second-stage
+payload downloaded from a C2, or content decoded from data embedded in the
+archive). Keeping these apart avoids mixing, e.g., ten package tarballs and two
+dropped files into a single flat list.


The paragraph could definitely be improved, although I am still keen to keep package artifacts (i.e. the archives downloaded from registries) separate from other files associated with the malware.

calebbrown · 2026-07-22T04:20:55Z

+				"path": string,
+				"note": string,
+				"source": string,
+				"digests": { "sha256": string, "tlsh": string }


The digests object should be reformatted to be spread across multiple lines for legibility.

calebbrown · 2026-07-22T04:23:56Z

+
+- `package-archive` — the file is part of the published package artifact.
+- `downloaded` — the file was retrieved at run time (e.g. from a C2).
+- `generated` — the file was produced at run time from code in the archive or


I'd prefer to not have unconstrained entries, or at least have a short conversation if a new value was desired.

As for "dropped" instead of "generated" how would that differ with "downloaded", as these are technically dropped as well?

I'd also be okay with just having "package-archive", "dropped" and perhaps "in-memory" as well to keep it simpler.

@kam193

…e digests) Per review from @kam193 and @calebbrown on ossf#1394: - path -> paths (a list; the same file can appear in several places) - simplify the source enum to package-archive | dropped | in-memory (downloaded/generated collapse into dropped; in-memory for memory-only artifacts) - digests: accept md5, sha1, sha256, tlsh, ssdeep (sha256 preferred/min recommended); hex digests validated case-insensitively and normalized to lowercase, ssdeep left as-is - multi-line the digests object in the overview; tighten the files[] intro paragraph - an entry must have at least one path or one digest Signed-off-by: Sascha Klein <akyroslabs@gmail.com>

akyroslabs · 2026-07-22T08:14:09Z

Thanks @kam193 and @calebbrown — pushed a commit addressing all points:

path → paths (a list; the same file can appear in several places)
simplified source to package-archive / dropped / in-memory (downloaded/generated collapse into dropped; in-memory for memory-only artifacts, extra detail in note)
digests now accepts md5, sha1, sha256, tlsh, ssdeep (sha256 preferred and recommended at a minimum); hex digests are validated case-insensitively and normalized to lowercase, ssdeep is left as-is
multi-lined the digests object in the overview block and tightened the files[] intro paragraph
an entry now requires at least one paths entry or one digest

Happy to keep iterating if anything else needs adjusting.

akyroslabs · 2026-07-22T08:39:15Z

One related thing while we finalize this: we also have several confirmed, novel malware reports open that all happen to carry a database_specific.iocs block — #1347, #1364, #1365, #1368, #1382, #1389, #1392, #1393 (all CI-green, DCO-signed, not in OSV/GHSA at detection).

If they're waiting on the iocs schema landing here, that's completely understandable — but since they're confirmed malware, I'd rather the reports themselves not be blocked on the schema discussion. Happy to do whatever unblocks them on your end:

update the iocs to the agreed iocs.files[] shape once Add iocs.files[] to database_specific for file indicators (per #1345 discussion) #1394 lands,
adjust it now to whatever you prefer, or
simply drop the iocs block so the plain reports can merge, and re-add indicators later.

Just let me know what's easiest — mainly want to make sure they don't fall through the cracks. Thanks!

KunalSin9h · 2026-07-24T06:21:05Z

@calebbrown loos good, hashes and path are paramount here.

calebbrown · 2026-07-24T06:26:15Z

update the iocs to the agreed iocs.files[] shape once Add iocs.files[] to database_specific for file indicators (per #1345 discussion) #1394 lands,

adjust it now to whatever you prefer, or

simply drop the iocs block so the plain reports can merge, and re-add indicators later.

As for your other reports I am happy to leave them as just "hashes" for now, with a view to replace them with files later.

calebbrown · 2026-07-24T06:29:06Z

I'm generally happy with this change, but I need to review the changes in more detail. I will have more comments on Monday.

The code changes and overall spec are the most important to get right, we can always clean up the other wording later.

I would also like a quick review from @awsactran too.

awsactran · 2026-07-24T06:59:17Z

Sorry for the late response as I am currently OOO. Overall I think this is a solid, well-scoped proposal. I do like that it stays inside database_specific rather than trying to alter OSV proper.

calebbrown

Only some minor comments, but otherwise this looks good. Thankyou!

For reference I am intending within the next week or so to tighten the validation and make it possible to also mutate the DatabaseSpecific fields (this is to clean up CWEs, Indicators and other things).

Also, a structure for listing package artifacts themselves separately to IOCs will need proposed to support that usage.

calebbrown · 2026-07-31T03:31:42Z

+		return fmt.Errorf("%w file note too long (%d > %d)", ErrUnexpectedOSV, len(f.Note), maxNoteLength)
+	}
+	d := f.Digests
+	hasDigest := d != nil && (d.MD5 != "" || d.SHA1 != "" || d.SHA256 != "" || d.TLSH != "" || d.SSDEEP != "")


I think the requirement should be for either MD5, SHA1 or SHA256 to be required. TLSH and SSDEEP are for content similarity checks, so they should not be one of the required.

calebbrown · 2026-07-31T03:34:21Z

+		return nil
+	}
+	var err error
+	if d.MD5, err = normHex("md5", d.MD5, md5RE); err != nil {


It is worth noting that "normHex" will not be persisted at the moment. The indicators struct (inside DatabaseSpecific) is only currently used for validation.

That is "okay" at the moment, but the normalization can be done later. It may improve clarity to remove the mutation for now.

calebbrown · 2026-07-31T03:35:17Z

+}
+
+// normHex validates a hex digest (case-insensitive) and returns it lowercased.
+func normHex(name, val string, re *regexp.Regexp) (string, error) {


This is function is doing two things. My preference is for the validation to be independent of the ToLower() as it makes it simpler to read and understand.

calebbrown · 2026-07-31T03:37:34Z

+The `source` field is an optional string recording where the file came from. It
+must be one of:
+
+- `package-archive` — the file is part of the published package artifact.


This should probably be reworded as:

- the file is extracted from the published package artifact

This eliminates any confusion about it potentially including the artifact itself.

calebbrown · 2026-07-31T03:40:31Z

+Each entry carries a `source` so that package artifacts (the published
+tarball/wheel, which may only ever be processed as a stream) are kept separate
+from other files associated with the malware — a second stage dropped at run
+time, or a payload that only ever lived in memory.


I think this paragraph could be improved. It should make it clearer what it is for, and be obvious that the package artifacts themselves do not belong in this set.

calebbrown · 2026-07-31T03:45:28Z

+- `package-archive` — the file is part of the published package artifact.
+- `dropped` — the file was written to disk at run time (e.g. a second stage
+  retrieved from a C2, or content decoded from data embedded in the archive).
+- `in-memory` — the file only ever existed in memory and never hit disk.


Can these enums be

"PACKAGE_ARCHIVE"

"DROPPED"

"IN_MEMORY"

This keeps the enums consistent with the OSV enums, and allows the move to protojson based parsing of DatabaseSpecific data as well.

akyroslabs · 2026-07-31T09:18:12Z

Thanks @calebbrown — addressed the review:

Enums renamed to OSV style: PACKAGE_ARCHIVE / DROPPED / IN_MEMORY (in indicators.go, the tests, and the docs), for consistency with the OSV enums and the move to protojson-based database_specific parsing.
Validation decoupled from normalization: normHex is now validateHex (a pure check, no mutation); the lowercasing of the hex digests happens as a separate step in validateDigests.
Docs: reworded PACKAGE_ARCHIVE to "the file is extracted from the published package artifact", and rewrote the intro paragraph so it's explicit that the published artifact itself does not belong in this set (it's already identified by affected.package; this field is for the individual files of interest).

Makes sense re: tightening validation, mutable DatabaseSpecific, and a separate structure for listing package artifacts — happy to help on those follow-ups.

Implements the structured file-IoC shape proposed in the review of ossf#1345: iocs.files[] records each file as {path, note, source, digests:{sha256,tlsh}}. The source enum (package-archive|downloaded|generated) keeps package artifacts separate from files dropped or generated at run time. path is optional when a file is identified only by a digest; each entry must carry a path or a digest. Adds the Indicators.Files field + validation (source enum, SHA-256/TLSH format, length limits), unit tests, and docs/schema_additions.md. Signed-off-by: Sascha Klein <akyroslabs@gmail.com>

@kam193

…e digests) Per review from @kam193 and @calebbrown on ossf#1394: - path -> paths (a list; the same file can appear in several places) - simplify the source enum to package-archive | dropped | in-memory (downloaded/generated collapse into dropped; in-memory for memory-only artifacts) - digests: accept md5, sha1, sha256, tlsh, ssdeep (sha256 preferred/min recommended); hex digests validated case-insensitively and normalized to lowercase, ssdeep left as-is - multi-line the digests object in the overview; tighten the files[] intro paragraph - an entry must have at least one path or one digest Signed-off-by: Sascha Klein <akyroslabs@gmail.com>

…nestif) The expanded digest handling pushed Indicators.UnmarshalJSON over the gocyclo threshold (34 > 30) and tripped nestif. Extract validateFile / validateDigests / normHex so UnmarshalJSON stays simple; behaviour unchanged, tests still pass. Signed-off-by: Sascha Klein <akyroslabs@gmail.com>

… normalization; clarify docs - Rename file-source enums to OSV style: PACKAGE_ARCHIVE / DROPPED / IN_MEMORY - Decouple hex validation from lowercasing (normHex -> validateHex; normalize in validateDigests) - Docs: PACKAGE_ARCHIVE = "extracted from the published package artifact"; clarify the artifact itself is not part of this set Signed-off-by: Sascha Klein <akyroslabs@gmail.com>

kam193 reviewed Jul 21, 2026

View reviewed changes

calebbrown reviewed Jul 22, 2026

View reviewed changes

calebbrown mentioned this pull request Jul 28, 2026

Malicious package: npm/fmt-date-lite (date-utility download-execute dropper) #1345

Merged

calebbrown requested changes Jul 31, 2026

View reviewed changes

calebbrown reviewed Jul 31, 2026

View reviewed changes

akyroslabs force-pushed the proposal/iocs-files branch from efcdb12 to 72c9a9e Compare July 31, 2026 09:22

akyroslabs added 4 commits August 3, 2026 10:55

akyroslabs force-pushed the proposal/iocs-files branch from 72c9a9e to 72feaac Compare August 3, 2026 09:02

calebbrown approved these changes Aug 4, 2026

View reviewed changes

Merge branch 'main' into proposal/iocs-files

809338a

calebbrown merged commit b1f7a00 into ossf:main Aug 4, 2026
10 checks passed


		#### iocs.files[].path field

		The `path` field is an optional string giving the file name as observed, either

		- `sha256` — a lowercase hex-encoded SHA-256 digest (64 hex characters).
		- `tlsh` — a hex-encoded [TLSH](https://tlsh.org/) fuzzy hash (70 hex

Uh oh!

Conversation

akyroslabs commented Jul 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

calebbrown left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akyroslabs commented Jul 22, 2026

Uh oh!

akyroslabs commented Jul 22, 2026

Uh oh!

KunalSin9h commented Jul 24, 2026

Uh oh!

calebbrown commented Jul 24, 2026

Uh oh!

calebbrown commented Jul 24, 2026

Uh oh!

awsactran commented Jul 24, 2026

Uh oh!

calebbrown left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akyroslabs commented Jul 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants