Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,39 @@ inspect-scan https://github.com/python
# Store JSON + HTML under ./storage with standardized names
inspect-scan pallets/flask --storage
inspect-scan psf --storage D:/audit-reports

# Enable/disable parts of the scoring methodology (see below)
inspect-scan pallets/flask --disable-category security --disable-metric popularity
inspect-scan pallets/flask --config scan-config.json --html report.html
```

### Scan configuration (enabling / disabling metrics)

Any part of the methodology — a component, a metric, or a whole category — can
be switched off for a scan. Disabled items are removed from scoring and the
remaining weights **renormalized** (never counted as zero), so scores stay on
the 1–100 scale. The configuration is embedded in the report (`config`) and
summarized in a **Scan configuration** section of the HTML.

```bash
inspect-scan pallets/flask --disable-category security # drop a category
inspect-scan pallets/flask --disable-metric popularity # drop a metric
inspect-scan pallets/flask --disable-component documentation:Wiki # drop a component
inspect-scan pallets/flask --config scan-config.json # from a file (+ flags merge on top)
```

```jsonc
// scan-config.json
{
"disabled_categories": ["security"],
"disabled_metrics": ["popularity"],
"disabled_components": { "documentation": ["Wiki"] }
}
```

Category/metric keys and component names are listed in
[docs/metrics.md](docs/metrics.md#configuration-enabling--disabling-metrics).

### Report storage

`--storage [DIR]` writes both report formats into a standardized layout:
Expand Down
44 changes: 44 additions & 0 deletions docs/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,50 @@ Repository reports also embed the owning account's public profile in
`data.owner` (both organizations and users); it feeds the `stewardship`
metric above.

## Configuration (enabling / disabling metrics)

A scan can switch off any part of the methodology — a **component**, a whole
**metric**, or a whole **category**. This is carried by a `ScanConfig` and
**embedded in every report** (`config` at the top level), so a score is always
reproducible and it is explicit what was and was not measured.

Disabling something works *exactly like missing data*: the item is excluded and
the remaining weights are **renormalized**, so the score stays on the 1..100
scale — it is never counted as a zero. Disabling every metric in a category (or
the category itself) drops that category from the overall, with its weight
renormalized away.

```jsonc
// scan-config.json
{
"disabled_categories": ["security"], // drop a whole category
"disabled_metrics": ["popularity"], // drop one metric
"disabled_components": { // drop components within a metric
"documentation": ["Wiki", "Topics"]
}
}
```

From the CLI, a config file and/or repeatable flags (flags merge on top of the
file):

```
inspect-scan owner/repo --config scan-config.json
inspect-scan owner/repo --disable-category security --disable-metric popularity
inspect-scan owner/repo --disable-component documentation:Wiki --html report.html
```

Category and metric **keys** are the identifiers in the tables above
(`security`, `popularity`, `security_posture`, …); component **names** are the
exact display names (`Wiki`, `Stars`, `README`, …). Unknown category/metric
keys are reported as warnings and ignored. The HTML report renders a **Scan
configuration** section listing what was disabled (or "Full methodology" when
nothing is), and each affected metric's `note` records the renormalization.

Configuration selects *which* parts of the fixed methodology are active; it does
not change any formula, weight, or threshold — those remain versioned by
`metrics_version`.

## Worked example

pallets/flask (organization-owned): Vitality 75, Community & Adoption 96,
Expand Down
25 changes: 22 additions & 3 deletions docs/report-schema.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Report schema

**Schema version: 0.6.0** (`schema_version` field in every report).
**Schema version: 0.7.0** (`schema_version` field in every report).
The schema is defined as Pydantic models in
[`src/scanner/models.py`](../src/scanner/models.py); this document describes
it for consumers. Any breaking structural change bumps `schema_version`.
Expand Down Expand Up @@ -30,9 +30,10 @@ data/metrics layering, the `Metric` object shape, and the band scale.
```jsonc
{
"report_type": "repository",
"schema_version": "0.6.0",
"schema_version": "0.7.0",
"generated_at": "2026-07-06T12:00:00Z", // UTC timestamp of the scan
"source": { ... }, // what was scanned
"config": { ... }, // scan configuration (see below)
"data": { ... }, // raw facts (data layer)
"metrics": { ... }, // 1..100 scores (metrics layer)
"warnings": ["..."] // non-fatal collection problems
Expand All @@ -44,6 +45,23 @@ computing statistics, truncated file trees). A warning means the related
fields are `null`/incomplete — affected metrics exclude those inputs rather
than scoring them as zero.

## `config` — scan configuration

Records which parts of the methodology were active for this scan, so the score
is reproducible. Empty collections mean the full methodology (everything
enabled). Disabled items are removed from scoring with the remaining weights
renormalized (see [metrics.md](metrics.md#configuration-enabling--disabling-metrics)).

```jsonc
"config": {
"disabled_categories": ["security"], // category keys
"disabled_metrics": ["popularity"], // metric keys
"disabled_components": { // metric key -> component names
"documentation": ["Wiki"]
}
}
```

## `source`

| Field | Type | Description |
Expand Down Expand Up @@ -245,9 +263,10 @@ Produced when the scan target is an organization (`inspect-scan orgname`).
```jsonc
{
"report_type": "organization",
"schema_version": "0.6.0",
"schema_version": "0.7.0",
"generated_at": "...",
"source": { "url": "...", "host": "github.com", "login": "psf" },
"config": { /* same ScanConfig shape as repository reports */ },
"data": {
"info": { /* OrgInfo: login, name, description, blog, location, email,
twitter_username, is_verified, public_repos, followers,
Expand Down
97 changes: 94 additions & 3 deletions src/scanner/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@

from .collect import scan_organization, scan_repository
from .github import GitHubError, parse_target, resolve_token
from .models import OrgReport, Report
from .metrics import validate_config
from .models import OrgReport, Report, ScanConfig
from .render import render_html, render_org_html
from .storage import STORAGE_ENV_VAR, resolve_storage, store_report

Expand Down Expand Up @@ -69,9 +70,88 @@ def build_parser() -> argparse.ArgumentParser:
action="store_true",
help="Emit compact single-line JSON instead of pretty-printed",
)

config_group = parser.add_argument_group(
"scan configuration",
"Enable/disable parts of the scoring methodology. Disabled items are "
"removed from scoring and the remaining weights renormalized (never "
"counted as zero). The resulting configuration is embedded in the report.",
)
config_group.add_argument(
"--config",
type=Path,
default=None,
metavar="PATH",
help="JSON file with a scan configuration "
'({"disabled_categories": [...], "disabled_metrics": [...], '
'"disabled_components": {"metric_key": ["Component name"]}}). '
"Command-line --disable-* flags are merged on top of it.",
)
config_group.add_argument(
"--disable-category",
action="append",
default=[],
metavar="KEY",
help="Exclude a whole category from scoring (repeatable), e.g. security",
)
config_group.add_argument(
"--disable-metric",
action="append",
default=[],
metavar="KEY",
help="Exclude a single metric from scoring (repeatable), e.g. popularity",
)
config_group.add_argument(
"--disable-component",
action="append",
default=[],
metavar="METRIC:COMPONENT",
help="Exclude one component within a metric (repeatable), e.g. "
"popularity:Watchers",
)
return parser


def build_config(args: argparse.Namespace) -> ScanConfig:
"""Assemble a ScanConfig from an optional --config file plus --disable-* flags."""
config = ScanConfig()
if args.config is not None:
config = ScanConfig.model_validate_json(args.config.read_text(encoding="utf-8"))

categories = list(config.disabled_categories)
for key in args.disable_category:
if key not in categories:
categories.append(key)

metrics = list(config.disabled_metrics)
for key in args.disable_metric:
if key not in metrics:
metrics.append(key)

components = {k: list(v) for k, v in config.disabled_components.items()}
for spec in args.disable_component:
metric_key, sep, component = spec.partition(":")
if not sep or not metric_key.strip() or not component.strip():
raise ValueError(
f"--disable-component expects METRIC:COMPONENT, got {spec!r}"
)
names = components.setdefault(metric_key.strip(), [])
if component.strip() not in names:
names.append(component.strip())

return ScanConfig(
disabled_categories=categories,
disabled_metrics=metrics,
disabled_components=components,
)


def _config_requested(args: argparse.Namespace) -> bool:
return bool(
args.config or args.disable_category or args.disable_metric or args.disable_component
)


def _load_report_file(path: Path) -> AnyReport:
raw = path.read_text(encoding="utf-8")
kind = json.loads(raw).get("report_type", "repository")
Expand All @@ -82,8 +162,19 @@ def _load_report_file(path: Path) -> AnyReport:
def _load_or_scan(args: argparse.Namespace) -> AnyReport:
target_path = Path(args.target)
if args.target.lower().endswith(".json") and target_path.is_file():
if _config_requested(args):
print(
"note: scan-configuration flags are ignored when re-rendering a "
"stored JSON report; the report keeps the configuration it was "
"scanned with",
file=sys.stderr,
)
return _load_report_file(target_path)

config = build_config(args)
for warning in validate_config(config):
print(f"warning: {warning}", file=sys.stderr)

kind, *ids = parse_target(args.target)
token = resolve_token(args.token)
if not token:
Expand All @@ -93,8 +184,8 @@ def _load_or_scan(args: argparse.Namespace) -> AnyReport:
file=sys.stderr,
)
if kind == "org":
return scan_organization(ids[0], token=token)
return scan_repository(args.target, token=token)
return scan_organization(ids[0], token=token, config=config)
return scan_repository(args.target, token=token, config=config)


def _render(report: AnyReport) -> str:
Expand Down
27 changes: 21 additions & 6 deletions src/scanner/collect.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
Report,
RepoInfo,
RepoRef,
ScanConfig,
SecuritySignals,
TopRepo,
)
Expand Down Expand Up @@ -100,8 +101,14 @@
TEST_FILE_PATTERNS = ("test_*.py", "*_test.py", "*_test.go", "*.test.js", "*.test.ts", "*.spec.js", "*.spec.ts", "*Test.java", "*Test.php")


def scan_repository(url: str, token: Optional[str] = None) -> Report:
"""Scan a public GitHub repository and return a populated Report."""
def scan_repository(
url: str, token: Optional[str] = None, config: Optional[ScanConfig] = None
) -> Report:
"""Scan a public GitHub repository and return a populated Report.

``config`` selects which metrics/categories/components are scored; it
defaults to the full methodology and is embedded in the returned Report."""
config = config or ScanConfig()
owner, name = parse_repo_url(url)
source = RepoRef(url=url, owner=owner, name=name)
warnings: list[str] = []
Expand Down Expand Up @@ -137,8 +144,9 @@ def scan_repository(url: str, token: Optional[str] = None) -> Report:
return Report(
generated_at=datetime.now(timezone.utc),
source=source,
config=config,
data=data,
metrics=compute_metrics(data),
metrics=compute_metrics(data, config),
warnings=warnings,
)

Expand Down Expand Up @@ -399,8 +407,14 @@ def _security(paths: list[str], community: CommunityHealth) -> SecuritySignals:
# ---------------------------------------------------------------------------


def scan_organization(login: str, token: Optional[str] = None) -> OrgReport:
"""Scan a GitHub organization's public profile and repository portfolio."""
def scan_organization(
login: str, token: Optional[str] = None, config: Optional[ScanConfig] = None
) -> OrgReport:
"""Scan a GitHub organization's public profile and repository portfolio.

``config`` selects which metrics/categories/components are scored; it
defaults to the full methodology and is embedded in the returned report."""
config = config or ScanConfig()
warnings: list[str] = []
with GitHubClient(token=token) as gh:
try:
Expand All @@ -421,8 +435,9 @@ def scan_organization(login: str, token: Optional[str] = None) -> OrgReport:
return OrgReport(
generated_at=datetime.now(timezone.utc),
source=OrgRef(url=f"https://github.com/{login}", login=data.info.login),
config=config,
data=data,
metrics=compute_org_metrics(data),
metrics=compute_org_metrics(data, config),
warnings=warnings,
)

Expand Down
Loading