Skip to content

Add remote plugin install, index discovery, and plugin dependencies#1422

Draft
ashtom wants to merge 8 commits into
mainfrom
ashtom/plugin-index
Draft

Add remote plugin install, index discovery, and plugin dependencies#1422
ashtom wants to merge 8 commits into
mainfrom
ashtom/plugin-index

Conversation

@ashtom

@ashtom ashtom commented Jun 12, 2026

Copy link
Copy Markdown
Member

Summary

Expands the kubectl-style external-command layer so users can discover and install plugins without cloning anything, from any git host, with dependencies between plugins (e.g. entire-brain needs entire-sem).

  • Remote installentire plugin install <name|url|path>. Newest semver tag via git ls-remote (forge-agnostic, inherits git auth/proxies; no forge REST client anywhere). Optional entire-plugin.yml metadata read via blobless shallow clone. Release assets downloaded through a small per-host URL convention table (GitHub/Gitea-style, GitLab-style, download_url template escape hatch), selected and verified via the release's checksums.txt when published, with goreleaser-convention probing as fallback. Binaries land in pkg/<name>/ with a provenance manifest.yml; bin/ stays the only dispatch surface and the resolver in plugin.go is untouched.
  • Upgradeentire plugin upgrade [name|--all], --pin to hold a tag, next-highest-tag fallback for pushed-but-unpublished releases, Windows locked-binary rename-aside.
  • Discovery — krew-style git-synced index (entireio/plugin-index): shallow-cloned into the user cache keyed by URL hash, TTL refresh (plugins.index_ttl_hours, default 24h), stale-copy-on-offline. New search/info/browse/index update subcommands; bare-name installs resolve through the index; non-index URLs need TTY confirmation or --yes. Index URL precedence: --index > ENTIRE_PLUGIN_INDEX_URL > plugins.index_url settings > built-in default — the repo-level setting lets a company commit an internal catalog.
  • Dependenciesentire-plugin.yml requires (name, repo_url, min_version only, no ranges). Install-time transitive planning (metadata-only, cycle-bounded), apt-style single confirmation, remove guard for depended-on plugins, and entire plugin doctor (missing/outdated deps, manifest drift, dangling symlinks, macOS quarantine). Dispatch stays zero-cost — no runtime dependency checks.

Design notes

  • Built on the recorded founding decisions of this layer: PATH dispatch unchanged, built-ins win, agent- prefix reserved, env filtering untouched, no new telemetry (official-allowlist posture unchanged).
  • The only forge-specific code is the download-URL convention table; version listing, metadata, and the index all ride on the git protocol. No go-github-class dependency added (uses existing gopkg.in/yaml.v3 + golang.org/x/mod).

Testing

  • ~40 unit tests against file:// git repos and httptest asset servers (zero real network in CI), covering tag resolution, asset selection/verification, extraction guards, index sync/TTL/offline, dependency planning, doctor.
  • 4 integration tests drive the spawned binary end-to-end: bare-name install → real dispatch of the downloaded plugin → dependency install → remove guard → doctor exit code.
  • mise run check green (fmt, lint, unit + integration + Vogon canary).
  • Validated against production: entire-run and entire-upgrade v0.1.0 published via goreleaser; a branch build with no overrides synced the real index and installed/dispatched entire-upgrade from its GitHub release.

Docs

  • docs/architecture/external-commands.md — new Remote install / Plugin index / Dependencies sections + key-file map.
  • README — user-facing Plugins section.
  • entireio/plugin-index README — schema and contribution guide.

🤖 Generated with Claude Code


Note

High Risk
The CLI now downloads, verifies, and executes third-party release binaries from user-supplied or index-listed git repos—a supply-chain surface that depends on checksum verification, trust prompts, and safe archive extraction behaving correctly.

Overview
Adds end-to-end plugin lifecycle beyond local symlink installs: discover plugins from a git-synced index, install from index names or arbitrary git repo URLs, upgrade pinned/remote installs, and manage transitive entire-plugin.yml dependencies with remove guards and plugin doctor.

Remote install resolves the newest semver tag via git ls-remote, optionally reads repo metadata with a shallow clone, downloads release assets (forge URL heuristics + optional download_url template), verifies checksums.txt when present, extracts from tar/zip with path guards, stores binaries under pkg/<name>/ with manifest.yml provenance, and links into the existing managed bin/ surface so entire <name> dispatch stays unchanged.

Discovery shallow-clones index.json into the user cache (TTL + offline stale fallback); new commands include search, info, browse, and index update. Untrusted repo URLs require confirmation or --yes; index URL comes from --index, ENTIRE_PLUGIN_INDEX_URL, or plugins.index_url in settings.

Dependencies are planned and installed at install time (optional --no-deps); plugin upgrade, --pin, and integration tests cover the full install → dispatch → dependency → doctor path.

Reviewed by Cursor Bugbot for commit 813670a. Configure here.

ashtom and others added 3 commits June 12, 2026 13:36
Typed PluginSettings sub-struct on EntireSettings, whole-object merge in
the local-override path (parallel to investigate), and post-merge
validation. Settings configure discovery only; plugin state (installed
versions, pins) lives in the managed dir's manifests. Index URL
precedence is resolved in the cli package: --index flag >
ENTIRE_PLUGIN_INDEX_URL > settings > built-in default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entire-Checkpoint: b1d40d91c483
Expands the kubectl-style external-command layer so users can discover
and install plugins without cloning anything, on any git host:

- Remote install (entire plugin install <url>|<name>): newest semver tag
  via git ls-remote (forge-agnostic, inherits git auth), optional
  entire-plugin.yml metadata via blobless shallow clone, release asset
  download through a per-host URL convention table (GitHub/Gitea-style,
  GitLab-style, download_url template escape hatch), checksums.txt
  verification, tar.gz/zip/raw extraction with traversal guards, and a
  pkg/<name>/manifest.yml provenance record. bin/ stays the only
  dispatch surface; the resolver in plugin.go is untouched.
- Upgrade (entire plugin upgrade [name|--all]) with --pin skip and a
  next-highest-tag fallback for pushed-but-unpublished releases.
- Discovery: krew-style git-synced index (index.json in a git repo,
  shallow-cloned into the user cache keyed by URL hash, TTL refresh,
  stale-on-offline). New search/info/browse/index update subcommands;
  bare-name installs resolve through the index; non-index URLs require
  TTY confirmation or --yes.
- Dependencies: entire-plugin.yml requires (name, repo_url,
  min_version), install-time transitive planning with cycle bounds,
  apt-style confirmation, a remove guard for depended-on plugins, and
  entire plugin doctor (missing/outdated deps, manifest drift, dangling
  symlinks, macOS quarantine).

Unit tests run against file:// repos and httptest asset servers;
integration tests exercise the spawned binary end to end including
dispatch of an installed plugin. No forge REST API anywhere — version
listing, metadata, and the index all ride on the git protocol.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entire-Checkpoint: 012eb5e80390
Plugins section covering install sources, discovery via the plugin
index, dependencies, and the corporate index_url override; plugin row
in the commands reference.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entire-Checkpoint: aa8b54a63324
Copilot AI review requested due to automatic review settings June 12, 2026 05:21
Comment thread cmd/entire/cli/plugin_group.go Outdated
Comment thread cmd/entire/cli/plugin_group.go Outdated
Comment thread cmd/entire/cli/plugin_group.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Expands the existing kubectl-style external-command layer to support a full plugin lifecycle: remote installs/upgrades (via git tag resolution + release asset download), discovery via a git-synced plugin index, and install-time dependency planning with a plugin doctor health check.

Changes:

  • Add remote plugin install/upgrade plumbing (git ls-remote tag resolution, optional entire-plugin.yml metadata fetch, release asset download + extraction, provenance manifests).
  • Add plugin index sync/search/info/browse/index-update flows with settings/env/flag precedence and TTL-based refresh.
  • Add dependency planning/execution + remove guard + plugin doctor, plus docs/README updates and new unit/integration tests.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
README.md Adds user-facing “Plugins” section and command examples.
docs/architecture/external-commands.md Documents remote install, index discovery, and dependency semantics.
cmd/entire/cli/settings/settings.go Adds repo-level plugin discovery settings (index URL + TTL) and validation/merge support.
cmd/entire/cli/settings/settings_plugins_test.go Unit tests for plugin settings validation/TTL and merge semantics.
cmd/entire/cli/plugin_manifest.go Introduces managed-install manifest + author metadata parsing (entire-plugin.yml).
cmd/entire/cli/plugin_manifest_test.go Tests manifest I/O and strict metadata parsing behavior.
cmd/entire/cli/plugin_install_remote.go Implements remote install orchestration, manifest writing, and upgrade path.
cmd/entire/cli/plugin_install_remote_test.go Unit tests for remote install, pinning, fallback, upgrade, and removal cleanup.
cmd/entire/cli/plugin_index.go Implements git-synced plugin index cache with TTL refresh and filtering.
cmd/entire/cli/plugin_index_test.go Tests index sync/refresh/offline behavior and install-arg classification.
cmd/entire/cli/plugin_group.go Adds Cobra commands/flags for install/upgrade/search/info/browse/doctor/index and dependency prompts.
cmd/entire/cli/plugin_gitremote.go Adds git-shellout helpers for semver tag listing and metadata fetch-at-tag.
cmd/entire/cli/plugin_gitremote_test.go Unit tests for tag sorting, metadata fetch, and repo-url name derivation.
cmd/entire/cli/plugin_fetch.go Adds release asset URL conventions, checksum handling, downloading, and archive extraction.
cmd/entire/cli/plugin_fetch_test.go Unit tests for URL conventions, checksum selection, extraction guards, and download verification.
cmd/entire/cli/plugin_deps.go Adds dependency planning/execution, remove guard helpers, and plugin doctor checks.
cmd/entire/cli/plugin_deps_test.go Unit tests for dependency planning, cycles, dependents, and doctor reporting.
cmd/entire/cli/login.go Minor constant usage change for GOOS comparison.
cmd/entire/cli/integration_test/plugin_remote_install_test.go End-to-end integration coverage for index install, URL confirmation gating, deps/remove-guard, and doctor exit behavior.
cmd/entire/cli/explain.go Introduces darwinGOOS constant for reuse.

Comment thread cmd/entire/cli/plugin_fetch.go
Comment thread cmd/entire/cli/plugin_index_test.go
- huh prompts use RunWithContext and map Ctrl+C/Esc through
  handleFormCancellation instead of conflating abort with decline;
  dependency-confirm failures are no longer reported as a skip the user
  chose. Non-interactive-without---yes gets a dedicated sentinel so the
  untrusted-install path fails while the post-install dependency path
  degrades to an informed skip.
- Context cancellation during sync/install/upgrade maps to SilentError
  per the clean.go/activity_cmd.go convention (silencePluginCancel,
  including the ctx.Err() check for killed git children).
- fetchAndVerify rejects asset names that could escape the staging dir
  (both separator kinds, dot segments) and removes partial downloads on
  checksum mismatch, oversize, or write failure.
- Index offline test uses os.RemoveAll instead of shelling to rm -rf
  (Windows portability).
- Strict-decode metadata test uses a non-near-miss unknown key; the
  previous deliberately misspelled field was silently corrected by a
  spell-fixing formatter pass, making the test vacuous.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entire-Checkpoint: 9770a0e2d1f8
@ashtom

ashtom commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

Addressed all five review findings in cd5984b:

  • Huh forms skip RunWithContext (Bugbot): confirmPluginAction and plugin browse now use form.RunWithContext(ctx) and route errors through handleFormCancellation, matching the doctor.go convention.
  • Confirm abort treated as skip deps (Bugbot): a prompt error is no longer reported as a user-chosen skip. Ctrl+C/Esc prints "Dependency install cancelled." (clean exit, main install stands); real prompt failures propagate as errors. Non-interactive runs without --yes get a dedicated sentinel — fatal for untrusted-URL installs, informed skip for post-install dependency confirmation.
  • Install path ignores context cancel (Bugbot): new silencePluginCancel maps cancellation to NewSilentError per the clean.go/activity_cmd.go convention, applied across install/upgrade/search/info/browse/index-update. It also checks ctx.Err() directly because a killed git child surfaces as "signal: killed" rather than context.Canceled.
  • Asset path traversal + partial files (Copilot): fetchAndVerify now rejects asset names containing either separator kind or dot segments before joining into the staging dir (today's callers only pass internally-generated candidates, but the boundary is now enforced where it belongs), and removes the partial download on checksum mismatch, oversize, or write failure.
  • rm -rf in test (Copilot): replaced with os.RemoveAll.

New unit tests cover the unsafe-name rejection and mismatch cleanup. mise run check (fmt + lint + unit + integration + canary) is green.

🤖 Generated with Claude Code

@ashtom

ashtom commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

bugbot run

Comment thread cmd/entire/cli/plugin_deps.go
Comment thread cmd/entire/cli/plugin_index.go
- planDeps walks a satisfied managed dependency's recorded manifest
  requirements, so installing a parent repairs gaps deeper in the chain
  (e.g. a grandchild removed with --force since install). Offline —
  reads the manifest, no extra network during planning.
- SyncPluginIndex sweeps a partial cache dir (no .git) before the
  initial clone; previously an interrupted first clone wedged discovery
  until the cache was cleared by hand, since git refuses to clone into
  a non-empty directory.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entire-Checkpoint: df589ff10cf4
@ashtom

ashtom commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

Addressed both round-two findings in 6ee0a3b:

  • Skipped transitive dependency planning: planDeps now walks a satisfied managed dependency's recorded manifest requirements, so installing a parent repairs gaps deeper in the chain (e.g. a grandchild removed with --force after install). Planning stays offline — it reads the local manifest, no extra network. PATH/local-dev-satisfied deps have no manifest to walk; those remain plugin doctor's domain. Regression test: satisfied parent with a missing grandchild produces exactly the grandchild action.
  • Failed index clone blocks retries: the un-cloned branch of SyncPluginIndex sweeps the cache directory before cloning, so a partial dir left by an interrupted first clone (no .git) can't wedge discovery behind git's non-empty-target refusal. Regression test: sync recovers from a junk-filled cache dir.

mise run check green.

🤖 Generated with Claude Code

@ashtom

ashtom commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

bugbot run

Comment thread cmd/entire/cli/plugin_group.go Outdated
Comment thread cmd/entire/cli/plugin_group.go
- Declining the untrusted-install prompt now prints "Install
  cancelled." and exits 0, matching the Esc/Ctrl+C path and the
  handleFormCancellation convention used by every other confirm.
  Exit codes no longer differ by how the user said no; automation
  never reaches this prompt (non-interactive fails earlier with the
  --yes hint).
- classifyInstallArg no longer stats bare arguments: names always
  resolve through the index, local paths must be explicit (./ or a
  separator), git-style. A stray CWD file sharing a plugin's name can
  no longer shadow the index; the index-miss error hints at
  'install ./<name>' when a matching local file exists.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entire-Checkpoint: 3bf711f1f5a4
@ashtom

ashtom commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

Addressed both round-three findings in 117b23b:

  • Install abort yields success exit: declining the untrusted-install prompt now prints "Install cancelled." and exits 0, identical to Esc/Ctrl+C — harmonized toward the handleFormCancellation convention every other confirm in the codebase uses. The automation concern can't arise in practice: non-interactive runs fail before the prompt with the --yes hint, so only interactive consistency was at stake.
  • Local file shadows index name: classifyInstallArg no longer stats bare arguments. Names always resolve through the index; local paths must be explicit (./entire-foo or any separator-containing path) — git-style disambiguation, and the two spaces are disjoint since plugin names can't contain separators. When an index lookup misses but a same-named local file exists, the error hints at entire plugin install ./<name>. Command help documents the rule.

mise run check green.

🤖 Generated with Claude Code

@ashtom

ashtom commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

bugbot run

Comment thread cmd/entire/cli/plugin_install_remote.go
UpgradeInstalledPlugin compared raw tag strings, so equivalent
spellings (v0.2.0 vs 0.2.0) triggered spurious reinstalls, and an
asset-less newest tag produced a misleading X → X upgrade line after
the install fell back to the already-installed version. Both
comparisons now go through semver.Compare on canonicalized tags: no
download when the newest tag is not strictly newer, and a fallback
that lands on the installed version reports up-to-date.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entire-Checkpoint: db4754bb3db1
@ashtom

ashtom commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

Addressed the round-four finding in 813670a:

  • Upgrade compares raw tag strings: both comparisons in UpgradeInstalledPlugin now use semver.Compare on canonicalized tags. Equivalent spellings (v0.2.0 vs 0.2.0) no longer reinstall — the check short-circuits before any download — and when an asset-less newest tag makes the install fall back to the already-installed version, the outcome reports up-to-date instead of an X → X upgrade line. Regression tests cover both (the spelling test runs against a repo with no asset server at all, so any download attempt would fail the test).

Known residual: a genuinely newer tag with an unpublished release still causes a re-download of the installed version on each upgrade run — avoiding that would require persisting known-asset-less-tag state, which doesn't seem worth it. Output is correct either way.

mise run check green.

🤖 Generated with Claude Code

@ashtom

ashtom commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 813670a. Configure here.

Comment thread cmd/entire/cli/plugin_fetch.go Outdated
Comment thread cmd/entire/cli/plugin_deps.go
Comment thread cmd/entire/cli/plugin_deps.go
- A checksum manifest that lists no asset for the current platform no
  longer aborts the download: selection continues through the other
  manifest candidates and the direct-probe fallback, so a stale or
  hand-written root checksums.txt can't mask an installable release.
  Verification is not weakened — an attacker controlling the manifest
  could list a malicious digest directly.
- Dependency planning warns when a scheduled dependency's tags or
  metadata can't be inspected, instead of silently omitting its nested
  requirements from the confirmed plan.
- Document why doctor's quarantine probe on the bin entry is
  sufficient: macOS xattr follows symlinks by default, so the check
  (and the suggested xattr -d fix) already operates on the pkg/
  target. Verified empirically; the round-five finding claiming
  otherwise was a false positive.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entire-Checkpoint: a11bfa5352d7
@ashtom

ashtom commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

Round five: two fixed, one rebutted, in 6da333c.

  • Checksum miss aborts other manifests (fixed): a manifest listing no asset for the platform now falls through to the remaining checksumCandidates names and the direct-probe fallback instead of aborting, so a stale or hand-written root checksums.txt can't mask an installable release. No verification weakening — an attacker who controls the manifest could list a malicious digest directly. Regression test: stale root manifest + published conventional asset installs via the probe.
  • Planning skips failed transitive fetch (fixed): when a scheduled dependency's tags or metadata can't be inspected, the plan now carries an explicit warning naming the plugin and pointing at entire plugin doctor, instead of silently omitting nested requirements. Regression test: untagged dependency repo plans the action plus the warning.
  • Quarantine check ignores symlink target (not a bug): macOS xattr follows symlinks by default — -s is the flag to act on the link itself. Verified empirically: xattr -p com.apple.quarantine <bin-symlink> returns the attribute set on the pkg/ target, exit 0. Doctor's existing probe on the bin entry therefore already covers the real binary, and the suggested xattr -d fix also operates on the target. Added a code comment documenting this so it doesn't resurface.

mise run check green. Findings across rounds: 3 → 2 → 2 → 1 → 2+1 false positive — at this point the remaining surface looks like speculative edge cases, so leaving further review to humans unless something specific comes up.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants