fix(block-kit): simplify and harden CLI validation by wildcard · Pull Request #119 · slackapi/slack-skills-plugin

wildcard · 2026-08-12T23:19:58Z

Summary

Closes #99.

resolve the public Slack CLI once before any Block Kit workflow, including
Fast Path and Modification Mode, including alias verification when the public
CLI is installed under a different command name
require the canonical no-auth CLI validation call when the CLI is available,
keep semantic validation errors on that transport, and disclose the final
validation transport precisely
request narrowly scoped permission before an identical retry for CLI
config/log write failures, and stop unvalidated when permission is denied
allow curl only when the CLI is unavailable or the canonical CLI invocation
itself cannot execute for a non-permission reason, with version/help captured
for that execution-failure path
remove the argument contract, developer-specific voice, duplicated pattern
list, and redundant surface/error guidance identified in the issue

Preview

This is a guidance-only change, so the preview is terminal evidence rather than
a product UI screenshot.

claude-code-review-correction-combined.mp4

Scenario	Prior supporting evidence	Exact local branch
installed CLI in Claude Code	`edb8e35` shows fingerprint followed immediately by the canonical validation call	same CLI-first state machine, plus alias verification and precise curl fallback only for genuine non-permission execution failure
semantic `ok:false`	payload corrected and retried on CLI	same transport-preserving retry contract
validation transport	CLI succeeded directly; zero curl	CLI when available, curl only when CLI is absent or the canonical invocation cannot execute; disclosure distinguishes those cases

The attached recording is prior supporting evidence from local candidate
edb8e35; it is not an exact-head recording of this branch. It demonstrates
the CLI correction that this branch keeps while also folding in issue #99 items
1-5. These runs are not claims of official client support or live Slack API
execution.

Testing

make lint
make typecheck
make test-unit — 26/26 passed on exact head
claude plugin validate .
PYTEST_ADDOPTS='-k skill-block-kit-modal' make test-eval — the one
change-relevant Block Kit tool-selection scenario passed
make test — run on exact head: units passed 26/26 and evals passed 27/31
Slack CLI v4.6.0 local checks: slack api --help documents --no-auth, and
slack api blocks.validate --no-auth 'blocks=[]' returned {"ok":true}

The four exact-head full-suite failures were outside this PR's Block Kit diff:
two existing scenarios expect slack_list_channel_members although the current
hosted MCP endpoint did not return that tool, and two unrelated scenarios
received transient Gemini 503 UNAVAILABLE responses. The changed Block Kit
evaluation passed.

For comparison, the same make test command on pristine upstream/main
77a10794 under the same host and credentials passed 18/18 units and 22/31
evals. Its failures reproduced the same two deterministic upstream eval
problems, six transient Gemini 503 UNAVAILABLE responses, and one model output
using slack_docs instead of the accepted slack-docs skill name. This
comparison shows the current branch introduces no demonstrated failure class;
it does not make the full command green.

Notes

This changes only skills/block-kit/SKILL.md, one focused unit contract file,
and one patch changeset. It does not modify Slack CLI, MCP/OAuth, plugin
manifests, or official client-support documentation.

Open PR #85 still overlaps the old Step 2 inline inspiration list. The clean
resolution is to keep #85's three new templates in
skills/block-kit/references/common-patterns.md while this branch keeps Step 2
as a generic pointer to that file instead of an inline example list. A local
replay against #85 head dccb83e8 completed with one textual conflict in
skills/block-kit/SKILL.md; the clean resolution retained all three templates,
kept the generic pointer, and passed make lint plus 24/24 unit tests.

Requirements

I've read and understood the Contributing Guidelines and have done my best effort to follow them.
I've read and agree to the Code of Conduct.
I've run make test and the tests pass.

The last requirement remains unchecked because the exact required command was
run and still exits nonzero.

changeset-bot · 2026-08-12T23:20:03Z

🦋 Changeset detected

Latest commit: fb77da2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
slack	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

salesforce-cla · 2026-08-12T23:20:04Z

Thanks for the contribution! Before we can merge this, we need @wildcard to sign the Salesforce Inc. Contributor License Agreement.

wildcard

Self-review guide for the narrowed CLI-validation slice. I checked the exact draft head against issue #99 items 6–10, the repository contribution rules, project-native tests, permission boundaries, and the open #85 overlap. These notes explain the key design decisions; they are not maintainer approval.

wildcard · 2026-08-12T23:20:40Z

+## Resolve Tooling
+
+Before choosing the Fast Path, Modification Mode, or full workflow, use the
+`slack:slack-cli` skill, **Step 1: Detect the Slack CLI**, to resolve the public


The preflight is deliberately capability-only and runs before every workflow path. Recording SLACK_CMD once avoids the original late-discovery failure and prevents validation and preview from repeating detection or drifting into the CLI skill's install/alias branch.

wildcard · 2026-08-12T23:20:40Z

+Use the `slack:slack-cli` skill, **Step 4: Calling Web API Methods (`slack api`)**,
+for the argument-passing contract. The canonical `blocks.validate` invocation
+below is already known: attempt it directly without running `api --help`,
+`--version`, or generic help first.


This is the narrow exception to generic help-driven command discovery: blocks.validate --no-auth is a documented Slack CLI contract, so the skill attempts it directly. The controlled Claude evidence showed that this removes three rejected help/version probes before validation.

wildcard · 2026-08-12T23:20:40Z


-**Path B: curl (fallback, when the CLI isn't installed).**
+If the command cannot start because the host blocks writes to the Slack CLI's
+config or log path, ask for the host's normal narrowly scoped permission and


The fallback boundary received the closest security review. A config/log denial may retry the identical command only through normal scoped host approval; if approval is denied, validation stops rather than using curl to route around the operator's decision. Semantic API and shared network failures likewise cannot be hidden by changing transport.

wildcard · 2026-08-12T23:20:40Z

+SKILL = Path("skills/block-kit/SKILL.md").read_text()
+
+
+def test_tooling_is_resolved_before_every_workflow_path() -> None:


These focused tests intentionally assert durable policy invariants rather than transcript wording: preflight ordering/reuse, canonical calls, semantic same-transport retry, permission non-bypass, and final disclosure. Native agent runs remain supporting behavioral evidence, not unit tests.

wildcard · 2026-08-13T05:24:08Z

Full-suite follow-up for exact draft head 5819e368:

make test was rerun with working Gemini and Slack MCP credentials.
Unit tests: 22/22 passed.
Gemini tool-selection evals: 26/31 passed.
The focused Block Kit scenario passed independently with
PYTEST_ADDOPTS='-k skill-block-kit-modal' make test-eval.

The five full-suite failures were:

Two transient Gemini 503 UNAVAILABLE responses.
Two existing scenarios expecting slack_list_channel_members, which was
absent from the 10 tools returned by the current hosted Slack MCP endpoint.
One model response choosing slack_docs rather than the accepted
slack-docs name.

As a same-environment control, I ran the identical full command on pristine
upstream/main at 77a10794. Its units passed 18/18, while its evals passed
22/31 and reproduced all three failure classes above. This indicates that the
five failures observed on this branch are not introduced by the three-file
Block Kit change. It does not convert the full run into a pass, so the PR's
make test requirement remains unchecked.

The change-specific regression contract remains deterministic: four unit tests
cover early one-time CLI resolution, canonical no-auth validation calls,
same-transport semantic retry, permission non-bypass, fallback classification,
and final transport disclosure.

WilliamBergamin

Hi @wildcard thanks for your interest and time spent on this contribution 💯 I think its a step in the right direction

Left a few comments, I don't think we need the python unit tests here

Is there any reason why your changes have a line limit shorter then the current implementation?

WilliamBergamin · 2026-08-18T19:39:56Z

+Slack CLI command once. Keep the result as either `SLACK_CMD` or an explicit
+reason the CLI is unavailable. This is a capability check only; do not
+authenticate, validate, or write Slack CLI configuration during this preflight.
+For Block Kit validation, follow that detection step's standard-path probe, the
+`PATH` probe, and the alias inquiry/verification branch. Do not propose
+installation unless the user independently asked to configure the Slack CLI. If
+the standard path and `PATH` probes fail, ask whether the public Slack CLI is
+installed under a different name or alias. If the user supplies an alias,
+verify it with `<alias> _fingerprint 2>/dev/null` before setting `SLACK_CMD`.
+If no alias is supplied or verified, record the CLI as unavailable and use
+curl.


I wonder if this should be part of 🤔

`slack:slack-cli` skill, **Step 1: Detect the Slack CLI**

@wildcard any chance you tested this out?

WilliamBergamin · 2026-08-18T19:40:19Z

+Determine the target surface from context. If it is genuinely ambiguous, ask
+the user with the host's available user-input mechanism:


WilliamBergamin · 2026-08-18T19:40:41Z

- "A feedback form with a text input and a category selector" (Simple Form Modal)
- "A notification message with an alert banner, description, and Approve/Reject buttons" (Notification Alert / Approval Message)
- "A dashboard home tab with a welcome header, key metrics in fields, and quick-action buttons" (Dashboard Home Tab)
- "A settings modal with dropdowns, checkboxes, and a time picker" (Settings Modal with Multiple Input Types)
- "A table of sprint tasks with status and points" (Data Table)


WilliamBergamin · 2026-08-18T19:40:57Z

-**Surface constraints to check:**
-
- Block count limit: 50 for messages, 100 for modals/home tabs
- Modal-specific: if using `input` blocks, the modal payload must include a `submit` field
- Table: only one `table` block per message
- Surface compatibility (whether a block is valid on the chosen surface) and element compatibility (whether an element is allowed inside a given block) are not always spelled out on a component's doc page. Build the layout from the docs, and let `blocks.validate` in Step 5 confirm it. It is the authoritative check.


WilliamBergamin · 2026-08-18T19:45:00Z

@@ -0,0 +1,91 @@
+import re


I don't think we need these unit test, the skills and project might change over time and these might get in the way 🤔

fix(block-kit): resolve CLI before validation (slackapi#99)

5819e36

salesforce-cla Bot added the cla:missing label Aug 12, 2026

wildcard commented Aug 12, 2026

View reviewed changes

wildcard mentioned this pull request Aug 12, 2026

Improve and simplify the block-kit skill #99

Open

salesforce-cla Bot added cla:signed and removed cla:missing labels Aug 13, 2026

fix(block-kit): complete issue validation contract (slackapi#99)

fb77da2

wildcard changed the title ~~fix(block-kit): resolve CLI before validation~~ fix(block-kit): simplify and harden CLI validation Aug 18, 2026

wildcard marked this pull request as ready for review August 18, 2026 07:14

wildcard requested a review from a team as a code owner August 18, 2026 07:14

WilliamBergamin requested changes Aug 18, 2026

View reviewed changes

		SKILL = Path("skills/block-kit/SKILL.md").read_text()


		def test_tooling_is_resolved_before_every_workflow_path() -> None:

		Determine the target surface from context. If it is genuinely ambiguous, ask
		the user with the host's available user-input mechanism:

Uh oh!

Conversation

wildcard commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Preview

Testing

Notes

Requirements

Uh oh!

changeset-bot Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

salesforce-cla Bot commented Aug 12, 2026

Uh oh!

wildcard left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wildcard commented Aug 13, 2026

Uh oh!

WilliamBergamin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wildcard commented Aug 12, 2026 •

edited

Loading

changeset-bot Bot commented Aug 12, 2026 •

edited

Loading