Skip to content

fix(model-benchmark): don't drop the claude provider on macOS subscription installs#1894

Open
jbetala7 wants to merge 1 commit into
garrytan:mainfrom
jbetala7:oss/fix-1890-claude-adapter-darwin-auth
Open

fix(model-benchmark): don't drop the claude provider on macOS subscription installs#1894
jbetala7 wants to merge 1 commit into
garrytan:mainfrom
jbetala7:oss/fix-1890-claude-adapter-darwin-auth

Conversation

@jbetala7
Copy link
Copy Markdown
Contributor

@jbetala7 jbetala7 commented Jun 7, 2026

Problem

ClaudeAdapter.available() (test/helpers/providers/claude.ts) treats Claude auth as present only when ~/.claude/.credentials.json exists or ANTHROPIC_API_KEY is set:

const credsPath = path.join(os.homedir(), '.claude', '.credentials.json');
const hasCreds = fs.existsSync(credsPath);
const hasKey = !!process.env.ANTHROPIC_API_KEY;
if (!hasCreds && !hasKey) {
  return { ok: false, reason: 'No Claude auth found. ...' };
}

On the default macOS Claude Code subscription install, ~/.claude/.credentials.json is absent — the credential lives in the login Keychain — yet the subscription session authenticates claude -p fine. So the file-existence check is a proxy that misses a valid auth state, and available() reports NOT READY for a provider whose run() path (claude -p --output-format json) would actually succeed. Any caller gated on this available() (model-benchmark, and the LLM-judge path) silently drops the claude provider for that whole population.

On main (cab774c)

$ gstack-model-benchmark --dry-run --prompt "hi"
  claude: NOT READY — No Claude auth found. ...

on a machine where ~/.claude/.credentials.json is absent, claude --version resolves, and claude -p works.

Root cause

An absent ~/.claude/.credentials.json on macOS is not evidence of missing auth — it is the expected state for a Keychain-backed subscription install. available() rejecting an auth state that run() can actually use is an internal inconsistency, not a user-config gap.

Fix

  • Keep the positive signals (creds file or ANTHROPIC_API_KEY) as a fast yes.
  • On darwin, be optimistic when the binary resolves and defer the real auth decision to run(), which already classifies a genuinely logged-out state as an auth error (/unauthorized|auth|login/i). So a logged-out mac still surfaces a clear auth error from run() instead of being silently dropped at available().
  • Non-macOS keeps the strict file/key check, where ~/.claude/.credentials.json is the actual credential store.

Testing

Adds test/claude-adapter-available.test.ts (resolver satisfied with a real binary via GSTACK_CLAUDE_BIN, os.homedir pointed at an empty dir, process.platform overridden — no live CLI):

  • macOS, no creds file, no key → available (fails on main, passes here)
  • non-macOS, no creds file, no key → not available, with the auth reason
  • ANTHROPIC_API_KEY set → available on any platform
  • present .credentials.json → available on non-macOS
  • unresolvable binary → not available, before the auth sniff
bun test test/claude-adapter-available.test.ts          # 5 pass
bun test test/benchmark-runner.test.ts test/benchmark-cli.test.ts test/claude-adapter-available.test.ts  # 29 pass

Fixes #1890

🤖 Generated with Claude Code

…ption installs

ClaudeAdapter.available() treated Claude auth as present only when
~/.claude/.credentials.json exists or ANTHROPIC_API_KEY is set. On the
default macOS Claude Code subscription install the credential lives in the
login Keychain, not that file, so available() returned NOT READY for a
provider whose run() path (`claude -p --output-format json`) authenticates
fine via the same subscription session. model-benchmark and the LLM-judge
path silently dropped the claude provider for that whole population.

An absent ~/.claude/.credentials.json on macOS is not evidence of missing
auth. Keep the positive signals (creds file or ANTHROPIC_API_KEY) as a fast
yes, but on darwin be optimistic when the binary resolves and defer the real
auth decision to run(), which already classifies a genuinely logged-out
state as an `auth` error. Non-macOS keeps the strict file/key check, where
~/.claude/.credentials.json is the actual credential store.

Adds test/claude-adapter-available.test.ts covering: macOS no-creds-no-key
now available; non-macOS no-creds-no-key still not available; API key and a
present creds file available on any platform; unresolvable binary still
reported before the auth sniff. The macOS case fails on main and passes with
this change.

Fixes garrytan#1890

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@trunk-io
Copy link
Copy Markdown

trunk-io Bot commented Jun 7, 2026

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

model-benchmark: claude adapter available() reports "No Claude auth found" for macOS subscription installs (claude -p works)

1 participant