Benchmarks for JSON Schema Integration

[jenny-s51]

#12 introduces the component-schemas tool, which provides structured JSON Schema data for PatternFly React components.

We need to benchmark the updated MCP server using actual AI agents in Cursor (or similar AI coding assistants).

Implement a benchmark that:

1. Uses AI Agent: Integrate with Cursor's AI agent or use an LLM API (e.g., Claude, GPT-4) to generate actual AI responses
2. Tests on a "Golden Set": Create a curated set of prompts that represent common developer queries about PatternFly components
3. Compares Two Scenarios:
   • Before: AI agent using only the use-patternfly-docs and fetch-docs tools (parsing unstructured documentation)
   • After: AI agent using all tools including component-schemas (structured JSON Schema data)
4. Measures Metrics:
   • Accuracy: Did the agent provide the correct prop names / types?
   • Completeness: Did it find all relevant props?
   • Response Time: How long did it take to generate response?

Success Criteria

• Golden set of prompts
• Benchmark script that runs prompts against the MCP server (with and without component-schemas)
• LLM responses captured and evaluated for accuracy
• Document results and share findings

[cdcabrera]

This needs to be modified slightly to remove the subjective/relativistic nature of the task.

Specifically, the variation in testing vs a control base against a model, user, user's machine and familiarity with prompting... all of which can lead to a subjective/relativistic results around the terms "accuracy", "speed", and "completeness"

We can test, does the tool (sans-model/llm)

• work - PARTIAL, minimally incorporated into the basic repo tests, we can expand into e2e
• respond with actual data from the schema - PARTIAL, minimally incorporated into the basic repo tests, we can expand into e2e

More subjective/relativistic results should be deferred in the short term until a testing environment is available. This could involve a containerized/imaged llm that we can use as a baseline, with a possible consideration that for the number of questions, type of answer, and factored in with a number of runs you could in theory produce something "accuracy-like" or "completeness-like" gauged within the number of test runs.