diff --git a/01-getting-started/README.md b/01-getting-started/README.md index 2c5549b..effc027 100644 --- a/01-getting-started/README.md +++ b/01-getting-started/README.md @@ -35,18 +35,6 @@ The server uses **streamable HTTP transport** with `mcp.run(transport="streamabl ## Quick Start -### 0. Configure the Judge LLM - -MCPChecker uses an LLM to verify test results. Set these environment variables before running tests, for instance with OpenAI: - -```bash -export JUDGE_BASE_URL="https://api.openai.com/v1" -export JUDGE_API_KEY="sk-your-key-here" -export JUDGE_MODEL_NAME="gpt-4o-mini" -``` - -The judge LLM evaluates whether the agent completed tasks correctly by analyzing the agent's output. - ### 1. Install Prerequisites **Install Claude Code** (AI agent): @@ -60,6 +48,14 @@ curl -fsSL https://claude.ai/install.sh | bash See the [official installation guide](https://github.com/anthropics/claude-code?tab=readme-ov-file#get-started) for Windows and other installation methods. +**Install the Claude ACP agent adapter:** + +```bash +npm install -g @agentclientprotocol/claude-agent-acp +``` + +This provides the `claude-agent-acp` command used by MCPChecker to run Claude Code as an ACP-compatible agent. + **Install uv** (Python package manager): ```bash curl -LsSf https://astral.sh/uv/install.sh | sh @@ -116,6 +112,20 @@ You should see: This quickstart includes a complete evaluation setup. Let's look at what gets tested and how it's defined: +### The Agent Configuration (`evals/agent.yaml`) + +```yaml +kind: Agent +metadata: + name: "claude-code-acp" +acp: + cmd: "claude-agent-acp" +``` + +**What this does:** +- Defines an ACP (Agent Client Protocol) agent that uses Claude Code via the `claude-agent-acp` adapter +- This agent configuration is referenced by both the eval runner and the LLM judge + ### The Main Eval Configuration (`evals/eval.yaml`) ```yaml @@ -124,19 +134,19 @@ metadata: name: "demo-server-test" config: - # Use Claude Code as the AI agent + # Use Claude Code as the AI agent (via ACP) agent: - type: "builtin.claude-code" + type: file + path: agent.yaml # MCP server configuration mcpConfigFile: mcp-config.yaml - # LLM judge configuration + # LLM judge configuration (reuses the same agent) llmJudge: - env: - baseUrlKey: JUDGE_BASE_URL - apiKeyKey: JUDGE_API_KEY - modelNameKey: JUDGE_MODEL_NAME + ref: + type: file + path: agent.yaml # Test tasks taskSets: @@ -150,9 +160,9 @@ config: ``` **What this does:** -- Configures **Claude Code** as the agent that will attempt the tasks +- Configures **Claude Code** as the agent via the ACP adapter defined in **agent.yaml** - Points to **mcp-config.yaml** to connect to your MCP server -- Defines the **judge LLM** settings (using your environment variables) +- The **LLM judge** also references **agent.yaml**, so no separate judge configuration is needed - Loads tasks from **tasks/add.yaml** and asserts the `add` tool must be used ### The Task Definition (`evals/tasks/add.yaml`) diff --git a/01-getting-started/evals/agent.yaml b/01-getting-started/evals/agent.yaml new file mode 100644 index 0000000..8c91869 --- /dev/null +++ b/01-getting-started/evals/agent.yaml @@ -0,0 +1,5 @@ +kind: Agent +metadata: + name: "claude-code-acp" +acp: + cmd: "claude-agent-acp" diff --git a/01-getting-started/evals/eval.yaml b/01-getting-started/evals/eval.yaml index a616360..cb511ae 100644 --- a/01-getting-started/evals/eval.yaml +++ b/01-getting-started/evals/eval.yaml @@ -5,17 +5,17 @@ metadata: config: # Use Claude Code as the AI agent agent: - type: "builtin.claude-code" + type: "file" + path: agent.yaml # MCP server configuration mcpConfigFile: mcp-config.yaml # LLM judge configuration llmJudge: - env: - baseUrlKey: JUDGE_BASE_URL - apiKeyKey: JUDGE_API_KEY - modelNameKey: JUDGE_MODEL_NAME + ref: + type: file + path: agent.yaml # Test tasks taskSets: diff --git a/02-linux-mcp-server/README.md b/02-linux-mcp-server/README.md index f00a7d4..8177d72 100644 --- a/02-linux-mcp-server/README.md +++ b/02-linux-mcp-server/README.md @@ -33,7 +33,15 @@ curl -fsSL https://anthropic.com/install-claude-code | sh For more installation options, see the [official installation guide](https://github.com/anthropics/claude-code). -### 2. Install MCPChecker +### 2. Install the Claude ACP Agent Adapter + +```bash +npm install -g @agentclientprotocol/claude-agent-acp +``` + +This provides the `claude-agent-acp` command used by MCPChecker to run Claude Code as an ACP-compatible agent. It is also used as the LLM judge. + +### 3. Install MCPChecker Download the latest release: @@ -44,7 +52,7 @@ chmod +x mcpchecker-linux-amd64 sudo mv mcpchecker-linux-amd64 /usr/local/bin/mcpchecker ``` -### 3. Install Linux MCP Server +### 4. Install Linux MCP Server ```bash pip install --user linux-mcp-server @@ -62,22 +70,6 @@ which linux-mcp-server For more installation options and documentation, see the [Linux MCP Server documentation](https://rhel-lightspeed.github.io/linux-mcp-server/). -### 4. Configure Judge LLM - -MCPChecker uses an LLM to verify test results. Set these environment variables: - -```bash -export JUDGE_BASE_URL="https://api.openai.com/v1" -export JUDGE_API_KEY="sk-your-key-here" -export JUDGE_MODEL_NAME="gpt-4o-mini" -``` - -**Why a judge LLM?** Testing AI agents requires flexible verification. Instead of exact string matching, we use an LLM to verify if the output is semantically correct. - -For example, instead of checking for the exact string "Fedora Linux 43", the judge checks if the output "contains information about the operating system". This allows the test to pass even if the formatting varies, as long as the required information is present. - -Each verification step includes a `reason` explaining what the judge is checking, which helps with debugging when tests fail. - ## What Gets Tested This quickstart tests two diagnostic tools from the Linux MCP Server: diff --git a/02-linux-mcp-server/evals/agent.yaml b/02-linux-mcp-server/evals/agent.yaml new file mode 100644 index 0000000..8c91869 --- /dev/null +++ b/02-linux-mcp-server/evals/agent.yaml @@ -0,0 +1,5 @@ +kind: Agent +metadata: + name: "claude-code-acp" +acp: + cmd: "claude-agent-acp" diff --git a/02-linux-mcp-server/evals/eval.yaml b/02-linux-mcp-server/evals/eval.yaml index 750d05a..6841f92 100644 --- a/02-linux-mcp-server/evals/eval.yaml +++ b/02-linux-mcp-server/evals/eval.yaml @@ -5,17 +5,17 @@ metadata: config: # Use Claude Code as the AI agent agent: - type: "builtin.claude-code" + type: file + path: agent.yaml # MCP server configuration mcpConfigFile: mcp-config.yaml # LLM judge configuration llmJudge: - env: - baseUrlKey: JUDGE_BASE_URL - apiKeyKey: JUDGE_API_KEY - modelNameKey: JUDGE_MODEL_NAME + ref: + type: file + path: agent.yaml # Test tasks taskSets: diff --git a/03-evolution-case-study/README.md b/03-evolution-case-study/README.md index c1b6323..e195fd5 100644 --- a/03-evolution-case-study/README.md +++ b/03-evolution-case-study/README.md @@ -181,18 +181,12 @@ Since the **code is identical**, differences in test results prove documentation ### Prerequisites -See [getting-started](../getting-started/) for installation of: +See [getting-started](../01-getting-started/) for installation of: - Claude Code +- `@agentclientprotocol/claude-agent-acp` (Claude ACP agent adapter) - mcpchecker - uv (Python package manager) -Set judge LLM environment variables: -```bash -export JUDGE_BASE_URL="https://api.openai.com/v1" -export JUDGE_API_KEY="sk-your-key-here" -export JUDGE_MODEL_NAME="gpt-4o-mini" -``` - ### Run Both Iterations **Iteration 1 - Bad Documentation:**