Tag: mcp
All the articles with the tag "mcp".
-
Eval Study #3 — Agent Eval, the Limits of Single-Step Scoring, and Multi-Step Grading
Expanding from RAG Eval to agent Eval. Agents have 6 axes to judge (tool selection, argument extraction, multi-step trajectory, termination judgment, safety guards, final answer quality), so separating axes—like unit tests before integration tests—makes it faster to pinpoint causes. Today I covered just 3: tool selection, multi-step, and trap regression. I fed in MCP server tool metadata (name, description, schema) and measured against a (question, expected tool, expected args, level) test set → tool selection accuracy 11/13 = 84.6%. Two failure cases were interesting — (1) a single-step eval falsely flagged a case that was actually correct as a multi-step trajectory, showing the limits of single-step scoring, and (2) an ambiguous tool description caused "electricity bill" to wrongly pick consumption, while "how much money is it?" got it right — patching the system prompt instead of fixing the description risks overfitting. I then switched to multi-step eval (did it call all necessary tools, in order, using prior results, with correct termination judgment) → 4/4 = 100%, and the case that failed under single-step scoring now passed.
-
MCP Study #3 — What Resources / Prompts Actually Are + Integration with LangGraph (`MultiServerMCPClient` · `ainvoke`)
Beyond Tools covered in #1 / #2, this post covers the other two components of MCP: Resources (data for the LLM to read, background context, read-only) and Prompts (predefined templates). I check a new server with Inspector + Claude Desktop → integrate the MCP server into LangGraph using langchain-mcp-adapters. Covers why `ainvoke` is needed since MCP communication is asynchronous, the secret behind how MultiServerMCPClient loads servers as-is (= MCP standard compliance), and the difference in domain response quality when injecting Resources as a system prompt.
-
MCP Study #2 — Connecting My MCP Server to Claude Desktop · Tool Calls + Approval UX
Actually connecting the energy-management MCP server I built yesterday to Claude Desktop. From finding claude_desktop_config.json (Settings → Developer → Edit Config) → registering the server → restarting → checking the connectors menu → asking 'show me the factory line list' and watching Claude call the tool with the 'Always Allow/Deny' approval UX. Question raised: how do you use RAG and MCP together? (Answer: wrap RAG in an MCP server)
-
MCP Study #1 — Getting Started with Model Context Protocol · First Call with stdio Server + Inspector
Starting to study MCP (Model Context Protocol) today. It's a standard protocol created by Anthropic that standardizes how LLMs access external systems. Comparing it with Tool Use → building my first MCP server (say_hello / add_numbers) with the Python SDK → verifying the stdio connection with MCP Inspector → building a fake energy management server that connects with FEMS. Connection failure due to venv python.exe path issue + fix.