We recently built an MCP server exposing part of one of our internal systems as tools for LLM agents to use (in our case, mostly querying and reading data, things like searches, lookups, reports). Now we’re figuring out how to properly test it.
Has anyone here gone through building an MCP at their company? How did you approach the test strategy for it? What are you actually using for automation on the QA side? Genuinely curious what’s working for people in practice right now.
I think it depends on what is behind the MCP. In our case - it is usually a REST API. Which requires separate testing activities..
Therefore, on the QA side, we only have test cases, and not even for all MCPs.
We do not test MCP protocol itself.
we have unit tests to cover everything we can
integration tests against real QA env with services. if we have test cases, usually we implement them specifically in form of integration tests.
i’d keep protocol and tool correctness separate from whatever sits behind it. REST tests can prove the service, but the MCP layer still needs schema checks, permission boundaries, argument validation, pagination, and deterministic fixture-backed integration tests. otherwise an agent can call the right API through the wrong contract and you only see it during evaluation.
for agent scenarios, assert the returned fact and side effect, not the model’s prose. log the tool version, exact arguments, source record ids, and final readback so a wrong answer can be reproduced without replaying the whole conversation. model evals and server correctness should fail in different buckets.
Just to add some color from my side to @dwaynesamuels point:
For MCP used by agents, tool names/descriptions, parameter schemas, prompts, and even error message matter. They are part of interface the model see.
You also do not control the model on the other side. Cursor, ollama, vs code plugins, etc., users can switch models whenever they want. So I do not optimize for particular model. I run the core scenarios against a weaker local model and tune.
If a smaller model can pick the right tools, build valid args, execute code, complete expected call sequence, that’s good enough for me.
I keep that separate from contract/integration tests: those prove the server works. Model runs prove the MCP interface makes sense to an LLM.
But your case may be completely different, so you’ll probably need to figure out what works best for your setup.
Cheers.