Open Source Summit + Embedded Linux Conference North America 2026

AI agents don’t fail like traditional software. They don’t just throw exceptions, they drift. They misinterpret tools, invoke the wrong functions or behave differently across environments. When deploying Arm’s Open Source custom MCP server to power AI assistants for architecture development, migration, and optimization, we faced a critical question: how do we test a system built for nondeterministic interaction? In this talk, I’ll share how we moved from manual validation to a repeatable, CI-enforced testing strategy using Pytest and Testcontainers. We spin up real MCP server in Docker during tests, validating tool discovery, invocation, and protocol compliance end-to-end.
This isn’t about mocking LLM output. It’s about testing the contract between agents and tools. The key insight: treat your MCP server like production infrastructure, not experimental glue code. Because “it worked on my machine” is not a deployment strategy.
Session takeaways:
• A demo of Arm’s Open Source MCP server(github.com/arm/mcp)
• Why unit tests are insufficient for agent-facing systems
• How we run MCP server inside containerized test environments
• How GitHub Actions automate CI integration testing