mcp-probe: a conformance and reliability checker for Model Context Protocol servers, and the replication package for the August 2026 MCP ecosystem study

Abstract

This archive is the mcp-probe repository at version 0.1.2, and it is the replication package for arXiv:2609.10962, "What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead". What changed since the 16 September archive (10.5281/zenodo.22781122), which this supersedes. The software is byte-for-byte the same; two documentation defects are fixed and a licence is added, and Zenodo does not allow files to be replaced on a published record. The root README.md in that archive labelled 7,414 as the sampling frame. That count includes deprecated servers; the frame drawn from was the active subset, 7,258, which is what the paper reports. Both are now shown with the difference explained. study/2026-08/LICENSE-DATA.md is new. The data and documentation are CC BY 4.0 and the scripts stay MIT, stated file by file rather than left to inference. The version string carries +archive.2 because this is the second archive of the same software release, not a new release of the software. The tool. mcp-probe is a dependency-light command-line tool that connects to any Model Context Protocol (MCP) server over stdio, performs the initialize handshake, enumerates advertised tools, validates each tool's JSON Schema against the MCP specification, measures wall-clock response time, and exercises safe round-trips. It is both a developer diagnostic and the instrument used in the study. The study artifacts are under study/2026-08/: census/: the 2026-08-22 registry-wide census of 24,135 servers, and the two-point 2026-07-14 to 2026-08-22 series, with a provenance note naming an uncommitted run that is cited nowhere. sample/: the seeded draw script, the resumable probe runner, the aggregator, and the per-server outcome of all 400 draws in study-2026-08-22.tsv, so every aggregate in the paper can be independently recounted rather than trusted. redundancy/: the three-corpus near-duplicate measurement over MCP, BFCL and UltraTool, its self-test, its threat-test suite, and a SHA-256 for every source file it read. A limit, stated. The 7,258-candidate sampling frame the August draw was made from did not survive the environment that produced it, and is not published here or anywhere. Its SHA-256, its count and its snapshot time are recorded in sample/summary-2026-08-22.json, so the draw can be checked but not re-executed; study/2026-08/README.md sets this out in full. The benchmark/candidates-registry.json in this archive is the earlier 2026-07-14 frame of 5,671 candidates, retained for the static tier. Source: github.com/itguruhaseeb/mcp-probe at commit b447e46. Paper: arXiv:2609.10962. Licence: MIT for the code, CC BY 4.0 for the data and documentation; see LICENSE and study/2026-08/LICENSE-DATA.md.