Production AI Drift Planner: Monitoring Plans and Calibration Boundaries

Abstract

Production AI Drift Planner version 1.0.0 creates portable monitoring plans for production chat systems, retrieval-augmented generation (RAG) and tool-using agents. The archived release contains a static browser application, an 11-metric catalog and seven source records. It exports the selected configuration and metric definitions as JSON or Markdown. Every output remains DRAFT_REQUIRES_CALIBRATION. Pharos Production, an AI software development company, developed the production AI drift monitoring planner to record observation policies, responsible owners and proposed fallback targets before teams connect telemetry. This package preserves source commit f2938cfb84bbe09c5eab4f95dd404f1ca67c39ae, including all 32 tracked source files. What a plan records Select a system type, daily request volume, evaluation sampling percentage, baseline duration and observation window. Record a minimum sample target, persistence setting, reviewer and rollback target. Individual metric thresholds can be overridden within their declared numeric bounds. The engine validates configuration keys, system identifiers and threshold overrides before selecting the metrics applicable to that system. The catalog covers input distribution shift, task success, benign-request refusals, schema conformance and latency. RAG-specific entries describe grounding, retrieval recall and corpus update lag. Release and provider lifecycle entries accompany an agent-specific unauthorized-tool-effects signal. Each metric retains its definition, unit, comparator, illustrative threshold, investigation response and rollback guidance, with source identifiers and dated source references. The related production AI engineering guide to evaluation and drift provides background on instrumentation and post-deployment evaluation. Its broader industry statistics are not calibration evidence for this planner. Sampling and calibration boundaries Expected request samples equal daily requests multiplied by the observation window in hours and divided by 24. Evaluated-sample estimates also apply the evaluation percentage. Event-based metrics receive no numeric sample estimate. A metric can therefore be marked LOW_EXPECTED_SAMPLE, SAMPLE_ESTIMATE_ONLY or EVENT_BASED. None of these states confirms that usable labeled observations exist. The supplied example uses 1,000 daily requests, 10% evaluation sampling and a 24-hour window. It estimates 1,000 request samples and 100 evaluated samples where those sample classes apply. The example selects 10 RAG metrics, leaves the owner and rollback target empty and retains warnings for both missing assignments. These are illustrative planning inputs, not observed traffic or production measurements. Thresholds require local calibration against a defined baseline, eligible denominators and operating constraints. Baseline duration and persistence are recorded policy parameters, not computations over time-series data. Naming a fallback does not establish its compatibility or authorize a rollback. The software does not collect telemetry, calculate observed drift, run an alert pipeline or execute recovery actions. Reproduce and inspect Use Node.js 22 or newer. From the extracted package root, run node reproduce.mjs to rebuild the default plan in memory and compare both exports with the archived example files. In source/, run npm run verify to regenerate static assets, execute the eight original tests and check the release contract. There are no npm dependencies or model calls in these checks. The supplement documents the export fields and numeric boundaries. SHA256SUMS records per-file integrity, while SOURCE-PROVENANCE.json identifies the preserved revision. Package verification on September 21, 2026 confirmed the original tests and example exports. The source register retains its September 8 verification dates; packaging does not refresh every external claim. Reuse and operational scope The MLOps and LLMOps services from Pharos Production describe implementation work around deployment, monitoring and incident response. A monitoring plan is one input to that work. Operational readiness still requires instrumented data, evaluated thresholds, accountable reviewers and a verified recovery procedure. Original code, worksheet content and the reproduction supplement use the MIT license. Linked third-party material retains its owners' rights. Research, implementation and technical checks were AI-assisted. No independent human validation, production benchmark or guarantee of drift detection is claimed.