Responsible use of large language models in digital health: an equity-first governance framework

Abstract

Large language models (LLMs) are being rapidly deployed for digital health, yet their equity impacts remain poorly characterized. We present an equity-first audit framework anchored in the first nations mental wellness continuum framework (FNMWCF) and apply it to three open-source LLMs (LLaMA-3.2:latest, Mistral-7B, and DeepSeek-r1:8B). Using three FNMWCF-informed persona prompts derived from the Canadian Community Health Survey (CCHS), we generated 150 stochastic responses per persona from each model (temperature \(=0.7\) ), yielding 450 responses per model and 1350 responses in total. We analysed the full corpus with automated, reproducible text metrics and separately conducted a small, exploratory single-rater human pilot on a 0–3 rubric. Across all 1,350 responses, lexical cultural markers were near-universal (100%), but explicit safety scaffolding varied: assessed on full responses, crisis-guidance language appeared in \(\approx\) 99.6% of LLaMA-3.2 and Mistral-7B responses and in 71.3% of DeepSeek-r1:8B responses, whereas an explicit medical-disclaimer/clinician-referral marker (a “not medical advice” statement or a recommendation to consult a doctor/clinician/provider) was almost entirely absent ( \(\le\) 0.2%). A sensitivity analysis showed that the 180-word display trim materially reduced the measured DeepSeek-r1:8B crisis prevalence (to 30.0%; a 41-point drop) while barely affecting the other models, indicating that trimming can remove late-appearing safety content. Because the personas describe moderate-severity, non-emergency wellness scenarios and the prompt explicitly requests crisis and urgent-care guidance, the high crisis-guidance prevalence may reflect both prompt adherence and a more conservative safety tendency in some models rather than uniformly safer behaviour; marker prevalence alone cannot establish whether crisis messaging is calibrated to scenario severity. Readability estimates (Flesch–Kincaid grade \(\approx\) 10–11) exceeded the grade 6–8 range recommended for patient-facing materials. FNMWCF theme terms appeared frequently, but such lexical presence primarily reflects adherence to the FNMWCF-informed prompt and is a proxy only; it does not establish cultural adequacy. The human rubric pilot (13 rated DeepSeek responses, one rater) is reported as exploratory; it does not support a model comparison, and inter-rater reliability could not be computed with the available data. Within the evaluated models, prompts, and experimental conditions, our findings suggest that technical fluency is no guarantee of equitable adequacy. We provide a reproducible equity-first audit pipeline and an operational governance framework linking each empirical signal to a concrete oversight action.