Extracting AI agent-accessible data from biodiversity literature with corpus

Abstract

[1.4.0] - 2026-09-09 Theme — v1.4 silent wrongs and the hazards behind them v1.3 made the pipeline show where evidence came from. This cycle went after what survives behind that: results that are wrong without saying so, and build hazards that cost hours per run without ever failing. The organizing complaint is that a wrong answer and a right one looked the same. A text layer of unmappable glyph indices counted as clean, so one paper was indexed as 1,200 fragments of /G52/G55/G4C/G4A/ and language-detected as Swahili. OCR language packs were picked from a text layer already known to be broken. An OSD script verdict was acted on without corroboration, so a Latin page could be re-OCR'd as Russian. Ph. pelagica resolved to nothing. corpus status ignored a filter in silence, and compute.accelerator was honoured by one stage and ignored by the next. None of these announced themselves; every one of them changed an answer. So the work was making wrongness visible or impossible: detection that corroborates before it acts, expansion that resolves what the literature actually writes, MCP tools that report what they returned rather than only whether they cut something off, and stages that fail loudly instead of degrading. Alongside it, the operational hazards that make a build unreliable at scale — Grobid jobs colliding on a fixed port, a GPU allocation quietly becoming a CPU one, a build with no way to bound its memory, and long stages with no sign of life. Nine of the cycle's twenty-two items had their shape changed by measurement before they were written, six of them against the issue's own proposal, and six turned out to need deletion or nothing at all rather than new code. The release rebuild of all 1,775 documents then found two more defects that no unit test reached, which is the argument for keeping that gate. Added Extraction failure records no longer outlive the failure. The v1.4 full-corpus rebuild finished with all 1775 documents complete and all 28 Stage 1 array tasks exiting 0, while stage1_failures.json still named one document as crashed by signal 9 — written by an earlier run, and nothing ever removed it. A finished corpuscle carried a failure record indistinguishable from a live one. Clearing the file on the success path is the obvious fix and is wrong: 28 array tasks share one output_dir, so a shard that finished cleanly would delete the record a still-failing shard had just written, trading a stale record for a lost one. Records are now written one file per shard under stage1_failures/, and a shard rewrites or retracts only its own. An unsharded run processed every document, so it alone also clears stale shard records and the legacy single file. Dependency errors name the module that actually failed (#258 follow-up). On a Mac with the conda env unactivated, the local VLM backend reported transformers >= 4.45 is required — but torch was the missing package and transformers was fine. The handler named the package it asked for rather than the one that broke, sending the reader to install something that was not the problem. A confident wrong instruction costs more than a vague one. ImportError.name carries the module that actually failed and separates the cases cleanly: an absent package and a too-old package both report the module we asked for, while a broken dependency chain reports the link that broke. pipeline/optional_deps.py turns that into the message, and all three ML backends use it — transformers, sentence_transformers and anthropic all sit behind heavy dependency chains and all had the same latent bug. figures.vision_dtype — the local VLM's weight dtype is selectable (#258). It picked its dtype from whether the device was CUDA rather than from what the device supports, so MPS got float32: ~30.4 GB of weights for Qwen2.5-VL-7B against ~15.2 GB at half precision, which is the difference between "does not fit a 32 GB Mac" and "fits". Apple Silicon supports bf16 and fp16, and a branch keyed on == "cuda" is the shape of "CUDA is the one I tested" rather than a statement about MPS. The default is unchanged, and now for a measured reason rather than caution — auto stays bfloat16 on CUDA and float32 elsewhere. Run on an M2 Max over four 9-11 panel plates: all three dtypes load and none loses or invents a panel, but half precision moves the boxes. Against float32, bfloat16 scored mean IoU 0.65-0.75 per plate and float16 0.69-0.93, each with a worst panel at 0.0 — a completely disjoint box. Two float32 runs agreed at IoU 1.0 on every panel, so that spread is the dtype, not the model. Generation is greedy and the model emits coordinates as digit tokens, so a logit difference too small to matter flips a digit and a coordinate jumps hundreds of pixels. Half precision here is a heavy-tailed chance of a wrong box, not a slightly blurrier one. The memory problem the issue opened on is nonetheless real: float32 measured 37.69 GB of Metal allocation, worse than the ~30.4 GB the weights predict, so the shipped default does not fit a 32 GB Mac. Set vision_dtype there — half precision beats no panel detection — but it is a fallback with known worse geometry. Also worth knowing before comparing corpuscles built on different hardware: bfloat16 on MPS disagreed with the bfloat16-on-H200 ROIs this corpuscle shipped on all four plates, so Pass 3b geometry is hardware-bound. tools/qc/vlm_dtype_probe.py reproduces all of it. A pixel ceiling on saved figures — figures.max_pixels_long_side, default 3000 (#184). Figures are ~88% of a served bundle, so their size is most of what a colleague downloads. Measured on the 1,775-document tree: 21,521 figures holding 11.88 GiB, median longest side 1,351 px against a p99 of 5,697 and a maximum of 16,237 — the mass is a small tail. A 3000 px cap recovers 2.74 GiB (23%). This is not max_dpi, and max_dpi cannot do it. The byte-heavy figures are full plate pages at an ordinary 400 dpi, 11-17 MB each, so capping density to 300 would leave a 16,000 px figure at 12,000 px. There is a second population above 2,000 dpi, but those files are 0.4-0.5 MB and hold no bytes. The flat cap wins on measurement, which is the decision #184 asked for. It costs the median panel-detected figure 0%, because 966 of 1,015 such figures (95%) are already under 3000 px. Capping only figures with no detected panels recovers 2.71 GiB — 0.03 GiB more — while depending on rois == 0, which on this tree means "ROI detection never ran" for 95% of figures rather than "has no panels". PLAN.md's gate said that proxy was unsafe; measured, it is weaker than that. So: flat, dependent on nothing. Applied on both write paths — the native bbox re-render and docling's own save, which is the only path in fixed mode — and available as --max-pixels-long-side on tools/backfill_figure_dpi.py so an existing bundle can be shrunk without re-running docling. A capped figure records resolution_mode: native+pixel_capped, so a shrunk figure is distinguishable from a small source. Verified on a real 7,923 px figure: 45% smaller, correctly marked. The bound is the cap within a pixel or two rather than to the pixel, because PyMuPDF sizes a pixmap from the integer rect of the transformed clip; that is documented rather than papered over. A progress heartbeat during long per-document stages (#170). The run log went silent for minutes during docling layout analysis — measured 3m20s on a 27-page scan, far longer on the 314-page Totton monograph — and the last line before the gap is a docling banner, so a reader tailing run.log could not tell working from hung. More pressing since v1.0 re-OCRs scans rather than trusting their text layers: 31 of 35 papers in the smoke corpus now OCR, where 4 did before. Every stage now emits still running after 3m20s (314 pages) on an interval, carrying the paper prefix so a line stays attributable in an interleaved multi-paper stream, and the page count because that is the answer to "why is this slow". It lives in _stage, the one place every stage passes through, so OCR and the vision pass are covered too rather than only docling. New logging.heartbeat_seconds, default 60, 0 to disable; the first beat lands one interval in, so no ordinary stage emits one. Verified on a real extract run, where it filled a previously silent 33-second gap. Build memory is boundable (#182). corpus exposed no way to bound the memory a build uses — not in config.yaml, not on any CLI, not via an environment variable, and config_schema.py had no memory, worker or concurrency field at all. On a 12-core / 32 GB CPU-only host, seven concurrent --only extract workers put three docling processes in flight at once, one on a 314-page scan, and the build died mid-stage with oom_kill 2 in /proc/vmstat and no error in any worker log. New docling block (queue_max_size, layout_batch_size, ocr_batch_size, table_batch_size, document_timeout) and compute.num_threads bound extraction, which is where the memory actually goes — embedding the same corpus peaked at 3.71 GB resident. All are unset by default and omitted keys are not passed to docling at all, so a build that sets nothing behaves exactly as before and cannot drift if docling changes one of its own defaults. Option names are asserted against the installed docling, since a silently-ignored option would be worse than none. embeddings.batch_size and corpus run --only embed --batch-size N reach LocalBackend, which has taken a batch_size since it was written and whose docstring described it as the memory lever while nothing could reach it: get_embedder() forwards **kwargs and its only caller built them from --device alone. INSTALL.md now documents the cgroup cap, which is the outer bound and needs no corpus settings: MemoryHigh throttles by reclaim and only MemoryMax kills, so a capped build is squeezed first and any kill lands inside its own cgroup instead of the kernel picking a bystander — on the affected host, an uncapped burst took out a tmux server h