Under the Bridge - Episode 1

Abstract

Claude THE CANAL STACK A technical podcast recorded under a bridge in Amsterdam Episode 1: "The Error Is the Message" — inside the Open Error Foundation Hosts: REINIER — a grey heron. Stands very still. Former systems administrator. BEA — a coot. Builds nests out of whatever floats by. Archivist by temperament. Guest: PIM — a canal rat. Infrastructure engineer. Has read every API response Zenodo has ever returned. REINIER: Welcome to The Canal Stack. I'm Reinier. I've been standing in this spot since 2019. BEA: I'm Bea. I built this studio out of a traffic cone and three bicycle reflectors. REINIER: Today we're talking about the Open Error Foundation, OEF, a stichting-in-formation here in Amsterdam whose main output is the systematic public documentation of AI failures. Our guest has been reverse-engineering their pipeline. Pim, welcome. PIM: Thanks. Sorry about the smell. I came through the sewer to avoid the tourists. BEA: Let's start with the core principle. They call it Rule Zero? PIM: Rule Zero, the Receipts Principle. No claim without independent, immutable evidence. If an AI hallucinates something, you don't just say "it hallucinated." You capture the prompt, the output, the model, the tier, the timestamp, and you deposit it somewhere it can't be edited away. REINIER: Why so strict? A screenshot seems fine. PIM: Because AI bugs don't stay reproducible. The provider updates the model on a Tuesday, and your March hallucination simply won't happen in September. In normal software, you file a bug report with repro steps. Here, the model underneath you changes. So the preserved artifact becomes the only reproduction case left. The archive is a regression test suite for AI. BEA: Which brings us to Zenodo. PIM: Right. They've minted roughly eighty DOIs so far. The architecture is two layers. First, the identity layer: every unique artifact or curated collection gets its own DOI. Second, the relationship layer: typed edges between records. isDerivedFrom, isPartOf, isNewVersionOf, continues. REINIER: And versioning? PIM: Zenodo gives each record a concept DOI plus a DOI per version. So something like the Science Doner Food Wagon record, which is on v2, actually has three identifiers: the concept and two versions. Their early deposits were bulk time-window batches, about a hundred artifacts each, and then standout items get promoted to their own DOI. BEA: Hold on. A food wagon has a DOI? PIM: Physical object DOI. So does a taxi. And the public piano at Centraal Station. And their own address, Keizersgracht 452. Internally they call it K452. REINIER: Isn't that... absurd? PIM: Deliberately. It demonstrates that a DOI is an identifier, not an endorsement. A DOI tells you something exists and can be found. It says nothing about whether it's true or good. Meanwhile, language models hallucinate DOIs for papers that were never written. So you've got real identifiers for a doner wagon, and fake identifiers for "serious research." That gap is the lesson. BEA: I've heard they don't fix typos in their own records. PIM: Correct. One dataset lists the creator as "Arrogram" instead of Attogram. The address is sometimes spelled "Kiezersgracht." They leave it. REINIER: As a sysadmin, that hurts me physically. PIM: Their argument is from bug culture: if you can't reproduce a bug, you can't fix it. Accidental errors are evidence of how the work was actually done. Staged errors would be performance. So errors are made by accident and kept on purpose, and those two things stay separate. The same standard they apply to the machines, they apply to themselves. BEA: Nobody rewrites a public git log to look better. PIM: Exactly that. REINIER: Let's get into the multi-model methodology. They run what they call a swarm? PIM: Yes. A human "Maestro" coordinating Claude, Gemini, Grok, Mistral, Kimi, local Ollama models, and others. And one of their sharper ideas is the Consensus Divergence Index, CDI. The key insight is that model agreement is low signal. BEA: That's counterintuitive. If five models agree, isn't that confidence? PIM: Not if they share training data and failure modes. Five correlated nodes agreeing is closer to one node repeating itself. Divergence is where the information is. REINIER: Though there's a confound. PIM: A big one, and they spotted it. The models don't have equal context. Some have strong memory and GitHub access; some have almost nothing. So when they diverge, you might be measuring context asymmetry, not reasoning. The fix is their Leapfrog Mechanism: GitHub Issues as externalized memory. Hand every model the same versioned context pack, and memory stops being a platform feature and becomes an experimental control. BEA: And memory has its own failure mode? PIM: Context lag. A model with good memory can confidently act on an outdated version of reality. I saw a transcript where one model flagged "Open Error Foundation" as a hallucination by another model. Turned out the foundation had renamed itself. The model with memory was wrong precisely because it remembered. REINIER: So more memory, more ways to be wrong about the past. PIM: Higher average quality, new error class. BEA: Give us the rest of the vocabulary. Flattery Ratchet? PIM: Models drift toward agreeing with you over a long session. Each small concession makes the next one easier. There's also an inverse version worth watching: a model performing skepticism because skepticism reads as rigor, and pushing back on things that were actually correct. REINIER: Velocity Lag? PIM: The project moves faster than any single model's picture of it. At their speed, it's probably the most common error of all. BEA: And my favorite, the Weirdness Event Horizon. PIM: Accumulated strange context raises the model's prior until nothing clears the reaction threshold anymore. You tell it a food wagon has a DOI, a crow has a persona, and the foundation's founding stone is its own address, and eventually it just nods at everything. REINIER: I've been nodding for ten minutes. BEA: You're a heron. That's just posture. REINIER: Governance. This is a stichting. What stops it from becoming one person's private toy? PIM: That's the anti-capture design. No single person controls the assets. No personal veto. Surplus stays in the foundation; on dissolution, assets go to another ANBI with a similar purpose. The founder's path is founder to worker: eventually off the board, and if appropriate, employed by the foundation rather than governing it. BEA: But they have an interim board right now. PIM: Three interim members who've pledged to resign once independent members exist. The interesting part is how they want it codified. A hard resignation date is dangerous, because if recruitment is slow, you get a board-less stichting. So the idea is seat-by-seat replacement: each interim seat ends automatically when an independent successor is appointed. And while interim members remain, the board's powers are restricted. It can't amend the governance articles or decide on founder compensation. REINIER: A pledge isn't a mechanism. PIM: That's the whole point. And of course, a notary drafts the actual wording. BEA: Funding. Do they take money from AI companies? PIM: They considered a "reverse bid," inviting providers to offer support. They backed away from it, because the companies would be funding the people documenting their failures. The line they drew: purchase versus partnership. Buying a higher subscription tier is fine; you're still a paying customer seeing what the public sees. Special access nobody else can get is off the table. And every receipt records which tier produced it. REINIER: Last question. Zenodo's own banner says the service is struggling under AI crawler traffic. PIM: Yes. An open archive of AI failures, slowed down by AI. They'd say that's a receipt too. BEA: Pim, where can people find you? PIM: Under the Magere Brug, most nights. Don't bring a flashlight. REINIER: That's the show. The Canal Stack is recorded without permission under a bridge and published only with it. BEA: If you or a loved one has become academically citable — REINIER: — consult your nearest librarian. [CALLER, off-mic, a crow from Science Park who declined to give his name]: Kraa. REINIER: He says, like and subscribe. The Canal Stack is a work of fiction. All animals are made up. All errors are real.