Pipeline viewer for the Jeanne Hersch papers

An internal inspection and correction tool for the OCR, layout, and TEI results produced in the digital cataloguing of the Hersch papers at the Zentralbibliothek Zurich. Not a public reader and not an edition frontend; ZBZ covers that via Oxygen, Alma, and Swisscovery.

What this tool does

Inspection: facsimile, layout regions, and transcription side by side, for every page of every document.
Manual correction: layout editor (move regions, set types, reorder), text editor (OCR or TEI); one save writes directly into the local repo clone, with file download as fallback.
Entity preview: GND-linked persons, organisations and works as a read-only layer with a review worklist; the delivered TEI does not carry the annotations yet.
Workflow status: traffic light per stream (OCR / Layout / TEI / Entities), kept in the per-document manifest with a processing history (E66/E67).
Demonstration: a clear view of the current pipeline state for ZBZ and academic stakeholders.

Project

Commissioned by: Zentralbibliothek Zurich (ZBZ)
Contractor: DHCraft (Digital Humanities Craft OG)
Subject: Automated OCR + TEI annotation for the Hersch papers
Corpus: 286 PDF texts delivered, 285 with final TEI; ~4,100 processed pages; focus on the 1970s/1980s
Languages: predominantly French (~70 %), German (~25 %), occasionally English and Italian
Status: 285/285 documents schema-valid; the workflow status per stream (OCR / Layout / TEI / Entities) reads “unverified” as the handover default: pipeline output exists, scholarly verification is open

Pipeline

Images: scripts/edition/extract_pages.py produces PNGs from the PDF scans.
OCR: Mistral Document AI 2512 (Azure).
Layout: Docling 2.75 for region recognition, Gemini 3.1 Flash Lite as QA/detect validator.
TEI: unified pipeline of scaffold, Gemini refinement, and assembly; valid against zbz_hersch.rng.
Entities: deterministic GND matching against the curated ZBZ list into read-only previews plus a review worklist (delivered TEI untouched).
Workflow status: status per stream (OCR / Layout / TEI / Entities) in the manifest, projected into <revisionDesc> (E66/E67, replaces the earlier agent screening).

Quality

Character Error Rate (CER) in the end-to-end comparison against the ZBZ reference TEIs. Fidelity CER (real OCR and transcription errors): mean 2.08 %, median 1.28 % (n = 25, percentile bootstrap, B = 10,000, seed 42). The full-text CER including pipeline surplus text sits clearly higher; why, plus all stratified values, limitations, and the literature comparison: see Method.

Architecture

Static single-page apps: the corpus overview as entry point, the viewer for the detail view of one document, and the entity overview for the corpus-wide entity picture. Data is loaded exclusively from static JSON/XML/MD files, no backend. Editor changes are saved directly into the local repo clone via the File System Access API (Chromium), with file download as the fallback. The viewer is split into the modules viewer-state, viewer-entities, viewer-status, viewer-persist, viewer-page and the viewer shell, loaded as classic scripts around the shared ZBZ.Viewer namespace. Fonts and OpenSeadragon are served from this repository, so the site requests no third-party resources.

Data: catalog.json plus a per-page mirror under data/pages/ (all 285 docs), thumbnails under data/thumbs/, per-document manifests under data/manifests/ with the aggregated data/manifest_index.json the corpus overview reads, the entity lookup data/entities.json, the entity overview aggregate data/entity_overview.json, and the CER statistics data/cer_statistics.json.

Contact

Project lead: DHCraft · office@dhcraft.org
ZBZ: project management, Zentralbibliothek Zurich
Source code: github.com/chpollin/zbz-ocr-tei