Method & Quality

What is measured

Character Error Rate (CER) as the Levenshtein distance between pipeline TEI and reference TEI, normalized to the reference length. The reference is 25 ZBZ TEIs created manually in Transkribus. All values are computed over all 25 references (no document exclusion). The fidelity CER is the reliable quality figure; full text and scope are diagnostic quantities. The comparison takes the pipeline end product (output/tei_final/) against the human transcription, not the raw OCR text.

Text extraction with <choice> resolution, footnotes excluded, symmetric Unicode normalization (NFC, quotes, apostrophe, whitespace). The aggregation unit is the document, not the page; the full-text comparison is immune to page-numbering drift. The bootstrap (B = 10,000) yields 95 % confidence intervals via the percentile method, blockwise over documents.

Headline values (all n = 25)

Why three numbers? The ZBZ reference TEIs are in part selective partial transcriptions (for example only one of several reviews on a page); the pipeline is often more complete. Large text insertions by the pipeline (≥ 50 characters) are therefore not errors but surplus text (scope). The fidelity CER measures only the real errors and is the reliable quality figure: fidelity + scope = full text. The Transkribus quality bands (< 2 % excellent, 2–5 % good) stem primarily from handwriting recognition practice and flatter a pure print OCR task; decisive is the print OCR literature comparison below. The fidelity median (1.28 %) is solid for historical print but not at the top.

The earlier figure of −14.83 pp for the pipeline gain was an artifact of the trimmed comparison and is retracted; after the reference-verified footnote demotions (E85) and the stock corrections (E94–E98) the gain is −10.08 pp and statistically significant (p = 0.0034, n = 25, 17/25 documents improved). The earlier scope-cleaned subset selection (n = 19) was removed: it followed no reproducible criterion. All values now compute over all 25 references.

Stratified (full-text diagnosis, scope included, n = 25)

The following values are the full-text CER (including pipeline surplus text), a diagnostic quantity, not a quality measure. They sit clearly above the fidelity headline because the ZBZ references are in part selective partial transcriptions and the pipeline surplus text counts as deviation here. A fidelity-exact breakdown per stratum is currently not available.

Two-column print (type B) remains the hardest (highest median), mostly column recognition at the layout stage. Such documents are reprocessing candidates, for example python -m scripts.layout.layout_qa_gemini --mode detect --doc {ID}.

Comparison with the research literature (printed historical documents)

The Hersch median (1.28 %) lies between Greif's best stack (Transkribus + LLM post-correction, 0.84 %) and Transkribus alone (3.67 %), better than Gemini 2.5 Pro zero-shot on comparable material. Only sources from 2025 onward are cited; the complete list is in the repo.

Limitations

• n = 25: measurement runs against 25 ZBZ reference TEIs; corpus statements beyond these 25 documents (full corpus 285) are estimates, not measurements.
• Selection bias n_chars (p = 0.0155): the reference subset deviates significantly from the full corpus in character volume. The median CER (volume-independent) is less affected than the mean CER.
• Stability measured on a pilot only: the LLM stages are non-deterministic. A pilot (5 documents, 3 runs each, 2026-07-07) measured a mean run-to-run standard deviation of 0.04 pp fidelity CER (max 0.13 pp, max range 0.25 pp); the variance is negligible at this scale, but the pilot covers 5 of the 25 reference documents.
• HCPR is frequency-based: it compares diacritics counts, not positions, and underestimates substitution errors.
• Multi-normalization regimes differ little: the upstream normalization makes the nominal normalization levels practically identical.
• The corpus proxy does not generalize: the composite OLS from the dictionary hit rate has LOOCV R² < 0. The corpus estimate of 21.96 % mean (full-text scale, scope included) is a plausibility bound, not an estimator.
• No inter-engine comparison: only one OCR engine run (Mistral), no second source as cross-validation.

The honest disclosure of these points is part of the methodology. Whoever cites the headline numbers should cite the limitations with them.

Tools & reproducibility

Statistics library

scripts/eval/cer_statistics.py

Orchestrator

scripts/eval/cer_statistics_full.py

Test suite

python -m pytest tests/test_cer_statistics.py -q

JSON generation

python -m scripts.eval.cer_statistics_full --seed 42 --bootstrap-n 10000

Output

docs/data/cer_statistics.json (deterministic at seed 42, versioned in the repo)

Methodology docs

knowledge/methodology.md (method), docs/project-report.md (values)

Schema

data/schema/zbz_hersch.rng (project-specific RelaxNG, the format authority for the delivered TEI)

Research literature (selection, all 2025+)

• Du 2025, "When +1% Is Not Enough", arXiv:2511.19794: paired-bootstrap protocol
• Levchenko 2025, arXiv:2510.06743 (peer-reviewed: LM4DH 2025 workshop at RANLP 2025, Varna, pp. 75–85, DOI 10.26615/978-954-452-106-6-007): HCPR/AIR adaption, no-ground-truth methodology, stability tests
• Greif, Griesshaber & Greif 2025, arXiv:2504.00414: multimodal LLMs for OCR and OCR post-correction on German-language address books (the printed-OCR comparison values above)
• Kanerva & Ledins 2025, arXiv:2502.01205: language dependence of LLM post-correction ("No Free Lunches")