Skip to content

[models] Add formula recognition (FormulaNet) with LaTeX export, HGNetV2 backbone and training references - #2227

Merged
felixdittrich92 merged 6 commits into
mindee:mainfrom
felixdittrich92:formula
Nov 17, 2026
Merged

felixdittrich92 merged 6 commits into
mindee:mainfrom
felixdittrich92:formula

Conversation

@felixdittrich92

@felixdittrich92 felixdittrich92 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR adds formula recognition to docTR: a new formula task that reads the Formula regions found by the layout model and returns their LaTeX source. It covers the model, the OCR pipeline, every export format, the reference training scripts, the API, the demo and the docs.

from doctr.models import ocr_predictor

predictor = ocr_predictor(pretrained=True, detect_formulas=True)
result = predictor(doc)
result.pages[0].formulas[0].value  # '\\frac { a } { b ^ { 2 } }'
result.export_as_markdown()        # formulas exported as $$ ... $$

The implementation is written from scratch under Apache-2.0. It follows the PP-FormulaNet design (HGNetV2 encoder with a small autoregressive decoder) and copies no code, vocabulary or weights from AGPL-licensed projects such as Texo.

Model

  • hgnetv2_b4 (new, doctr/models/classification/hgnetv2): an HGNetV2-B4 backbone written as an nn.Sequential like the other classification models, with pretrained weights.
  • formulanet (new, doctr/models/formula), 26.4M parameters:
    • The HGNetV2-B4 feature map (stride 32) is flattened, projected to 384 dimensions and given 2D sinusoidal positional encodings.
    • A 4-layer transformer decoder (16 heads, FFN 1536) decodes greedily with a key/value cache.
    • The input is 256×768, wide rather than square, so long equations are not shrunk. A 768×768 input was tried and did not improve the results.
    • max_length=4096 is the longest label the model is trained on; max_decode_length=2048 caps the tokens generated at inference.
  • formula_predictor / FormulaPredictor: standalone predictor, plus predict_regions(pages, layout_regions) to read formulas straight from layout predictions.

Tokenization

  • LATEX_TOKENS: a fixed list of 1,324 tokens, with no BPE. It holds the printable ASCII characters, the control symbols, the commands and environments supported by KaTeX, and every other token of the UniMER-1M / HME100K / UniMER-Test labels, so those datasets are fully covered.
  • tokenize_latex / normalize_latex: split any LaTeX string into these tokens. The model predicts the tokens separated by spaces, e.g. \frac { a } { b ^ { 2 } }.
  • Token-list vocabs: encode_string, decode_sequence and encode_sequences now also accept a list of tokens, not only a string of characters.
  • Removed VOCABS["latex"], which LATEX_TOKENS replaces.

Pipeline and exports

  • Options: ocr_predictor(detect_formulas=True), also as --detect_formulas in the CLI. It enables the layout model, masks the Formula regions out of text detection and reads them with the formula model.
  • Elements:
    • new Formula element with value, confidence, geometry and objectness_score
    • new Page.formulas attribute, included in export() / from_dict()
  • Reading order: formulas are placed in the text flow.
  • Coordinates: they are remapped to the original page with straighten_pages + preserve_original_coords.

Export formats:

Format Rendering
render() / text the LaTeX source
Markdown $$ / LaTeX / $$ on separate lines
HTML <div class="formula">$$...$$</div> (HTML-escaped)
AsciiDoc [latexmath] passthrough block
hOCR ocr_math element, declared in ocr-capabilities
JSON formulas key of each page

Other additions

  • Metric: FormulaMetric (doctr/utils/metrics.py) reports exact match, token edit similarity and corpus BLEU-4, all computed on LaTeX tokens.
  • Transform: RandomPad (doctr.transforms) pads an image with its background color and keeps box and polygon targets aligned. FormulaNet uses it in training so the model reads layout crops with their margins as they are.
  • Hub: formula is a supported task for push_to_hf_hub / from_hub.
  • API: detect_formulas in the /ocr request, and a formulas list in the response.
  • Demo: "Detect formulas" checkbox; formulas are shown rendered with st.latex next to their source.
  • References (references/formula/):
    • train.py with DDP, AMP, warmup, --save-interval-epoch (as in the other training scripts) and a --max-tokens option that splits long padded batches into gradient-accumulated chunks
    • evaluate.py, latency.py and build_vocab.py
    • a README
    • train / evaluate / latency jobs in references.yml, run on a toy set

Training data and KaTeX compatibility

The model is trained on UniMER-1M plus the HME100K train split (1.06M formulas). Before training, the labels were canonicalized with rewrites that keep the rendering the same:

  • One spelling per formula: a bare script argument gets braces (x ^ 2 → x ^ { 2 }), and aliases take one spelling (\le → \leq, \to → \rightarrow...).
  • KaTeX-compatible commands: commands KaTeX does not support are replaced by the KaTeX equivalent with the same LaTeX definition:
    • \eqno ( n ) → \tag { $ n $ }
    • \ensuremath { x } → x
    • \Tilde / \Bar / \Vec... → \tilde / \bar / \vec...
    • \hdots → \ldots, \slash → /

So the model predicts equation numbers as \tag, which renders in KaTeX-based Markdown and HTML viewers. After the rewrite, KaTeX renders 99.34% of the training labels. The rest use commands with no KaTeX equivalent (e.g. \boldmath, \upmu, \lefteqn), as documented on the model page.

Benchmark

Full UniMER-Test (23,757 formulas, labels canonicalized as above), greedy decoding:

Subset Exact match Edit similarity BLEU
SPE (simple printed) 47.59 94.27 0.885
CPE (complex printed) 25.22 93.54 0.847
SCE (screen-captured) 37.52 80.53 0.502
HWE (handwritten) 64.07 94.11 0.896
All 44.40 91.30 0.840

End-to-end on five arXiv papers (1706.03762, 2410.12628, 2501.02962, 2503.17213, 2512.02498):

  • Recognition: all 21 Formula regions are read, at 0.95–0.99 confidence.
  • Rendering: all 21 formulas render in KaTeX, with equation numbers as \tag.
  • Exports: every export format checks out, including a JSON round trip.

Tests

  • New: formula model, predictor, ONNX export, torch.compile, pipeline, export, CLI, API, metric, tokenizer and RandomPad tests.
  • Results: tests/common 833 passed; the formula, zoo, HGNetV2, transforms and metric tests of tests/pytorch 162 passed; ruff is clean.

@felixdittrich92 felixdittrich92 added this to the 1.2.0 milestone Nov 16, 2026
@felixdittrich92 felixdittrich92 self-assigned this Nov 16, 2026
@felixdittrich92 felixdittrich92 added topic: documentation Improvements or additions to documentation topic: ci Related to CI module: models Related to doctr.models module: io Related to doctr.io ext: tests Related to tests folder benchmark This issue reports a benchmark ext: demo Related to demo folder ext: references Related to references folder module: transforms Related to doctr.transforms ext: api Related to api folder framework: pytorch Related to PyTorch backend type: new feature New feature ext: docs Related to docs folder topic: formula recognition Related to the task of formula recognition labels Nov 16, 2026
@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.01800% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.21%. Comparing base (dbcee0d) to head (91cc417).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
doctr/models/factory/hub.py 0.00% 2 Missing ⚠️
doctr/models/formula/formulanet/base.py 95.23% 1 Missing ⚠️
doctr/models/formula/formulanet/pytorch.py 99.48% 1 Missing ⚠️
doctr/models/formula/predictor/pytorch.py 98.24% 1 Missing ⚠️
doctr/models/predictor/base.py 90.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2227      +/-   ##
==========================================
+ Coverage   98.16%   98.21%   +0.05%     
==========================================
  Files         172      181       +9     
  Lines       11977    12578     +601     
==========================================
+ Hits        11757    12354     +597     
- Misses        220      224       +4     
Flag Coverage Δ
unittests 98.21% <99.01%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@felixdittrich92
felixdittrich92 merged commit 2459abc into mindee:main Nov 17, 2026
57 of 60 checks passed
@felixdittrich92
felixdittrich92 deleted the formula branch November 17, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmark This issue reports a benchmark ext: api Related to api folder ext: demo Related to demo folder ext: docs Related to docs folder ext: references Related to references folder ext: tests Related to tests folder framework: pytorch Related to PyTorch backend module: io Related to doctr.io module: models Related to doctr.models module: transforms Related to doctr.transforms topic: ci Related to CI topic: documentation Improvements or additions to documentation topic: formula recognition Related to the task of formula recognition type: new feature New feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant