Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .github/workflows/references.yml
Original file line number Diff line number Diff line change
Expand Up @@ -296,3 +296,64 @@ jobs:
python-version: ${{ matrix.python }}
- name: Benchmark latency
run: python references/table/latency.py tablecenternet --it 5 --size 512

train-formula-recognition:
runs-on: ${{ matrix.os }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python: ["3.11"]
steps:
- uses: actions/checkout@v7
- name: Setup environment
uses: ./.github/actions/setup-env
with:
python-version: ${{ matrix.python }}
extra-requirements: references/requirements.txt
- name: Download and extract toy set
run: |
wget https://github.com/mindee/doctr/releases/download/v1.1.0/toy_formula_set-8f54029b.zip
unzip toy_formula_set-8f54029b.zip -d formula_set
- name: Train for a short epoch
run: python references/formula/train.py formulanet --train_path ./formula_set --val_path ./formula_set -b 2 --epochs 1 --max_decode_length 64

evaluate-formula-recognition:
runs-on: ${{ matrix.os }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python: ["3.11"]
steps:
- uses: actions/checkout@v7
- name: Setup environment
uses: ./.github/actions/setup-env
with:
python-version: ${{ matrix.python }}
extra-requirements: references/requirements.txt
- name: Download and extract toy set
run: |
wget https://github.com/mindee/doctr/releases/download/v1.1.0/toy_formula_set-8f54029b.zip
unzip toy_formula_set-8f54029b.zip -d formula_set
- name: Evaluate formula recognition
run: python references/formula/evaluate.py formulanet ./formula_set -b 2 --max_decode_length 64

latency-formula-recognition:
runs-on: ${{ matrix.os }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python: ["3.11"]
steps:
- uses: actions/checkout@v7
- name: Setup environment
uses: ./.github/actions/setup-env
with:
python-version: ${{ matrix.python }}
- name: Benchmark latency
run: python references/formula/latency.py formulanet --it 5 --tokens 32
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,10 @@ Credits where it's due: this repository is implementing, among others, architect

- TableCenterNet: [A one-stage network for table structure recognition](https://arxiv.org/abs/2504.17522).

### Formula Recognition

- FormulaNet (HGNetV2 encoder + transformer decoder): [PP-FormulaNet: Bridging Accuracy and Efficiency in Advanced Formula Recognition](https://arxiv.org/abs/2503.18382).

## More goodies

### Documentation
Expand Down
22 changes: 21 additions & 1 deletion api/app/routes/ocr.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,18 @@

from fastapi import APIRouter, Depends, File, HTTPException, UploadFile, status

from app.schemas import LayoutElementOut, OCRBlock, OCRIn, OCRLine, OCROut, OCRPage, OCRWord, TableCellOut, TableOut
from app.schemas import (
FormulaOut,
LayoutElementOut,
OCRBlock,
OCRIn,
OCRLine,
OCROut,
OCRPage,
OCRWord,
TableCellOut,
TableOut,
)
from app.utils import get_documents, resolve_geometry
from app.vision import init_predictor

Expand Down Expand Up @@ -63,6 +74,15 @@ async def perform_ocr(request: OCRIn = Depends(), files: list[UploadFile] = [Fil
)
for table in page.tables
],
formulas=[
FormulaOut(
value=formula.value,
geometry=resolve_geometry(formula.geometry),
confidence=round(formula.confidence, 2),
objectness_score=(None if formula.objectness_score is None else round(formula.objectness_score, 2)),
)
for formula in page.formulas
],
items=[
OCRPage(
blocks=[
Expand Down
21 changes: 21 additions & 0 deletions api/app/schemas.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ class OCRIn(KIEIn, BaseModel):
resolve_blocks: bool = Field(default=False, examples=[False])
paragraph_break: float = Field(default=0.0035, examples=[0.0035])
detect_tables: bool = Field(default=False, examples=[False])
detect_formulas: bool = Field(default=False, examples=[False])


class RecognitionIn(BaseModel):
Expand Down Expand Up @@ -173,6 +174,13 @@ class TableOut(BaseModel):
)


class FormulaOut(BaseModel):
value: str = Field(..., examples=["\\frac { a } { b }"])
geometry: list[float] = Field(..., examples=[[0.0, 0.0, 0.0, 0.0]])
confidence: float = Field(..., examples=[0.99])
objectness_score: float | None = Field(..., examples=[0.99])


class OCROut(BaseModel):
name: str = Field(..., examples=["example.jpg"])
orientation: dict[str, float | None] = Field(..., examples=[{"value": 0.0, "confidence": 0.99}])
Expand Down Expand Up @@ -207,6 +215,19 @@ class OCROut(BaseModel):
]
],
)
formulas: list[FormulaOut] = Field(
default=[],
examples=[
[
{
"value": "\\frac { a } { b }",
"geometry": [0.0, 0.0, 0.0, 0.0],
"confidence": 0.99,
"objectness_score": 0.99,
}
]
],
)
items: list[OCRPage] = Field(
...,
examples=[
Expand Down
35 changes: 34 additions & 1 deletion api/tests/routes/test_ocr.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import numpy as np
import pytest

from doctr.io.elements import Document, Page, Table, TableCell
from doctr.io.elements import Document, Formula, Page, Table, TableCell


def common_test(json_response, expected_response):
Expand Down Expand Up @@ -144,6 +144,39 @@ async def test_ocr_table_cells(test_app_asyncio, mock_detection_image, monkeypat
assert [cell["objectness_score"] for cell in out_cells] == [0.81, None]


@pytest.mark.asyncio
async def test_ocr_formulas(test_app_asyncio, mock_detection_image, monkeypatch):
# The recognized formulas are returned with their LaTeX source, and `detect_formulas` reaches the predictor
formulas = [
Formula("\\frac { a } { b }", 0.9876, ((0.1, 0.1), (0.5, 0.2)), objectness_score=0.8123),
Formula("x ^ { 2 }", 0.5, ((0.1, 0.3), (0.5, 0.4))),
]
doc = Document([Page(np.zeros((100, 100, 3), dtype=np.uint8), [], 0, (100, 100), formulas=formulas)])
requests = []

def fake_init_predictor(request):
requests.append(request)
return lambda pages: doc

monkeypatch.setattr("app.routes.ocr.init_predictor", fake_init_predictor)
files = [("files", ("test.jpg", mock_detection_image, "image/jpeg"))]
response = await test_app_asyncio.post(
"/ocr", params={"detect_formulas": True}, files=files, headers={"accept": "application/json"}
)
assert response.status_code == 200
assert requests[-1].detect_formulas is True
assert response.json()[0]["formulas"] == [
{"value": "\\frac { a } { b }", "geometry": [0.1, 0.1, 0.5, 0.2], "confidence": 0.99, "objectness_score": 0.81},
{"value": "x ^ { 2 }", "geometry": [0.1, 0.3, 0.5, 0.4], "confidence": 0.5, "objectness_score": None},
]

# No formulas by default
doc = Document([Page(np.zeros((100, 100, 3), dtype=np.uint8), [], 0, (100, 100))])
response = await test_app_asyncio.post("/ocr", files=files, headers={"accept": "application/json"})
assert requests[-1].detect_formulas is False
assert response.json()[0]["formulas"] == []


@pytest.mark.asyncio
async def test_ocr_invalid_file(test_app_asyncio, mock_txt_file):
headers = {
Expand Down
11 changes: 11 additions & 0 deletions demo/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,8 @@ def main(det_archs, reco_archs, layout_archs):
layout_arch = st.sidebar.selectbox("Layout detection model", layout_archs, disabled=not detect_layout)
# Table detection (relies on the layout model to locate tables)
detect_tables = st.sidebar.checkbox("Detect tables", value=False)
# Formula recognition (relies on the layout model to locate formulas)
detect_formulas = st.sidebar.checkbox("Detect formulas", value=False)
st.sidebar.write("\n")
# Binarization threshold
bin_thresh = st.sidebar.slider("Binarization threshold", min_value=0.1, max_value=0.9, value=0.3, step=0.1)
Expand Down Expand Up @@ -101,6 +103,7 @@ def main(det_archs, reco_archs, layout_archs):
detect_layout=detect_layout,
layout_arch=layout_arch,
detect_tables=detect_tables,
detect_formulas=detect_formulas,
)

with st.spinner("Analyzing..."):
Expand Down Expand Up @@ -133,6 +136,14 @@ def main(det_archs, reco_archs, layout_archs):
st.markdown(f"**Table {idx + 1}** ({table.num_rows} x {table.num_cols})")
st.dataframe(pd.DataFrame(table.to_grid()))

# Display recognized formulas (if any): rendered, and their LaTeX source
if out.pages[0].formulas:
st.markdown("\nRecognized formulas:")
for idx, formula in enumerate(out.pages[0].formulas):
st.markdown(f"**Formula {idx + 1}** (confidence {formula.confidence:.2f})")
st.latex(formula.value)
st.code(formula.value, language="latex")

# Display JSON
st.markdown("\nHere are your analysis results in JSON format:")
st.json(page_export, expanded=False)
Expand Down
4 changes: 4 additions & 0 deletions demo/backend/pytorch.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ def load_predictor(
detect_layout: bool,
layout_arch: str,
detect_tables: bool,
detect_formulas: bool = False,
) -> OCRPredictor:
"""Load a predictor from doctr.models

Expand All @@ -68,6 +69,8 @@ def load_predictor(
detect_layout: whether to run a layout detection model and attach the regions to each page
layout_arch: layout architecture to use when detect_layout is True
detect_tables: whether to detect tables (via the layout model), structure them and attach them to each page
detect_formulas: whether to read the formulas (located via the layout model) as LaTeX and attach them to each
page

Returns:
instance of OCRPredictor
Expand All @@ -85,6 +88,7 @@ def load_predictor(
detect_layout=detect_layout,
layout_arch=layout_arch,
detect_tables=detect_tables,
detect_formulas=detect_formulas,
).to(device)
predictor.det_predictor.model.postprocessor.bin_thresh = bin_thresh
predictor.det_predictor.model.postprocessor.box_thresh = box_thresh
Expand Down
11 changes: 11 additions & 0 deletions docs/source/api/datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,13 +72,24 @@ for a guide.
:type: dict[str, str]

The supported vocabularies, by name. See :doc:`../models/vocabs` for the complete list.

.. data:: LATEX_TOKENS
:type: list[str]

The 1,324 LaTeX tokens of formula recognition: the printable ASCII characters, the control symbols, the commands and
environments supported by KaTeX, and the other commands and characters found in public formula datasets (UniMER-1M,
HME100K, UniMER-Test). See :doc:`../models/formula`.
```

## Utilities

```{eval-rst}
.. autofunction:: translate

.. autofunction:: tokenize_latex

.. autofunction:: normalize_latex

.. autofunction:: encode_string

.. autofunction:: decode_sequence
Expand Down
4 changes: 3 additions & 1 deletion docs/source/api/io.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ The elements returned by the predictors, from the largest to the smallest. See
.. autoclass:: Artefact
```

### Layout and tables
### Layout, tables and formulas

```{eval-rst}
.. autoclass:: LayoutElement
Expand All @@ -105,6 +105,8 @@ The elements returned by the predictors, from the largest to the smallest. See

.. autoclass:: TableCell
:members: row_span, col_span

.. autoclass:: Formula
```

### Key information extraction
Expand Down
13 changes: 13 additions & 0 deletions docs/source/api/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,17 @@ directly. See the [model zoo](../models/index.md) for benchmarks.
.. autofunction:: doctr.models.table_structure.tablecenternet
```

## Formula recognition

```{eval-rst}
.. autofunction:: doctr.models.formula.formula_predictor

.. autoclass:: doctr.models.formula.predictor.FormulaPredictor
:members: predict_regions

.. autofunction:: doctr.models.formula.formulanet
```

## Classification

Orientation classifiers, and the backbones used by the detection and recognition models.
Expand Down Expand Up @@ -146,6 +157,8 @@ Orientation classifiers, and the backbones used by the detection and recognition
.. autofunction:: doctr.models.classification.vit_det_m

.. autofunction:: doctr.models.classification.starnet_s3

.. autofunction:: doctr.models.classification.hgnetv2_b4
```

## Hugging Face Hub
Expand Down
2 changes: 2 additions & 0 deletions docs/source/api/transforms.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ img_transforms = Compose([

.. autoclass:: RandomResize

.. autoclass:: RandomPad

.. autoclass:: GaussianNoise

.. autoclass:: GaussianBlur
Expand Down
3 changes: 3 additions & 0 deletions docs/source/api/utils.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,7 @@ Metrics to evaluate models on each task, used by the evaluation scripts of the r

.. autoclass:: TableCellMetric
:members: update, summary

.. autoclass:: FormulaMetric
:members: update, summary
```
2 changes: 1 addition & 1 deletion docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@
# Vocabularies listed on `models/vocabs.md`, grouped by script. Any vocabulary missing from these groups is listed
# under "Other scripts", so that the page always reflects `doctr.datasets.VOCABS`.
VOCAB_GROUPS = {
"General purpose": ["latin", "english", "multilingual", "latex"],
"General purpose": ["latin", "english", "multilingual"],
"Latin script": [
"albanian",
"afrikaans",
Expand Down
1 change: 1 addition & 0 deletions docs/source/deployment/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ from doctr.models import ocr_predictor
ocr_predictor(pretrained=True) # detection + recognition
ocr_predictor(pretrained=True, assume_straight_pages=False) # + orientation classifiers
ocr_predictor(pretrained=True, detect_tables=True) # + layout and table models
ocr_predictor(pretrained=True, detect_formulas=True) # + layout and formula models
```

Copy the cache directory (`~/.cache/doctr` or `$DOCTR_CACHE_DIR`) to the same location on the offline machine.
Expand Down
3 changes: 2 additions & 1 deletion docs/source/deployment/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,8 @@ Pass all the pages of a document (or several documents) in **one call** so that
the more expensive rotated-box post-processing.
- With rotated documents, `disable_page_orientation=True` and `disable_crop_orientation=True` skip the orientation
classifiers when you know the text is horizontal (see [](../user_guide/rotated_documents.md)).
- Only enable `detect_layout`, `detect_tables`, `detect_orientation` and `detect_language` when you use them.
- Only enable `detect_layout`, `detect_tables`, `detect_formulas`, `detect_orientation` and `detect_language` when you
use them. The formula model decodes its LaTeX token by token: its latency grows with the length of the formulas.
- Use lighter architectures, see the [model zoo](../models/index.md).

## Compile the models
Expand Down
Loading
Loading