Skip to content

[models] Speed up pre/post-processing, fuse attention and cache MASTER/PARSeq decoding, fix a crash on 1-px word crops - #2228

Merged
felixdittrich92 merged 5 commits into
mindee:mainfrom
felixdittrich92:perf-and-fixes
Nov 17, 2026
Merged

felixdittrich92 merged 5 commits into
mindee:mainfrom
felixdittrich92:perf-and-fixes

Conversation

@felixdittrich92

@felixdittrich92 felixdittrich92 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes and speed-ups from an end-to-end sweep of the library. It ran 5 predictor configurations (default, layout + tables + formulas, rotated/straightened, rotated + layout, KIE) over ~120 real images and 10 PDFs, and exercised every exporter on each result. Predictions are unchanged throughout (see Verification).

Fixes

  • Crash with assume_straight_pages=False on a degenerate word crop. np.rot90 of a 1-pixel-high crop has a negative stride on an axis of size 1. NumPy still flags it as C-contiguous, so np.ascontiguousarray returns it unchanged, and torch.from_numpy rejects it. The whole document failed. PreProcessor now copies such arrays.
  • doctr.io.tensor_from_numpy failed on any flipped or rotated view (e.g. img[::-1]), for every dtype. It now copies them first.
  • SVT / IC03: the XML element text is checked before use. An entry without a file name now raises the existing FileNotFoundError instead of crashing in os.path.join. In SVT's recognition mode, a word without text is skipped instead of raising a TypeError on len(None). This also fixes 19 mypy errors that only show up when types-tqdm is installed, as it is in some dev environments: the tqdm(...) loop then keeps the Element type instead of Any.
  • PyTorch layout tests were skipped silently. tests/common/test_models_layout.py and tests/pytorch/test_models_layout.py shared a basename, so collecting both folders in one session raised an "import file mismatch" error. Renamed the latter to test_models_layout_pt.py.

Performance

  1. No thread pool in PreProcessor and DetectionPostProcessor. The per-item work (word crop resizing, per-box contour processing) is bound by Python overhead, so threads only competed for the GIL and oversubscribed the cores on top of OpenCV/PyTorch's own threads:

    • Crop preprocessing: 1.05 s → 0.58 s on 6,347 crops
    • Detection post-processing: 2–5× faster
    • Large page resizes are already parallelized by PyTorch.
  2. modules/transformer: attention uses F.scaled_dot_product_attention. Recognition gets 5–16% faster for PARSeq / ViTSTR / MASTER. scaled_dot_product_attention keeps its signature but returns None in place of the attention weights, which nothing used.

  3. Incremental, KV-cached autoregressive decoding for MASTER and PARSeq. The image features are projected once per batch, and each step only processes the newest token, with the previous ones cached. Before, every step re-ran the decoder over the whole sequence. MASTER did so over all 50 positions, re-projecting the encoder memory of its 3 layers each time. A token only attends to the previous ones, so the result is the same.

    • It also applies to the ONNX export (legacy exporter, same inputs/outputs). The traced graph still runs every step, since tracing cannot express an early stop, but each step is now cheap.
    • New MultiHeadAttention.project_kv / attend and an optional cache in the transformer Decoder; PARSeqDecoder accepts precomputed content_kv / memory_kv.
    • Training and teacher-forced validation still go through the full-sequence path.
    1,024 real crops before after
    MASTER, ONNX CPU 84.1 s 16.7 s (5.0×)
    MASTER, PyTorch GPU / CPU 2.45 s / 48.5 s 1.06 s / 18.5 s (2.3× / 2.6×)
    PARSeq, ONNX CPU 8.4 s 7.5 s (1.12×)
    PARSeq, PyTorch GPU / CPU 0.82 s / 10.9 s 0.65 s / 9.5 s (1.27× / 1.14×)

    MASTER's ONNX export also takes 8 s instead of 123 s.

    MASTER notes:

    • The logits keep their shape (N, max_length, vocab_size + 1).
    • After an early stop, the positions beyond every sequence's EOS are zeros instead of logits computed from padding tokens. They are never read by the post-processor.
    • The last position is now predicted from the last decoded token rather than from a padding token. This only matters for a word without EOS in 49 characters.

End-to-end latency per page from items 1 and 2 (RTX 4080, db_mobilenet_v3_large + parseq, alternating runs of main vs this branch):

Config Input Before After Change
default arXiv PDF (12 p) 459 ms 389 ms −15%
default scanned PDF (6 p) 114 ms 93 ms −19%
default photos (8) 261 ms 229 ms −12%
default receipts (12) 141 ms 118 ms −16%
layout + tables + formulas arXiv PDF (12 p) 517 ms 433 ms −16%
layout + tables + formulas photos (8) 343 ms 313 ms −9%
CPU, crnn_mobilenet_v3_small photos (8) 364 ms 323 ms −11%

Tests

  • tests/conftest.py, pytest-xdist only:
    • Each worker gets cpu_count // num_workers PyTorch/OpenCV threads. Before, every worker started one thread per core: test_add_whitelist*[master] took up to 1064 s with -n 8 versus ~2 s alone.
    • Cached GPU memory is released after each test, so the workers can share one GPU. With -n 4 the suite had 29 CUDA out-of-memory failures before; now it is green.
    • Full suite with -n 8: 41 min → 1 min 54 s, 1366 passed. CI runs serially and is unaffected.
  • Refactors, with all assertions kept:
    • test_resize (174 lines) → 7 focused, parametrized tests.
    • test_documentbuilder_tables (139 lines) → 6 tests, one per scenario.
    • The LayoutDataset invalid-label cases write their labels to tmp_path instead of overwriting the session-wide fixture.
    • The dataset "missing image" checks use a _moved_away context manager, so the shared folder is restored even if a check fails.
  • New regression tests: test_preprocessor_views and test_tensor_from_numpy_views. Both fail without the fixes.

Verification

  • Pipeline: predictions identical to main on 66 document × config runs (words, geometries, tables, layout).
  • Recognition: words identical on 6,347 real crops for PARSeq, ViTSTR-small/base and MASTER (confidence diff ≤ 2.4e-5).
  • ONNX: export and onnxruntime parity tests pass for every affected model; ONNX outputs give the same words as PyTorch on 1,024 crops.
  • add_whitelist on the full ocr_predictor: 17 mostly multilingual documents, with mask (German, Polish + German), nearest (anyascii and mapping="weights"), the context manager and remove(). Same words as main for PARSeq and MASTER, and no character outside the whitelist.
  • Training: MASTER / PARSeq losses, gradients and weights after 3 AdamW steps are bit-identical to main on CPU. On CUDA they differ only within the run-to-run noise of main itself.
  • make quality is clean.

@felixdittrich92 felixdittrich92 self-assigned this Nov 17, 2026
@felixdittrich92 felixdittrich92 added this to the 1.2.0 milestone Nov 17, 2026
@felixdittrich92 felixdittrich92 added type: bug Something isn't working type: enhancement Improvement module: models Related to doctr.models module: io Related to doctr.io ext: tests Related to tests folder labels Nov 17, 2026
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.93814% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.24%. Comparing base (2459abc) to head (2702b34).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
doctr/datasets/ic03.py 83.33% 1 Missing ⚠️
doctr/datasets/svt.py 85.71% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2228      +/-   ##
==========================================
+ Coverage   98.21%   98.24%   +0.03%     
==========================================
  Files         181      181              
  Lines       12578    12624      +46     
==========================================
+ Hits        12354    12403      +49     
+ Misses        224      221       -3     
Flag Coverage Δ
unittests 98.24% <97.93%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@felixdittrich92 felixdittrich92 changed the title [models] Fix a crash on 1-px word crops, speed up pre/post-processing and transformer attention, fix parallel test runs [models] Speed up pre/post-processing, fuse attention and cache MASTER/PARSeq decoding, fix a crash on 1-px word crops Nov 17, 2026
@felixdittrich92 felixdittrich92 added module: datasets Related to doctr.datasets benchmark This issue reports a benchmark framework: pytorch Related to PyTorch backend topic: text recognition Related to the task of text recognition ext: docs Related to docs folder labels Nov 17, 2026
@felixdittrich92
felixdittrich92 merged commit 4c49d79 into mindee:main Nov 17, 2026
54 of 56 checks passed
@felixdittrich92
felixdittrich92 deleted the perf-and-fixes branch November 17, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmark This issue reports a benchmark ext: docs Related to docs folder ext: tests Related to tests folder framework: pytorch Related to PyTorch backend module: datasets Related to doctr.datasets module: io Related to doctr.io module: models Related to doctr.models topic: text recognition Related to the task of text recognition type: bug Something isn't working type: enhancement Improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant