[JiaxinD]

Background

ModernBERT serial and TP builders select local RoPE settings but pass only the padding mask to every layer. Tokens beyond the local window therefore remain visible. Short sequences hide this error. Found during compatible Granite embedding source audit; no originating issue.

Exit Criteria

Local layers enforce inclusive bidirectional half-window boundaries together with padding; global layers retain padding-only attention. Both config layouts/builders are covered, with long official-checkpoint workloads registered for serial and TP validation.

Implementation

• Normalize legacy local_attention // 2 and materialized sliding_window, rejecting invalid values and half-windows outside the graph INT32 range.
• Build one dtype-matched window mask per network, using dynamic runtime positions in the serial builder, and select it only for local/sliding layers.
• Add 162-token cases for ModernBERT base, TP4, and Granite r2, verified with pinned official tokenizers; retain FP32 and the existing 0.995 threshold. Guard against short or out-of-profile reference inputs.
• Scope: fourteen files within ModernBERT. No shared API, ABI, bundle schema, dependency, or checkpoint mapping changes. Existing bundles require rebuilding to receive corrected long-input behavior.

Change categories

• Model or runtime behavior
• Public API
• ABI
• Bundle or artifact format
• Dependencies
• Documentation only
• CI or developer tooling

Validation

Commands and Results

Command	Result
python -m pytest families/modernbert/tests/test_attention_contract.py families/modernbert/tests/test_local_attention.py families/modernbert/tests/test_checkpoint_loading.py families/modernbert/tests/test_parallel_contract.py families/modernbert/tests/test_optimized_checkpoint_guards.py families/modernbert/tests/test_e2e_window_contract.py tools/tests/test_architecture.py tools/tests/test_pr_metadata.py tools/tests/test_family_impact.py -q	228 passed
python -m ruff check families/modernbert/config.py families/modernbert/graph/model.py families/modernbert/model.py families/modernbert/parallel.py families/modernbert/tests/test_attention_contract.py families/modernbert/tests/test_local_attention.py	Passed
python -m tools.model_ci validate; python tools/test_impact.py --validate; python tools/legal_headers.py --check; git diff --check	Passed, zero legal findings

Initial mask regression: 25 failed, 9 passed. Window-bound regression: 2 failed, 18 passed. New E2E workload guards: 9 failed before, 9 passed after. CPU tests cover actual builders, dynamic lengths, TP2, aliases, padding, all-global networks, and attention-value exclusion of distant/padded tokens across FP32/FP16/BF16.

Hardware, Environment, and Revisions

Windows CPU, Python 3.13, NumPy and ml_dtypes; head 359d13d671e7b90dc8e3e4665b699f8f35556f7e, base c3481588. Reference semantics: Transformers 77cfe272, inclusive abs(q-k) <= sliding_window. Synthetic weights, no downloaded checkpoint.

Not Run / Remaining Gaps

Native TensorRT build, GPU inference, checkpoint parity, TP communication, memory, and performance are unrun because authorized H200s are fully assigned. Full-builder tests stub RoPE/attention/collectives; separate numerical tests execute production attention helpers over a NumPy IAttention simulation. Six GPU E2E cases remain skipped; the new long prompts have not been proven to reject the old implementation at 0.995. No GPU or performance claim. Serial BF16 integration remains separate in #1846.

Contributor Self-Review

• I have completed a self-review of this change.

Independent follow-up: 68 focused passes, 134 family passes and 6 GPU skips; no blocker. Earlier overflow finding resolved.

Notes For Future Readers

Review config/mask/routing, then E2E workload guards. Upstream checkpoint guards retained by a history-preserving merge. No merge dependency on #1846; revalidate serial BF16 after combining. Dense mask storage is quadratic: one FP32 mask at 8192 tokens is 256 MiB before intermediates. Target-runtime resource qualification remains necessary.

Risk level

• Low
• Medium
• High

Long-input attention intentionally changes to the published local contract, with an additional dense mask and unmeasured runtime resource cost.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[JiaxinD]

@yifeif-nv Could you please run protected premerge for current head 9ea5577? Community CPU / Required, Stable, and Dev passed.