Skip to content

Include postlex retained terminals in parser cache key - #1685

Open
FanWu-ai wants to merge 1 commit into
lark-parser:masterfrom
FanWu-ai:fix-postlex-cache-terminals-20261021
Open

FanWu-ai wants to merge 1 commit into
lark-parser:masterfrom
FanWu-ai:fix-postlex-cache-terminals-20261021

Conversation

@FanWu-ai

Copy link
Copy Markdown

Summary

postlex.always_accept affects which terminals survive grammar compilation, but postlex is excluded from the parser cache key. Reusing a cache built without postlex (or with a different retained-terminal set) can discard required tokens and reject otherwise valid input.

  • Include the retained-terminal set in the cache key.
  • Snapshot its iterable once and reuse it for grammar compilation, preserving one-shot iterators.
  • Leave lexer-only mode's iterable untouched.
  • Cover both basic/contextual lexers, changing postlex requirements, repeated cache loads, one-shot iterators, and a lexer-only control.

Validation

  • Regression on original code fails in both lexer modes with UnexpectedCharacters.
  • Pristine Python 3.10.22: 1337 tests, OK (148 existing skips).
  • Fixed Python 3.10.22: 1340 tests, OK (148 existing skips), including all 8 Nearley tests with unchanged Js2Py==0.68.
  • Python 3.12 and 3.13: 1332 tests each, OK (148 existing skips); pinned Js2Py cannot run Nearley on these interpreters.
  • mypy==1.12: all 34 source files pass.
  • All configured pre-commit checks and git diff --check pass.

AI assistance was used to investigate, implement, review, and test this change.

@FanWu-ai
FanWu-ai marked this pull request as ready for review October 20, 2026 21:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant