[naaz234]

Description

ReferenceProcessor.run records one reference definition, puts the rest of the block back at the front of blocks and returns. The next pass runs every block processor's test over the remainder again, and BlockQuoteProcessor, HashHeaderProcessor and HRProcessor search the whole block, so a block of n consecutive definitions costs O(n²). On CPython 3.12, 8000 definitions (247 KiB) take 18 s to convert with no extensions. FootnoteBlockProcessor.run in the footnotes extension has the same one-definition-per-pass shape, and 8000 footnote definitions (177 KiB) take 14 s.

Both processors now record a run of definitions in one pass. Only a definition that is directly followed by another definition is taken early. The last one, and any definition that is indented or spans lines, is left for the next pass as before, because the block starting at it can belong to another processor: an indented code block when tab_length is below 4, or a setext header such as [b]: /url followed by === (covered by a new test). The same inputs now take 0.01 s and 0.23 s.

Output is identical to master over about 1.2 million randomly generated documents rendered with no extensions, extra, footnotes, tables, def_list, admonition, and tab_length of 2 and 3. Definitions separated by other text are still handled one per pass, since what follows a definition can change how the earlier processors treat the block.

Timing script used:

import time, markdown
for n in (2000, 4000, 8000):
    s = ''.join(f'[{i}]: http://example.com/{i}\n' for i in range(n))
    t = time.perf_counter(); markdown.markdown(s); print('references', n, round(time.perf_counter() - t, 2))
md = markdown.Markdown(extensions=['footnotes'])
for n in (2000, 4000, 8000):
    s = ''.join(f'[^{i}]: Footnote {i}\n' for i in range(n))
    md.reset(); t = time.perf_counter(); md.convert(s); print('footnotes', n, round(time.perf_counter() - t, 2))

AI Assistance Disclosure

• No AI tools were used in preparing this PR.

• AI tools were used and I have disclosed which ones, and have fully reviewed and verified their output.

  AI tools were use to assist with the following:

  • Code
  • Code Comments
  • Documentation
  • Changelog Entry
  • Commit Message
  • The Description Above
  • Other: tests and the randomized output comparison against master

Checklist

• This PR follows the contribution guidelines.
• The code follows the Code Style Guide.
• The commit message follows the Commit Message Style Guide.
• I have added or updated relevant docs, including release notes if applicable which follow the Documentation Style Guide.
• I have added or updated relevant tests.
• I have not requested, and will not request, an automated AI review for this PR.

[waylan]

This is not the correct behavior. I see that that is how Python-Markdown currently behaves. You seem to have found a bug. Now I understand why you were ignoring the last reference if it was followed by a non-reference without a blank line. I think you can remove that restriction.

The reference implementation outputs <p>===</p> as it should. We should as well. Of course, references should include both a and b.

What I don't understand is why you are ignoring a reference if it is followed by a line which starts with indentation. The reference is not indented and should not be part of the indented block. What tests fail if that restriction is removed?

Same goes for footnote references.

[naaz234]

Makes sense, I'd been treating master's output as the behaviour to preserve. I've dropped that restriction in both processors, so this input now gives <p>===</p> with both a and b recorded, and the test is updated to match. A lone [b]: /b followed by === is still a header, since SetextHeaderProcessor runs before ReferenceProcessor. I've left that alone as it looks like it'd need a priority change.

The indentation check wasn't about the following line. It checked whether the next definition itself started with tab_length spaces, which can only happen with tab_length of 2 or 3 since the regex allows up to 3 leading spaces. On master that definition starts its own block on the next pass and CodeBlockProcessor claims it, so [a]: /a followed by [b]: /b with tab_length=2 renders [b]: /b as code. I was only keeping that output. No tests fail without it, so it's gone from both, and that case now records b.

I did keep one check in ReferenceProcessor: a definition whose link or title sits on a following line still ends the run. Without it [a]: /a, [b]:, *** on three lines would record b as a link to ***, where master gives a [b]: paragraph and an <hr />. For footnotes, a === after a run now just lazily continues the last footnote, as any unindented line after a footnote does.

[waylan]

A lone [b]: /b followed by === is still a header, since SetextHeaderProcessor runs before ReferenceProcessor. I've left that alone as it looks like it'd need a priority change.

That is a bug. Either we can address is here or separately.

[waylan]

Now that we have removed this operation between the first and second for loops, we can probably combine the two loops into one.

[waylan]

Do we have a test demonstrating this scenario described in the comments? Overall, I feel like this entire patch could use more tests to better cover every possible scenario.

[waylan]

This simplified the code significantly. Thank you.