Skip to content

fix(scrape): return non-HTML bodies as fetched instead of running markdownify - #522

Merged
Jeomon merged 1 commit into
CursorTouch:mainfrom
Plastic-time:rrfz/scrape-non-html
Oct 31, 2026
Merged

Jeomon merged 1 commit into
CursorTouch:mainfrom
Plastic-time:rrfz/scrape-non-html

Conversation

@Plastic-time

Copy link
Copy Markdown
Contributor

Problem and outcome

Scrape (default use_dom=False) fetches the URL and passes every body to markdownify, whatever its Content-Type. markdownify parses the body as HTML, so a non-HTML body becomes one big text node:

  • line breaks and indentation collapse: Python source loses its indentation, blank lines disappear;
  • _ and * are backslash-escaped: "user_name" comes back as "user\_name", *.py as \*.py.

So a JSON API response, a raw file on GitHub, or a plain-text page reaches the model altered. With use_sampling=False, which exists to get the raw content, the caller still gets this altered text.

This PR keeps markdownify for HTML (text/html, application/xhtml+xml) and returns any other body as fetched (response.text, after the existing charset handling). A response without a Content-Type keeps the old HTML path. Redirects, the SSRF check, error mapping, charset sniffing and the DOM mode are unchanged.

Evidence and validation

Real path. Desktop.scrape (what the Scrape tool calls with use_dom=False) on a raw GitHub file, served as text/plain; charset=utf-8 (https://raw.githubusercontent.com/CursorTouch/Windows-MCP/acb2e35/pyproject.toml):

identical to the served body lines (served: 82) \_ in result
main acb2e35 no 70 9
this PR yes 82 0

Regression tests. New tests/test_scrape_content_type.py reuses the loopback server helper from tests/test_scrape_encoding.py and goes through Desktop.scrape:

  • test_scrape_returns_non_html_body_verbatim: indented Python source as text/plain; charset=utf-8 and text/x-python, and a JSON object as application/json with and without a charset, must come back byte-for-byte.
  • test_scrape_still_converts_html_to_markdown: text/html, TEXT/HTML; charset=utf-8 and application/xhtml+xml still produce exactly what markdownify produced before.
pytest tests/test_scrape_content_type.py -q
  • Before (main acb2e35 source): 4 failed, 3 passed. For example, the Python source comes back as 'def add(a, b):\nreturn a \\* b # snake\\_case\\_name\nprint(add(2, 3))'. The three HTML cases pass, as they should.
  • After: 7 passed

tests/test_scrape_encoding.py, tests/test_scrape_ssrf.py and tests/test_scrape_tool.py still pass (26 passed together with the new file).

Full suite: pytest tests -q --deselect tests/test_control_recovery_stdio.py gives 1369 passed, 8 skipped. That deselected test fails the same way on unmodified main in my environment (its stdio child cannot import pytest), as noted in #521.

Environment: Windows 11 (build 26200), CPython 3.14.8, markdownify 1.2.3 (the version in uv.lock), requests from an existing windows-mcp environment, pytest 9.2.0. ruff check and ruff format pass on the changed files.

Not verified: other non-HTML types such as XML feeds or binary downloads were not tested. They are now also returned as text without conversion, while before they went through markdownify.

…kdownify

markdownify parses any body as HTML, so JSON, source files and plain text
lost their line breaks and indentation and had "_" and "*" escaped. Convert
only text/html and application/xhtml+xml; keep the old path when the
response has no Content-Type.
@Jeomon
Jeomon merged commit 28dd6a7 into CursorTouch:main Oct 31, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants