Repository navigation
fix(scrape): return non-HTML bodies as fetched instead of running markdownify - #522
Merged
Merged
Conversation
…kdownify markdownify parses any body as HTML, so JSON, source files and plain text lost their line breaks and indentation and had "_" and "*" escaped. Convert only text/html and application/xhtml+xml; keep the old path when the response has no Content-Type.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and outcome
Scrape(defaultuse_dom=False) fetches the URL and passes every body tomarkdownify, whatever itsContent-Type. markdownify parses the body as HTML, so a non-HTML body becomes one big text node:_and*are backslash-escaped:"user_name"comes back as"user\_name",*.pyas\*.py.So a JSON API response, a raw file on GitHub, or a plain-text page reaches the model altered. With
use_sampling=False, which exists to get the raw content, the caller still gets this altered text.This PR keeps
markdownifyfor HTML (text/html,application/xhtml+xml) and returns any other body as fetched (response.text, after the existing charset handling). A response without aContent-Typekeeps the old HTML path. Redirects, the SSRF check, error mapping, charset sniffing and the DOM mode are unchanged.Evidence and validation
Real path.
Desktop.scrape(what theScrapetool calls withuse_dom=False) on a raw GitHub file, served astext/plain; charset=utf-8(https://raw.githubusercontent.com/CursorTouch/Windows-MCP/acb2e35/pyproject.toml):\_in resultmainacb2e35Regression tests. New
tests/test_scrape_content_type.pyreuses the loopback server helper fromtests/test_scrape_encoding.pyand goes throughDesktop.scrape:test_scrape_returns_non_html_body_verbatim: indented Python source astext/plain; charset=utf-8andtext/x-python, and a JSON object asapplication/jsonwith and without a charset, must come back byte-for-byte.test_scrape_still_converts_html_to_markdown:text/html,TEXT/HTML; charset=utf-8andapplication/xhtml+xmlstill produce exactly whatmarkdownifyproduced before.mainacb2e35 source): 4 failed, 3 passed. For example, the Python source comes back as'def add(a, b):\nreturn a \\* b # snake\\_case\\_name\nprint(add(2, 3))'. The three HTML cases pass, as they should.tests/test_scrape_encoding.py,tests/test_scrape_ssrf.pyandtests/test_scrape_tool.pystill pass (26 passed together with the new file).Full suite:
pytest tests -q --deselect tests/test_control_recovery_stdio.pygives 1369 passed, 8 skipped. That deselected test fails the same way on unmodifiedmainin my environment (its stdio child cannot importpytest), as noted in #521.Environment: Windows 11 (build 26200), CPython 3.14.8, markdownify 1.2.3 (the version in
uv.lock), requests from an existing windows-mcp environment, pytest 9.2.0.ruff checkandruff formatpass on the changed files.Not verified: other non-HTML types such as XML feeds or binary downloads were not tested. They are now also returned as text without conversion, while before they went through
markdownify.