Skip to content

fix(filesystem): decode BOM-marked UTF-16 and UTF-8 files in read mode - #516

Merged
Jeomon merged 1 commit into
CursorTouch:mainfrom
jaeminbird:fix-read-bom
Oct 29, 2026
Merged

Jeomon merged 1 commit into
CursorTouch:mainfrom
jaeminbird:fix-read-bom

Conversation

@jaeminbird

Copy link
Copy Markdown
Contributor

read always decodes as UTF-8, so files written by Windows PowerShell 5.1's > or reg export (UTF-16 LE with BOM) come back as replacement characters and NULs, and Notepad's UTF-8-with-BOM files start with . When the requested encoding is the default UTF-8, read now honours a UTF-8 or UTF-16 BOM, as Get-Content and Notepad do; an explicit encoding is left alone.

  • New tests (UTF-8-sig, UTF-16 LE/BE, with and without offset/limit) fail on main and pass on Windows (Python 3.14).
  • uv run --extra dev pytest passes and ruff check is clean.

@Jeomon
Jeomon merged commit 84817cd into CursorTouch:main Oct 29, 2026
1 check failed
@jaeminbird
jaeminbird deleted the fix-read-bom branch October 30, 2026 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants