Skip to content

Store hotlinked arXiv figures; seo_title for long titles; search tests - #139

Merged
bits-bytes-nn merged 2 commits into
mainfrom
fix/localize-figures-seo-titles
Nov 17, 2026
Merged

bits-bytes-nn merged 2 commits into
mainfrom
fix/localize-figures-seo-titles

Conversation

@bits-bytes-nn

Copy link
Copy Markdown
Owner

This PR carries out three of the decisions left open after #138: storing hotlinked figures, requiring seo_title for long titles, and testing the search script. The other three are not done; the reasons are at the end.

1. Hotlinked arXiv figures are now stored

Eighteen older posts (19 files, counting the ko/en pair) still hotlinked 122 figures from arXiv. Those images had no width/height, so the page jumped as they loaded, and the links break whenever arXiv renumbers a revision.

I stored them with scholar-lens's own ImageLocalizer, run from its main (0bccf60), so they get the generator's names (assets/images/<slug>-<n>.<ext>) and its handling of ar5iv placeholders and SVGs. Only image URLs changed in the posts: 131 lines changed and 131 replaced, each only in the src. Images from other sites (AWS, Medium, Databricks and others, 25 in all) stay links, as scholar-lens leaves them, because they belong to their owners. The repo grows by about 42 MB (117 files).

Hand fixes to generated posts (regenerating a paper would produce the same files):

  • llama-2, deepseek-llm, deepseekmoe, deepseek-coder, raptor, mistral-7b, deepseekmath, mixtral-of-experts, kimi-k2, deepseek-v2, deepseek-r1, deepseek-v3, gemma-2, qwen3, pixtral-12b, direct-preference-optimization, deepseek-v3.2: the arXiv figure URLs now point to /assets/images/<slug>-<n>.png.
  • ai-ready-data-semantic-layer-knowledge-graph (ko/en, hand-written): 5 figures, shared by both files.

2. seo_title for long titles

_data/post_conventions.yml change: new key seo_title_max_length: 70. scholar-lens already reads this field and uses 70 as its default, so its behaviour does not change. A scholar-lens PR syncs its copy of the file.

New lint rules (script/lint-posts.rb):

  • A title longer than seo_title_max_length needs a seo_title, and the seo_title is held to the same limit. This matches scholar-lens's validation.py check.
    • Fails: a 71-character title with no seo_title; a 71-character seo_title.
    • Passes: a 71-character title with seo_title: "Short".
  • An arXiv image URL is rejected.
    • Fails: ![Fig](https://arxiv.org/html/2310.06825v1/x1.png), <img src="https://www.arxiv.org/x.png">.
    • Passes: /assets/images/mistral-7b-1.png; a CloudFront image (it belongs to its owner); a link [the paper](https://arxiv.org/abs/…); an image inside a code fence.
  • The message of the existing ar5iv rule no longer suggests hotlinking arxiv.org/html.

Hand fixes: seo_title was added to 7 posts whose titles run 73–93 characters. Each new seo_title is 34–68 characters:

  • From Local to Global (GraphRAG)
  • DeepSeekMath
  • DeepSeek-V2
  • DeepSeek-R1
  • DPO
  • ai-ready-data ko/en

3. Search tests

js/search.js now exports its pure functions (ranking, pair dedupe, excerpts, markup) when loaded in node. test/js/search.test.js has 9 tests and runs with node --test test/js/, with no packages to install. CI runs it after the Ruby tests. Two mutation checks confirmed the tests catch a broken pair rule and a broken title weight.

Not done

  • Korean title suffix for surveys and guides: scholar-lens hardcodes the suffix in validation.py, and that file has someone else's uncommitted work in progress. The value is low and the change would collide with that work.
  • dateModified: it needs either a new front-matter key in the generator or the full git history in CI, and the impact is low.
  • Unused default*.jpg covers: kept. scholar-lens's DEFAULT_COVER uses default.jpg.

Verification

  • Gates: Ruby unit tests (221), node tests (9), lint (43 posts), actionlint, strict build, htmlproofer and validate-site.sh all pass.
  • Built pages: the stored figures carry width/height.
  • Search: in headless Chrome, "llama" still puts the Llama papers first.

…, test search in node

- 122 arXiv figures that 18 older posts still hotlinked are stored as
  assets/images/<slug>-<n>.<ext>, with scholar-lens's own ImageLocalizer
  (its names, ar5iv-placeholder and SVG handling); only image URLs change.
  They now get width/height, and no longer rot when arXiv renumbers.
  Other sites' images stay links, as scholar-lens leaves them.
- Lint: an arXiv image URL is rejected (store it); a title over the new
  spec key seo_title_max_length (70, scholar-lens's existing limit) needs
  a seo_title, held to the same limit. Seven posts get one.
- js/search.js exports its pure functions under node; test/js/ covers
  ranking, pair dedupe, escaping and excerpts (node --test, no packages),
  and CI runs it.
@bits-bytes-nn
bits-bytes-nn merged commit b905440 into main Nov 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant