Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/workflows/jekyll.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ on:
# reached main and merely failed to deploy, leaving main red.
pull_request:
branches: [ "main" ]
# Daily, so the citation counts printed into the paper posts are at most a
# day old (script/fetch-citations.rb); the browser refreshes older ones.
schedule:
- cron: "41 20 * * *" # 20:41 UTC, 05:41 KST
workflow_dispatch:

# Read-only by default. Only `deploy` gets the Pages and OIDC grants, so a PR
Expand Down Expand Up @@ -56,6 +60,19 @@ jobs:
- name: Setup Pages
id: pages
uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0
- name: Fetch citation counts
# One batch request to Semantic Scholar, written to _data/citations.json
# for the build to print. When it fails, the counts the live site
# published last are kept; with neither, it warns and the site builds
# without counts, which the browser then asks for. Run on PRs too, so
# the gates below check the pages as they will be published. It fails
# only on a refused SEMANTIC_SCHOLAR_API_KEY; its own timeouts and
# retries end it in about six minutes at worst.
run: bundle exec ruby script/fetch-citations.rb
timeout-minutes: 8
env:
SEMANTIC_SCHOLAR_API_KEY: ${{ secrets.SEMANTIC_SCHOLAR_API_KEY }}
BASE_PATH: ${{ steps.pages.outputs.base_path }}
- name: Build with Jekyll
run: bundle exec jekyll build --baseurl "$BASE_PATH" --strict-front-matter
env:
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,6 @@ _posts_bak/
# Editor and toolchain leftovers
node_modules/
.vscode/

# Citation counts, fetched before each build (script/fetch-citations.rb)
_data/citations.json
9 changes: 9 additions & 0 deletions _data/citation_ids.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# The Semantic Scholar record (paperId) of a paper_doi whose usual id
# (arXiv:<id>, DOI:<doi>) finds no record there, or a duplicate with no
# citations. Keyed by paper_doi, so a regenerated post keeps it.
# _plugins/citations.rb fails the build on a key no post gives or a value that
# is not a paperId. The paperId is the hex string at the end of a paper's
# semanticscholar.org URL.

# DeepSeek-V3: arXiv:2412.19437 leads to a duplicate record with 0 citations.
10.48550/arXiv.2412.19437: 5b6fdb2aea424d02c141da81d62b04d739d62b96
2 changes: 1 addition & 1 deletion _includes/footer.html
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ <h2 class="footer-heading">{{ site.title }}</h2>
{% endif %}

<script src="{{ '/js/main.js' | prepend: site.baseurl }}" defer></script>
{% if page.paper_doi %}<script src="{{ '/js/citations.js' | prepend: site.baseurl }}" defer></script>{% endif %}
{% if page.citation_id %}<script src="{{ '/js/citations.js' | prepend: site.baseurl }}" defer></script>{% endif %}

<!-- Do not write Liquid tag delimiters in these HTML comments. Liquid tokenizes
tags inside comments too, so a tag named in prose is opened for real and the
Expand Down
16 changes: 12 additions & 4 deletions _layouts/post.html
Original file line number Diff line number Diff line change
Expand Up @@ -48,12 +48,20 @@ <h1 class="title">{{ page.title | escape }}</h1>
</div>
{% endif %}
{%- comment -%}
Hidden until js/citations.js fills it from Semantic Scholar. Last, so
the fields before it stay put when it appears.
The count the build fetched (`citation`, _plugins/citations.rb);
js/citations.js refreshes it once it is a day old, and fills the
field, hidden till then, when the build has none. The link's text and
label are the ones the script writes. Last, so the fields before it
stay put when it appears.
{%- endcomment -%}
{% if page.paper_doi %}
<div class="post-citations" data-paper-doi="{{ page.paper_doi | escape }}" hidden>
{% if page.citation_id %}
{%- assign cited = page.citation -%}
<div class="post-citations" data-citation-id="{{ page.citation_id | escape }}"
{%- if cited %} data-count="{{ cited.count }}" data-paper-id="{{ cited.paper_id | escape }}" data-fetched-at="{{ site.data.citations.fetched_at | escape }}"{% else %} hidden{% endif %}>
<span class="meta-label">Citations</span>
{%- if cited %}
<a class="meta-value" title="Semantic Scholar" aria-label="{{ cited.shown }} {% if cited.count == 1 %}citation{% else %}citations{% endif %} on Semantic Scholar" href="https://www.semanticscholar.org/paper/{{ cited.paper_id | escape }}">{{ cited.shown }}</a>
{%- endif %}
</div>
{% endif %}
</div>
Expand Down
94 changes: 94 additions & 0 deletions _plugins/citations.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# frozen_string_literal: true
#
# A paper post's citation count: which Semantic Scholar record it is, and the
# count the last build fetched for it.
#
# Every post with a `paper_doi` gets `citation_id`, the id Semantic Scholar is
# asked by. script/fetch-citations.rb asks for all of them at once before the
# build and writes _data/citations.json (not in Git); a post whose record it
# found also gets `citation` ({count, paper_id, shown}). _layouts/post.html
# prints the count, and js/citations.js refreshes it once it is a day old.
# With no file (a local build) or no entry, the browser asks for it.
#
# Semantic Scholar finds an arXiv paper by its arXiv ID, not by the DOI arXiv
# registers for it. A few papers are not there under that ID, or it leads to a
# duplicate record with no citations: _data/citation_ids.yml names the record
# for those.

module Citations
ARXIV_DOI = %r{\A10\.48550/arXiv\.(.+)\z}i
PAPER_ID = /\A\h{40}\z/

# The id for a DOI: the record named in `overrides` (DOI => paperId), else
# arXiv:<id> for an arXiv DOI, else DOI:<doi>.
def self.lookup_id(doi, overrides = {})
return overrides[doi] if overrides.key?(doi)
arxiv = doi.match(ARXIV_DOI)
arxiv ? "arXiv:#{arxiv[1]}" : "DOI:#{doi}"
end

# _data/citation_ids.yml, checked: each key a DOI some post gives, each value
# a paperId. Raises on anything else, so a mistyped entry fails the build
# rather than silently asking for the wrong paper.
def self.overrides(data, dois)
data ||= {}
raise ArgumentError, "_data/citation_ids.yml must map DOIs to paperIds" unless data.is_a?(Hash)
data.each do |doi, id|
raise ArgumentError, "_data/citation_ids.yml: #{doi} is no post's paper_doi" unless dois.include?(doi)
raise ArgumentError, "_data/citation_ids.yml: #{doi} maps to #{id.inspect}, not a 40-hex paperId" unless id.is_a?(String) && id.match?(PAPER_ID)
end
data
end

# 21110 as "21,110", as js/citations.js prints it (toLocaleString("en-US")).
def self.delimited(count)
count.to_s.reverse.scan(/\d{1,3}/).join(",").reverse
end

# One record of the batch answer, as _data/citations.json keeps it. nil for
# a paper not found, or one with no citations: Semantic Scholar answers 0
# for a duplicate record, and a zero is not shown.
def self.entry(record)
return nil unless record.is_a?(Hash)
count = record["citationCount"]
id = record["paperId"].to_s
return nil unless count.is_a?(Integer) && count.positive? && id.match?(PAPER_ID)
{ "count" => count, "paper_id" => id }
end

# Sets `citation_id`, and `citation` when the fetched counts have one, on
# each of `posts` (anything with a `data` Hash) that has a `paper_doi`.
# `data` is site.data. Raises ArgumentError on a bad citation_ids.yml.
def self.annotate(posts, data)
papers = posts.select { |post| post.data["paper_doi"] }
overrides = overrides(data["citation_ids"], papers.map { |post| post.data["paper_doi"].to_s })
fetched = data["citations"].is_a?(Hash) ? data["citations"]["papers"] : nil
fetched = {} unless fetched.is_a?(Hash)
papers.each do |post|
id = lookup_id(post.data["paper_doi"].to_s, overrides)
post.data["citation_id"] = id
stored = fetched[id].is_a?(Hash) ? fetched[id] : {}
e = entry("citationCount" => stored["count"], "paperId" => stored["paper_id"])
post.data["citation"] = e.merge("shown" => delimited(e["count"])) if e
end
end

# `ids` and the batch answer, which lists one record (or null) per id in
# order. Raises when the two do not line up: a count must not land on
# another paper.
def self.entries(ids, records)
unless records.is_a?(Array) && records.size == ids.size
raise ArgumentError, "Semantic Scholar answered #{records.is_a?(Array) ? records.size : records.class} records for #{ids.size} ids"
end
ids.zip(records).filter_map { |id, record| (e = entry(record)) && [id, e] }.to_h
end
end

# Guarded so test/ can require this file without Jekyll loaded.
if defined?(Jekyll::Hooks)
Jekyll::Hooks.register :site, :post_read do |site|
Citations.annotate(site.posts.docs, site.data)
rescue ArgumentError => e
raise Jekyll::Errors::FatalException, e.message
end
end
9 changes: 9 additions & 0 deletions citations.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
layout: null
---
{%- comment -%}
The citation counts this build printed (_data/citations.json), published so
the next build can keep them when Semantic Scholar does not answer
(script/fetch-citations.rb). `null` from a build without them.
{%- endcomment -%}
{{ site.data.citations | jsonify }}
43 changes: 33 additions & 10 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,14 @@ _plugins/ build-time Ruby, one concern per file (see Plugins)
_sass/ _layout, _post, _tags, _search, _syntax, _dark, base/;
bourbon/ and neat/ are vendored, not edited
_data/ post_conventions.yml (the conventions, shared with
scholar-lens), covers.yml (cover credits)
scholar-lens), covers.yml (cover credits),
citation_ids.yml (Semantic Scholar records); citations.json
is fetched before the build, not in Git
css/ js/ main.scss; main.js, search.js, citations.js
assets/images/ covers (+ -960/-1280 JPEG and WebP copies), figures, diagrams
feed.xml search.json the Atom feed and the search index, as Liquid pages
script/ lint-posts.rb, validate-site.sh, check-plugin-output.rb,
published-urls.txt, cover-variants.sh
published-urls.txt, cover-variants.sh, fetch-citations.rb
test/ minitest for _plugins/ and the lint
docs/ these documents and their diagrams
.github/ workflows (deploy, workflow lint, external links), Dependabot
Expand All @@ -76,7 +78,8 @@ excluded from the site.
URL to `redirect_from`; `posted_date.rb` sets `posted_at` and
`last_modified_at` and re-sorts the posts; `translations.rb` marks the English
twin of a pair `hidden` and links each post of a pair to the other
(`page.translation`).
(`page.translation`); `citations.rb` sets each paper post's Semantic
Scholar id and fetched count.
3. **Generators**: pagination, the sitemap and the redirect stubs
(`_layouts/redirect.html`).
4. **`:pre_render` hooks**: `post_description.rb` fills in each post's
Expand All @@ -101,6 +104,7 @@ and safe to delete.
|---|---|---|
| `posted_date.rb` | `:post_read`, and a `Document#<=>` override | `posted_at` = the file-name day when later than `date:`, else `date:`. Posts compare by it, so `site.posts`, `site.tags`, `site.categories`, pagination and everything iterating them are in posting order. Sets `last_modified_at` to a later file-name day |
| `legacy_urls.rb` | `:post_read` | adds the pre-2026-10-11 URL (`/paper%20reviews/…`) of each older post to `redirect_from` |
| `citations.rb` | `:post_read` | sets `citation_id` (the Semantic Scholar id) on each post with a `paper_doi`, and `citation` from `_data/citations.json`; fails the build on a bad `_data/citation_ids.yml` entry (see [Citation counts](#search)) |
| `translations.rb` | `:post_read`, `listed` filter | sets `page.translation` (the other post of a pair, for the language switcher and `hreflang`) and hides the twin from every listing; `listed` drops hidden posts |
| `post_description.rb` | `:pre_render`, filter | derives `description` from the first prose paragraph |
| `related_posts.rb` | `:pre_render` | ranks related posts by shared tags (weights at the top of the file) and sets the previous/next post in posting order, same language, never the twin |
Expand Down Expand Up @@ -210,13 +214,27 @@ site.
- `_config.yml` cannot configure the search: it is browser JavaScript. Change
`search.json` or `js/search.js`.

**Citation counts** on posts with a `paper_doi` come from Semantic Scholar in
the browser (`js/citations.js`), looked up as `arXiv:<id>` for an arXiv DOI and
`DOI:<doi>` otherwise, and cached a day in `localStorage`. A failed request
shows the last count it got, or hides the field. A zero hides it too: Semantic
Scholar returns 0 for a paper's duplicate records. OpenAlex was not used because
it counts the preprint and the published version separately, well under the
real total.
**Citation counts** on posts with a `paper_doi` come from Semantic Scholar.
Before each build, `script/fetch-citations.rb` asks for every paper in one
batch request and writes `_data/citations.json` (not in Git); the build prints
each count on its post (`_plugins/citations.rb`), and CI runs once a day so the
counts stay about a day old. The count is in the HTML, so it shows without
JavaScript. `js/citations.js` shows that count and asks
Semantic Scholar itself only when the newest count it has, the build's or its
`localStorage` one, is a day old (the build's gets six more hours, for a
late daily run). The keyless API answers 429 often, to a batch too, so the
script retries with backoff. A failed fetch
keeps the counts the live site published last (`/citations.json`) when it can
read them; with none, it warns and builds without them.
A failed browser request keeps the count shown, or hides the field. A zero hides it too: Semantic Scholar returns 0 for a paper's
duplicate records.

A paper is looked up as `arXiv:<id>` for an arXiv DOI and `DOI:<doi>` otherwise,
unless `_data/citation_ids.yml` names its record (paperId): for a paper missing
under its arXiv ID, or one that leads to a duplicate. It is keyed by
`paper_doi`, so a regenerated post keeps it. `SEMANTIC_SCHOLAR_API_KEY`, a
repository secret, is sent when set. OpenAlex was not used because it counts
the preprint and the published version separately, well under the real total.

## Styles

Expand Down Expand Up @@ -316,6 +334,11 @@ build.
matching has no morphology. A real fix is a different search engine.
- **The index grows with every post.** It loads only when search is used; split
it or move search server-side if it outgrows that.
- **Citation counts age when the daily run stops.** GitHub turns off a
scheduled workflow after 60 days with no repository activity; the counts in
the pages then age, and readers' browsers ask Semantic Scholar once a day.
Re-enable it under Actions. A paper Semantic Scholar's API does not have
(Gemini 1.0, `arXiv:2312.11805`, today) shows no count.
- **Excerpts can show TeX** (`\theta`) when the match falls inside math.
- **Long posts stay slow in mobile lab tests.** Lighthouse's throttled phone
puts LCP at 5–6 s on the longest posts (DeepSeek-V3, the Strands guide) even
Expand Down
25 changes: 24 additions & 1 deletion docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ shellcheck script/*.sh assets/images/*.sh && actionlint .github/workflows/*.yml

| Workflow | Runs on | Does |
|---|---|---|
| `jekyll.yml` | PRs to `main`, pushes to `main`, manual | tests → lint → production build → htmlproofer → validate-site; on `main` only, deploys to GitHub Pages |
| `jekyll.yml` | PRs to `main`, pushes to `main`, daily 20:41 UTC, manual | tests → lint → citation counts → production build → htmlproofer → validate-site; on `main` only, deploys to GitHub Pages (the daily run refreshes the citation counts) |
| `workflow-lint.yml` | PRs and pushes to `main`, manual | actionlint and shellcheck |
| `external-links.yml` | Mondays, manual | htmlproofer with external links on; fails (and notifies) on a dead link, blocks nothing |
| Dependabot | weekly | PRs for GitHub Actions and gems |
Expand All @@ -58,6 +58,29 @@ shellcheck script/*.sh assets/images/*.sh && actionlint .github/workflows/*.yml
hosts that fail any scripted request while working in a browser; its comment
says which.

## Citation counts

`script/fetch-citations.rb` runs in every `jekyll.yml` run, before the build
(how it works: docs/architecture.md, "Citation counts").

- **A run's warnings** ("Citation counts" annotations) say when Semantic
Scholar did not answer and which counts were kept, or which papers have no
count. They block nothing.
- **An API key**, for a higher rate limit: request one from Semantic Scholar
and save it as the repository secret `SEMANTIC_SCHOLAR_API_KEY`. A key the
API refuses fails every run, PRs from this repository included, until the
secret is renewed or deleted.
- **A paper with no count, or the wrong one**: find its record on
semanticscholar.org and add `paper_doi: <paperId>` to
`_data/citation_ids.yml`. Check the paperId answers first
(`https://api.semanticscholar.org/graph/v1/paper/<paperId>?fields=citationCount`):
some records on the website are not in the API.
- **Counts in a local build**: run `bundle exec ruby script/fetch-citations.rb`
before `jekyll build`; `_data/citations.json` is not in Git, and a run that
gets no answer leaves an older one in place.
- **The daily run stopped**: GitHub turns schedules off after 60 days without
repository activity; turn it back on under Actions.

## When a deploy fails

GitHub Pages keeps serving the last good deploy, so a failure leaves the live
Expand Down
2 changes: 1 addition & 1 deletion docs/writing.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ title: "<Post Title>"
subtitle: "<one-line pitch>" # paper posts: required, not the title; else optional
date: YYYY-MM-DD HH:MM:SS # paper posts: the paper's date; else the posting date
paper_author: "<Org>" # the paper's org, not people; paper posts only
paper_doi: "10.48550/arXiv.<id>" # optional, paper posts only: the citation count in the meta panel
paper_doi: "10.48550/arXiv.<id>" # optional, paper posts only: the citation count in the meta panel (docs/architecture.md)
seo_title: "<shorter title>" # optional: for <title> when the title is too long
description: >- # optional, plain text; see below
<search snippet, ~150 chars>
Expand Down
Loading
Loading