Repository navigation
feat(regulations): compile drafts in a sandboxed LaTeX worker - #112
Draft
lorenzocorallo wants to merge 2 commits into
Draft
lorenzocorallo wants to merge 2 commits into
lorenzocorallo wants to merge 2 commits into
Conversation
A separate service and image (FROM ghcr.io/polinetworkorg/legal-texlive) that runs xelatex without shell escape, with reads and writes confined to the workdir (openin_any/openout_any paranoid), a timeout and a memory limit. It keeps a warm workdir per draft so .aux survives, runs one compile per draft (latest wins) and a few in parallel, and needs no database or network: the sources come in the request and the asset bundle (fonts, the shared class) is uploaded once per hash. Workdir names are keyed with a per-process secret and the worker is non-dumpable, so a draft's LaTeX can read neither another draft nor the worker's environment. The compiler caches results by source hash, records each draft's compile state for compile.status and serves only the PDFs compiled for that draft.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #111. The PDF preview compiler (spec decision 4): a separate, sandboxed worker service plus the
DB-backed compiler that the collab service drives. The backend never runs xelatex in-process.
Worker (
src/regulations/worker,docker/latex-worker.Dockerfile)FROM ghcr.io/polinetworkorg/legal-texlive(build argTEXLIVE_IMAGE). It adds only the bun binary andone bundled
worker.js. CI buildsghcr.io/polinetworkorg/latex-workeronmainfor amd64 and arm64. Sincelegalis private, its package must grant this repository read access.POST /compile,PUT /assets/<hash>,GET /health, with a bearer secret (WORKER_SECRET). There's nodatabase and no egress. Sources come in the request. The asset bundle (fonts,
shared/*.cls, anything elsethat isn't a source) is uploaded once per content hash and cached on tmpfs. The client re-uploads it when the
worker answers
assets_missing, e.g. after a restart.superseded.WORKER_CONCURRENCYoverall..auxsurvives. Only changed sources are rewritten. LRU eviction.-no-shell-escape -halt-on-error -file-line-error,openin_any=p,openout_any=p,shell_escape=f, a clean environment,prlimiton address space, and a total timeout (SIGKILL). It rerunswhile the log asks, up to 3 passes. Errors are parsed with file and line.
\inputanother's. The process is made non-dumpable (
prctl), so xelatex can't read its/proc/<pid>/environ.Compiler (
src/regulations/compile)createCompiler: one in flight per draft, latest wins, cached bysourceHash(entry + sources + assetshash). Results are shared across drafts and viewers. It stores
regulations_compiles(with the PDF inregulations_blobs), the per-draftregulations_draft_compiles(state, hash,lastGoodHash) andcompile_refs, and emits events for the collab service to broadcast. An unreachable worker shows up as anerror but isn't cached.
readCompileStatus/readCompilePdfbackcompile.status/compile.pdf. A PDF is served only for hashescompiled for that draft.
readDocumentsplits a legal checkout's document directory into sources (legal.jsonsourcesglobs) and theasset bundle.
Verified
tests/regulations-compile.test.ts: queue, hashing and path checks, log parsing, compiler cache andlatest-wins with a fake worker. If
xelatexis installed, it also runs a bun fixture against a real worker:it compiles, uploads missing assets, reports errors with files, refuses writes outside the workdir, runs no
shell escape, and refuses requests without the secret.
legal/docker/texlive) and this worker on top of it,then ran it with
--read-only, tmpfs/workand/tmp,--cap-drop ALL,no-new-privileges, uid 1000,1.5 GiB / 2 CPUs, on an
--internalnetwork:0.75 s (1 pass). The edit is in the PDF.
\input{/proc/1/environ},\input{/etc/passwd}and the\@@inputprimitive on absolute paths are allrefused.
/proc/1/environis permission-denied for the worker's own user. No egress (EAI_AGAIN).openin_any=pdid not stop the\inputprimitive from readingabsolute paths. That's why the HMAC'd workdirs and the non-dumpable process are there.