Repository navigation
Conversation
A dedicated uploadQueue trigger mails the owner on the first real failure — first retry-count increase or terminal failed, so compaction holds and self-healing blips can't false-alarm — then counts silently while the queue stays dirty, and re-arms when it drains (including drain-by-delete: cleanupOldEntries removing the last entry is a drain). One transaction reads experiment and owner, checks the armed flag and the 24h backstop, and creates the mail doc atomically: the 20-way race test proves exactly one mail, nineteen counted. The email leads with 'The file is not lost', states the real retry window (five attempts over ~31 hours), names the provider error, and routes legacy-OSF cases to the FAQ. 29 pure + 15 emulator tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two bearer-token endpoints: one mails a 6-digit code (hash-only storage in the default-deny verifications doc, 60s resend cooldown), one verifies inside a transaction that re-reads the live address so a code sent to an old address can never verify a new one, caps attempts before comparing hashes, and is the sole writer of contactEmailVerified=true. Verification never gates anything — it moves the account section's StatusIndicator from neutral to ok. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The backfill iterates Firebase Auth users — never OSF-supplied doc emails, which are all synthetic — dry-run by default, idempotent, batched. Account purge now also removes the verification doc and the user's queued mail (datapipe.owner is the handle). The runbook covers rules-before-functions ordering, the Trigger Email extension with the SPF/DKIM warning, and the rollout order that seeds most researchers silently before the gate ever renders. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Required contact email for all users + first-failure upload notifications
…outes lib/docs-nav.js is the single source of truth (sidebar order is reading order and drives prev/next); DocsLayout carries the 1100px shell with a 240px sticky sidebar and a collapsible-not-drawer mobile treatment; the sidebar marks the current page three ways. Every page ships its final contract-stable section ids now, so content packages never renumber and the redirect map targets real anchors. ApiPrimitives lifts the four api-docs table components onto semantic tokens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Choosing/connecting/reconnecting providers with the per-provider behavior that was documented nowhere: Drive's folder semantics and quota, Dataverse's draft/contention/rename/ingest quirks and its 60-day tokens (warned only on the new-experiment form — and why), Zenodo's 100-file/50GB ceilings and flat keyspace, token expiry as the researcher actually experiences it, and the two-step reconnect reality. Account page covers credentials handling and deletion, consequence first. OSF wind-down dates via osfSunsetLabel, never restated. Four inventory inaccuracies reported upstream rather than copied forward. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Creating an experiment, sending data, conditions, validation and Psych-DS metadata, with the fine print that lived nowhere: the counter's transaction semantics and the n-lowered-mid-study wrap behavior, per-format validation rules and the on-by-default trial_type trap, session-limit details, base64's separate gating, and the unpkg dependency behind 'unknown' variable descriptions given a boxed lead. Two undocumented behaviors discovered during source verification are now documented; a metadata-error inventory half-truth is stated exactly. CodeHints renders once, by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The queue's real mechanics documented for the first time: what triggers queueing, the five-attempt ~31-hour retry ladder (the 'one week' conflation is gone — the week is payload retention, stated as such), accelerated tiers, the recovery sweep, and drain semantics. File lifecycle covers per-provider collision behavior, flattening, compaction and the 80-of-100 watermark. Finalization is framed as the Zenodo feature it is. The encryption-at-rest sentence now describes the shipped payload-crypto implementation exactly. The API reference moves verbatim onto ApiPrimitives and gains Limits plus the two previously undocumented endpoints. Four inventory errors reported, not propagated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A single editorial pass over all 16 pages: the queue vocabulary is unified (a queued upload / the upload queue / the queued files panel / archive merge / finalizing — compaction and finalization retired from researcher-facing prose), answers are front-loaded for the arrived-by-search reader, every number carries its clock, FAQ Q&A-register survivals are adapted, and a self-contradiction on the data page is resolved. Four irreversible consequences (rejected data, mid-collection edits, finalizing, account deletion) move from prose into visible callouts. Net -107 words. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The navbar reads Getting Started | Documentation | My Experiments at both breakpoints, with Documentation prefix-matched so every /docs subpage keeps it current. /api-docs 308s to /docs/api; /faq becomes a client shim whose 18-entry anchor map routes every historical #item-N deep link to its exact new section (one plan-table error caught and corrected against the live pages). Getting Started gains four GuidanceLines into the docs and the owner-ratified step-8 correction: finalizing is a Zenodo feature, Drive and Dataverse have no ceiling to relieve. Every internal /faq and /api-docs call site is rewired; api-docs.js is deleted, its primitives having moved in Package A. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documentation restructure: /docs with sidebar, FAQ folded in, full service coverage
…rkflow In CI the functions emulator hosts the real onuploadfailure trigger, which races the suite's direct calls on every uploadQueue write. The three write-driven tests now assert the concurrency-invariant state (the machine's actual contract) with benign outcome sets, two latent fixture-level races are defused at the source, and every strict assertion documents why the trigger can't reach it. A cross-suite audit confirms no other queue-writing suite is exposed. The run-migration workflow gains the contact-email backfill option, and the script honors the workflow's DRY_RUN env alongside --apply. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tension The Firebase Extensions platform is being wound down, so mail delivery moves in-house: onmailcreated claims each mail document transactionally (lease-owned, stale-claim takeover, attempts capped at 3) and sends via the SES v2 API directly. Double-send is engineered against as the primary failure mode: SES has no idempotency token, so ambiguous outcomes (timeout after the request left) are terminal-and-loud rather than retried, and the SDK's own invisible retries are disabled. The emulator-hosted instance gates itself off entirely, which solves the CI live-trigger race in production code once. Terminal documents carry delivery.expireAt = endTime + 7 days — the product's universal retention clock — and the Firestore TTL policy keys on it. Deploy workflows gain the SES secret plumbing (test site deliberately unset: records MailConfigMissingError instead of mailing real people). The runbook's §2 now covers SES domain verification, sandbox exit, and the ses:SendEmail-scoped IAM user, all in us-east-2. tsconfig module ES6→es2020 for the SDK's lazy dynamic import (kept off apidata's cold start). 46 pure + 26 emulator tests green, purge and notify regressions included. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dashboard used to store an empty "required fields" textbox as [""] (an array containing one empty string) in the experiment's Firestore doc. validateJSON/validateCSV then ran requiredFields.every(field => keys.includes(field)), which demands a field literally named "" -- something no submission ever has -- so validation always failed with INVALID_DATA against these legacy docs. The dashboard now filters empty entries on input (components/dashboard/ExperimentValidation.js), but existing experiment docs still carry the old [""] shape and the server did no defensive normalization. Both validators now drop empty/whitespace-only entries from the incoming required-fields list before checking it, and treat a resulting empty list the same as "no required fields" (format-validity check only). Behavior is otherwise unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was no pages/404.js, so unmatched routes fell through to Next's default unstyled error page instead of the site's own chrome. Custom404 is a plain component with no getLayout override, so it picks up the standard Navbar + Footer from _app.js like every other page. Copy follows DESIGN.md's page-title role and the 560px single-subject column (contact.js's layout), and links use brandGreen.fg with a persistent underline per DESIGN.md's link rules. No card, illustration, or eyebrow label, per the design system's EmptyState guidance on text- only zero-state content -- this just isn't a bordered EmptyState panel, since it stands in for the whole page rather than a section within one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
plugin-pipe 0.6.0 gzips payloads automatically (jspsych-contrib#236), and a smoke test against the deployed test environment confirms the server inflates gzip bodies with no DataPipe-specific code: a 37.3 MB JSON body sent as 1.1 MB on the wire was decompressed, parsed, and answered with a normal API error. The same body sent uncompressed dies at the hosting layer as a bare 500 Internal Error before any endpoint runs -- which is exactly the failure mode this paragraph now documents, since it leaves no trace in the dashboard and no queued retry. Refs #148 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`npm audit fix` (no --force) plus `npm update yaml` to pull cosmiconfig's yaml onto 1.10.3. Everything resolved inside the existing semver ranges in package.json, so no direct dependency declaration changed. Notable bumps: next 16.1.6 -> 16.3.2, protobufjs 7.5.4 -> 7.6.5, websocket-driver 0.7.4 -> 0.7.5, tar 7.5.9 -> 7.5.22, sharp 0.34.5 -> 0.35.3, postcss 8.4.31 -> 8.5.23, nanoid 3.3.11/5.1.6 -> 3.3.18/5.1.16, @grpc/grpc-js 1.9.15/1.14.3 -> 1.9.16/1.14.4, ws 7.5.10/8.19.0 -> 8.21.3, lodash 4.17.23 -> 4.18.1, firebase-tools 15.8.0 -> 15.28.1. npm audit: 39 -> 5 vulnerabilities; npm audit --omit=dev: 11 -> 0. The 5 that remain are dev-only transitives of firebase-tools (@opentelemetry/core, @google-cloud/pubsub, gaxios, uuid); a later commit takes firebase-tools to v16 to clear them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`npm audit fix` (no --force) plus four narrowly scoped overrides.
Lockfile-only bumps: protobufjs 7.5.4 -> 7.6.5,
websocket-driver 0.7.4 -> 0.7.5, node-forge 1.3.3 -> 1.4.0,
lodash 4.17.23 -> 4.18.1, fast-xml-parser 5.4.1 -> 5.11.0,
fast-xml-builder 1.0.0 -> 1.3.1, joi 17.13.3 -> 17.13.6,
qs 6.14.2 -> 6.15.3, express 4.22.1 -> 4.22.2,
path-to-regexp 0.1.12 -> 0.1.13, form-data 2.5.5 -> 2.5.6,
@grpc/grpc-js 1.14.3 -> 1.14.4, js-yaml 3.14.2 -> 3.15.1,
@google-cloud/storage 7.19.0 -> 7.22.0, body-parser 1.20.4 -> 1.20.6.
Overrides added, and why each one is needed:
gaxios/google-gax/teeny-request -> uuid ^11.1.1
GHSA-w5hq-g745-h8pq. These three pin uuid ^9, and the only upstream
fix npm offers is firebase-admin 14 (a major upgrade). All three call
only uuid.v4() with no `buf` argument, so the advisory is unreachable
here; the override just removes the vulnerable copy. Clears the last
8 production advisories (uuid, gaxios, google-gax, teeny-request,
retry-request, @google-cloud/storage, @google-cloud/firestore,
firebase-admin all chain from this one package).
firebase-functions-test -> ts-deepmerge ^8.0.0
ts-deepmerge prototype-override DoS; dev-only, and npm's suggested
fix is downgrading firebase-functions-test to 0.3.3.
npm audit: 29 -> 0 vulnerabilities, dev and production alike.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v14 breaking changes and how each one lands here:
Drops Node 18/20 (engines: node >=22)
Already satisfied. functions/package.json pins engines.node "22" and
every workflow uses node-version 22.x, since 466f393 moved us onto
Node 22 with firebase-functions v7. (The "Node 20 runtime" note in
CLAUDE.md is stale and should be corrected separately.)
Removes the deprecated legacy namespace
No impact. Every call site already uses the modular subpath entry
points -- firebase-admin/app, /firestore, /auth, /storage, /functions.
There is not a single `import * as admin from "firebase-admin"` or
`require("firebase-admin")` in the tree.
Removes the Instance ID service and the legacy FCM messaging types
No impact; neither service is used.
Error handling revamp (#3140)
Additive for us -- it exposes error classes and adds `cause`/
`HttpResponse` to ErrorInfo. Nothing in functions/src branches on a
firebase-admin error code; the only `e.code` reads are in
mail-delivery.ts, against AWS SES / node syscall errors.
`tsc` compiles clean with no source changes. Two things did need changing:
firebase-functions ^7.2.2 -> ^7.3.2. Not incidental: 7.2.2 declares peer
firebase-admin "^11.10.0 || ^12.0.0 || ^13.0.0" and a clean install fails
ERESOLVE against it. 7.3.2 adds ^14.0.0 to that range.
jest.config.js has to widen next/jest's transformIgnorePatterns to let
`jose` through babel. v14 -> jwks-rsa 4.x -> jose 6.x, which is ESM-only
with no `require` condition, and Jest leaves node_modules untransformed --
so every suite reaching firebase-admin/auth failed to parse. Without this,
26 of 74 suites and 284 tests go red; with it, the suite is 74/74 and
1000/1000 under the emulators. The rationale is written out at the change
itself. Production never had this problem: functions/ is "type": "module"
on Node 22.
Also drops firebase-functions-test from devDependencies. This is the one
change here that is not a version bump, so: the package is imported by
exactly zero files in the repo -- the backend suite drives the functions
over HTTP against the emulator instead -- and it is the hard blocker for
v14, because 3.5.0 (latest) still declares peer firebase-admin "^8.0.0 ||
... || ^13.0.0" with no v14-compatible release published. A clean
`npm install` into an empty node_modules fails ERESOLVE with it present;
only an incremental install over an existing tree masks it, so CI's
`npm ci` would have broken. Removing an unused package beats pinning CI to
--legacy-peer-deps. Revert this line and firebase-admin has to stay on v13.
That removal also retires the firebase-functions-test -> ts-deepmerge ^8
override from the previous commit, and v14's move to @google-cloud/
firestore 8.x / google-gax 5.x / gaxios 7.x retires the google-gax -> uuid
override. The gaxios and teeny-request uuid overrides are still
load-bearing: @google-cloud/storage 7.22.0 still pulls gaxios 6.7.1 and
teeny-request 9.0.0, both of which pin uuid ^9. Verified by removing them
and watching uuid ^9 come back.
Dependency tree drops from 654 packages to 417. npm audit stays at 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There is no firebase-tools v16 to upgrade to -- the package has never published one. Latest is 15.28.1, which the lockfile fix already installed; my earlier note that clearing the remaining root dev advisories "needs firebase-tools 15 -> 16" was speculation and is wrong. Those 5 advisories (@opentelemetry/core and @google-cloud/pubsub, plus gaxios -> uuid, and firebase-tools itself by association) have no fix at any published version. What was actually stale is CI. All three workflows install a *global* firebase-tools@15.8.0, and that global binary -- not the lockfile copy -- is what runs emulators:exec in the test job and `deploy` in both deploy jobs. So CI kept executing the exact version whose advisories the lockfile bump had just cleared. Pinning all three to 15.28.1 lines CI up with the audited tree. Also raises the declared floor from ^15.8.0 to ^15.28.1 so a future lockfile regeneration cannot drift back onto a vulnerable 15.8.x. 15.8.0 -> 15.28.1 is a same-major bump and every CLI surface the workflows use is unchanged: `deploy --only <targets> --force`, `emulators:exec <script>`, `use <alias>`, `experiments:enable webframeworks`. Verified against the 15.28.1 binary's own --help, and the backend suite was run through node_modules/.bin/firebase at 15.28.1 rather than a global. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dependency security sweep: clear all resolvable Dependabot alerts, firebase-admin v14
There was a problem hiding this comment.
Pull request overview
Synchronizes main with 187 commits from test, delivering the multi-provider architecture, encrypted queued payloads, account/contact-email workflows, redesigned UI, documentation, and CI upgrades.
Changes:
- Adds Drive, Dataverse, and Zenodo storage workflows, compaction/finalization, OAuth, and retry infrastructure.
- Introduces payload encryption, contact-email verification, upload notifications, and expanded account cleanup.
- Refreshes branding, accessibility, documentation, tests, dependencies, and CI.
Reviewed changes
Copilot reviewed 98 out of 304 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
styles/Home.module.css |
Removes obsolete landing styles. |
public/site.webmanifest |
Adds DataPipe PWA branding. |
public/icon-tile-outline.svg |
Adds outlined app icon. |
public/icon-tile-light.svg |
Adds light app icon. |
public/icon-tile-dark.svg |
Adds dark app icon. |
public/favicon.svg |
Adds vector favicon. |
public/favicon-32.svg |
Adds 32px favicon source. |
public/favicon-16.svg |
Adds 16px favicon source. |
pages/contact.js |
Aligns contact-page spacing. |
pages/404.js |
Adds branded 404 page. |
pages/_app.js |
Adds theming, metadata, and layout updates. |
package.json |
Updates linting and dependencies. |
next.config.js |
Redirects legacy API documentation. |
lib/utils.js |
Delegates authentication error messages. |
lib/user-bootstrap.js |
Creates missing federated-user documents. |
lib/osf-utils.js |
Removes obsolete OSF helpers. |
lib/osf-sunset.js |
Centralizes OSF retirement behavior. |
lib/feature-flags.js |
Enables color-mode controls. |
jest.config.js |
Supports ESM-only jose. |
functions/tsconfig.json |
Enables ES2020 module output. |
functions/src/validate-json.ts |
Normalizes blank required fields. |
functions/src/validate-csv.ts |
Normalizes blank required columns. |
functions/src/scheduled-token-refresh.ts |
Dispatches provider-specific refreshes. |
functions/src/resolve-token.ts |
Resolves credentials through providers. |
functions/src/put-file-osf.ts |
Supports nested OSF paths. |
functions/src/providers/registry.ts |
Adds provider registry. |
functions/src/providers/index.ts |
Registers provider adapters. |
functions/src/providers/gdrive-oauth.ts |
Implements Drive token refresh. |
functions/src/persist-pending.ts |
Encrypts pending payloads. |
functions/src/on-user-deleted.ts |
Uses centralized account purge. |
functions/src/metadata-process.ts |
Removes superseded metadata logic. |
functions/src/metadata-download.ts |
Removes superseded download logic. |
functions/src/metadata-derived-files.ts |
Builds flattened Psych-DS files. |
functions/src/index.ts |
Exports new Cloud Functions. |
functions/src/get-provider-access-token.ts |
Exposes authorized Picker tokens. |
functions/src/generate-oauth-state.ts |
Supports provider OAuth state. |
functions/src/delete-account.ts |
Moves account deletion server-side. |
functions/src/compaction-gate.ts |
Diverts writes during compaction. |
functions/src/app.ts |
Exposes Cloud Tasks client. |
functions/src/api-queue-status.ts |
Decrypts downloadable queue data. |
functions/src/api-messages.ts |
Adds provider/finalization messages. |
functions/src/__tests__/scheduled-pending-recovery-emulator.test.js |
Tests recovery path mapping. |
functions/src/__tests__/providers-registry.test.js |
Tests provider registration. |
functions/src/__tests__/providers-container-input.test.js |
Tests provider input declarations. |
functions/src/__tests__/oauth-connect-scheduled-regression.test.js |
Guards scheduled OSF behavior. |
functions/src/__tests__/metadata-process.test.js |
Removes obsolete metadata tests. |
functions/src/__tests__/early-persist-emulator.test.js |
Extends the OSF mock listing. |
functions/src/__tests__/base64data-emulator.test.js |
Isolates test log identifiers. |
functions/package.json |
Updates function dependencies and tooling. |
functions/metadata/VENDORED_FROM.json |
Records vendored metadata provenance. |
functions/metadata/tsconfig.json |
Removes obsolete vendor build config. |
functions/metadata/tests/metadata-module.test.ts |
Removes obsolete vendor tests. |
functions/metadata/src/AuthorsMap.ts |
Removes vendored source copy. |
functions/metadata/rollup.config.mjs |
Removes obsolete Rollup config. |
functions/metadata/README.md |
Documents vendored metadata workflow. |
functions/metadata/package.json |
Defines prebuilt metadata package. |
functions/metadata/LICENSE |
Adds upstream license. |
functions/metadata/jest.config.cjs |
Removes obsolete Jest config. |
functions/metadata/dist/AuthorsMap.d.ts |
Adds generated type declarations. |
functions/.env.local |
Adds emulator-only provider settings. |
functions/.env.datapipe-test |
Adds deployed test configuration. |
firestore.indexes.json |
Adds queue query index. |
firebase.json |
Routes new backend endpoints. |
eslint.config.mjs |
Migrates to ESLint flat config. |
docs/brand/logo/mark-mono.svg |
Adds monochrome logo asset. |
docs/brand/logo/mark-light.svg |
Adds light logo asset. |
docs/brand/logo/mark-dark.svg |
Adds dark logo asset. |
docs/brand/logo/icon-tile-outline.svg |
Adds outlined brand icon. |
docs/brand/logo/icon-tile-light.svg |
Adds light brand icon. |
docs/brand/logo/icon-tile-dark.svg |
Adds dark brand icon. |
docs/brand/logo/favicon-32.svg |
Adds branded favicon source. |
docs/brand/logo/favicon-16.svg |
Adds compact favicon source. |
components/ui/GuidanceLine.js |
Adds shared guidance text. |
components/ui/FormErrorAlert.js |
Adds shared error surface. |
components/ui/EmptyState.js |
Adds shared empty-state panel. |
components/ui/ConfirmDialog.js |
Adds reusable confirmation dialog. |
components/ThemeSelect.js |
Adds theme preference menu. |
components/TestEnvironmentWarning.js |
Improves test-environment warning. |
components/SignUpWithOSF.js |
Removes OSF signup. |
components/SignInWithOSF.js |
Updates legacy OSF sign-in. |
components/OsfSunsetNotice.js |
Adds OSF retirement notice. |
components/LogoMark.js |
Adds reusable animated logo mark. |
components/Loader.js |
Removes obsolete loader. |
components/home/hero-snippets.js |
Defines landing-page code samples. |
components/home/BandMark.js |
Adds decorative landing mark. |
components/docs/DocsSection.js |
Adds anchored documentation sections. |
components/docs/DocsPager.js |
Adds documentation navigation. |
components/docs/DocsLayout.js |
Adds shared documentation layout. |
components/docs/ApiPrimitives.js |
Adds API documentation primitives. |
components/dashboard/MetadataControl.js |
Uses reliable shared settings control. |
components/CopyButton.js |
Improves clipboard feedback. |
components/AuthCheck.js |
Adds contact-email gate. |
components/auth/AuthProviderButtons.js |
Adds federated authentication controls. |
components/account/OSFToken.js |
Removes legacy OSF token UI. |
components/account/OsfRelinkButton.js |
Adds OSF reauthorization control. |
components/account/OneClickAuth.js |
Removes obsolete OSF linking UI. |
components/account/AddSignInMethodBanner.js |
Prompts legacy users to link sign-in. |
.gitignore |
Includes generated metadata and emulator config. |
.github/workflows/run-migration.yml |
Adds contact-email migration. |
.github/workflows/node.js.yml |
Adds linting and stabilizes CI. |
.github/workflows/metadata-drift-check.yml |
Monitors vendored metadata drift. |
.github/workflows/firebase-deploy.yml |
Adds production SES configuration. |
.github/workflows/firebase-deploy-test.yml |
Adds test provider and SES configuration. |
.eslintrc.json |
Removes legacy ESLint config. |
__tests__/validate-json.test.js |
Tests blank required fields. |
__tests__/validate-csv.test.js |
Tests blank required columns. |
__tests__/status-indicator.test.jsx |
Tests status accessibility. |
__tests__/settings-section.test.jsx |
Tests settings-section structure. |
__tests__/osf-sunset.test.js |
Tests OSF retirement helpers. |
__tests__/google-picker.test.js |
Tests Drive Picker integration. |
__tests__/form-error-alert.test.jsx |
Tests shared error alerts. |
__tests__/experiment-info.test.jsx |
Tests provider-aware experiment details. |
__tests__/confirm-dialog.test.jsx |
Tests confirmation behavior. |
__tests__/404.test.jsx |
Tests custom 404 content. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| * CSV/sidecar stems, which follow the same encoded name, collision-free too). | ||
| */ | ||
| function flattenName(dataFilename: string): string { | ||
| return dataFilename.replace(/[/\\]+/g, '-'); |
| # the real FIRESTORE_KEY_TEST secret on the deployed test site because | ||
| # .env.<project> beats .env. Left as-is to avoid orphaning already-encrypted | ||
| # test tokens; worth revisiting as a separate change. | ||
| TOKEN_ENCRYPTION_KEY=abababababababababababababababababababababababababababababababab |
| {deadline ? ` and will stop writing to it after ${deadline}` : ""}. Data | ||
| already on OSF is unaffected and stays in your OSF account. To keep | ||
| collecting, connect another storage provider in{" "} | ||
| <ChakraLink asChild color="brandOrange.300"> |
| }; | ||
|
|
||
| return ( | ||
| <Dialog.Root open={open} onOpenChange={onOpenChange}> |
Comment on lines
+29
to
+33
| // gdrive pass was wrapped and swallowed. A uniform loop must pick | ||
| // one convention -- we wrap every provider, so one provider's | ||
| // failure can never prevent another's pass from running. Failures | ||
| // are logged rather than thrown, and are not re-thrown in | ||
| // aggregate after the loop. |
Comment on lines
+41
to
+43
| } catch (err) { | ||
| console.error("Failed to initiate OSF authorization:", err); | ||
| setIsLoading(false); |
Comment on lines
+5
to
+6
| `"file:metadata"` and `functions/src` imports `../metadata/dist/index.js` directly, so | ||
| deploys need no metadata build step. |
- metadata-derived-files: make flattenName injective by percent-escaping %, / and \ instead of collapsing separators to hyphens, so "condition-A/data.json" can no longer shadow "condition-A-data.json" at the raw path and collision-cache claim; regression tests added - ConfirmDialog: ignore Escape/backdrop closes while the confirm request is pending, so the progress and error surface can't vanish mid-flight - OsfSunsetNotice: mode-aware brandOrange.subtle/border surface and brandOrange.fg underlined link, fixing dark-on-dark text and color-only link differentiation in light mode - OsfRelinkButton: surface authorization-start failures via the shared FormErrorAlert instead of console-only logging - functions/metadata/README: imports go through the @jspsych/metadata package name, not ../metadata/dist paths Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner decision: the light rendering never looked right, and a twice-a-year control is not worth carrying UI surface and a second design target for every future change. - _app.js pins forcedTheme="dark" so a Light/System preference stored while the toggle existed cannot resurrect the retired mode - ThemeSelect and the COLOR_MODE_TOGGLE flag (lib/feature-flags.js) are deleted; the navbar drops both gated menu blocks - DESIGN.md §2 rewritten as the retirement record: semantic tokens remain mandatory, _light branches stay as the inert escape hatch Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six targeted edits across four pages: a grammar fix and two clarity fixes on sending-data (name the Accept new data switch, align the "three answers" claim with the API reference's 500 row), parallel phrasing for the Dataverse connect section, an early Zenodo-only note on the finalizing page, and smoother wording in condition assignment. No section ids, nav entries, or structure changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sgaqqAebNujGsZ5mKbK52
The testbed's result contract goes to schema 2 (jspsych/datapipe-testbed#4), and the runbook is what has to use it. The procedure now waits for `ready` before sending the start key and treats `running` as proof that the key landed -- which is the first verification that step has ever had -- and polls `data-testbed-trials-completed` to act at a trial instead of timing it off the clock. It also reads `schema` first and falls back: a cached schema-1 page has neither, and a driver that assumes otherwise silently reverts to the guesswork this change exists to remove. The dashboard expectations catch up with #261 and #262. The rejections panel is rewritten from the merged component -- the neutral panel with its red left edge, the exact headline and body, the "raw data was kept" line on METADATA_ERROR rows, the "Clear this list" button, and the fact that counts are since the last clear rather than lifetime. The queue panel section says the polish PR is in flight, so a run asserts on the three kinds and the panel's weight and reports a column heading it does not recognise as mid-polish rather than as a failure. Four things the 2026-09-19 run learned the hard way are written down: the rejections panel is hidden whenever anything is queued (which is why #261's reworded copy was never seen on the new build, and why checking it after a recovery scenario means waiting out an hour), Firebase ID tokens expire after about an hour so a poller must re-read from IndexedDB, a fullscreen Chrome window ignores resize_window so responsive checks need a window that is not, and a METADATA_ERROR copy is visible in the queue for only about five minutes. The METADATA_ERROR reword is recorded as the argument for "assert on the code, never the message" -- the same probe, six hours apart, with two different messages and one code. Production still answers the old string, and the old UI with it, until test is promoted to main; a run pointed there has to say so. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Checked against datapipe-test: Hosting matches a rewrite with or without the slash. The 308 only happens on a path with no rewrite. Not an upstream bug in datapipe-client. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Queue panel: fix retention label/value, layout wrapping, and filename noise
e2e runbook: use the schema-2 result contract, and today's dashboard copy
Corrections from a second live run against datapipe-test: the start-key procedure now leads with a screenshot (a hidden tab swallows keypresses), computer-tool coordinates need scaling to CSS px, javascript_tool's ~45s ceiling and the extension's cookie/query-string block are documented, and a real theme override (next-themes, forcedTheme="dark") replaces the old "no theme toggle" claim. The queue panel's "mid-polish" warning is replaced with the now-landed concrete strings, and the rejections panel section gets its first-ever "Clear this list" measurements. Several timing facts are corrected with exact OBSERVED figures, including that a METADATA_ERROR-kept queue entry's nextRetryAt is +1 minute, not +60 like a recovered partial. SKILL.md stays under its 250-line hard limit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pages/_app.js forces dark (forcedTheme) and says why: light is a retired mode. The runbook described a way to flip to light for screenshots under a heading that a theme override exists; what that shows is unsupported, and the datapipe-color-mode localStorage key a run saw is a leftover from when a toggle existed. Nothing to check in light mode. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The runbook and reference files had OBSERVED stamps, clock times, and run-by-run narration baked in from being written live against actual test runs. None of that belongs in a public repo — it's the record of a specific run against datapipe-test, not documentation. Rewrite every such sentence to keep the durable fact or procedure (timing ranges, literal UI strings, known issues, the production/test wording gap) and drop the date, the clock time, and the narration of what a particular run did. Also add an explicit rule (SKILL.md §10, one sentence in docs/e2e-testing.md, and instructions at the top of output-template.md): what a run learns goes in its per-run report in the gitignored e2e-reports/, never back into these files, which hold only timeless procedure or facts. That's what stops the history from growing back. SKILL.md stays at the 250-line hard limit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The driver runbook, its overview and the run reports belong with the pages and the manifest they drive, and run reports do not belong in the product repo at all. jspsych/datapipe-testbed now holds all of it; this repo keeps a short pointer and nothing else. e2e-reports/ is gone from .gitignore because reports no longer land here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
scheduled-staging-sweep.ts hands a recovered session to queueUpload with no providerErrorCode -- nothing has been attempted, so there is no provider failure to report -- and queueUpload's tiering put anything without a code on the hour-long default. That default exists for a genuine first failure with an unknown cause; a recovered partial has no failure at all, so it landed on it by accident. Observed live: recovered 00:15:13Z, first attempt not scheduled until 01:15:13Z. It also defeated an ordering scheduled-sweep-core.ts's runSweep documents on purpose: the staging sweep runs before upload retry in the same invocation specifically so a session recovered this tick can be picked up by the SAME tick's upload-retry pass. With nextRetryAt an hour out, the retry worker's nextRetryAt <= now query skipped it for roughly twelve ticks regardless. queueUpload takes an optional attemptImmediately, set only by the staging sweep's call, which sets nextRetryAt = now instead of the hour default. Every later retry is untouched -- computeBackoffMs still keys off the providerErrorCode of whichever attempt actually failed, so a recovered partial that fails on its first (now immediate) try gets exactly the same tier and notification treatment as any other first failure. The one existing branch that could have used this -- refreshing an already-pending doc's payload -- deliberately does not: that doc has its own retry history (a real backoff, a compaction hold, or an earlier recovery already on the immediate schedule), and attemptImmediately only describes an entry this call is about to WRITE. Docs and the e2e skill corrected: recovery still waits out ABANDON_GRACE_MS (10 minutes) plus the sweep cadence, but the "roughly an hour" and "65-75 minutes" delivery estimates that assumed the old default are gone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
For a metadata-active experiment, metadata-derived-files.ts includes
.psychds-ignore among the derived files on every submission, and
metadata-derived-upload.ts relies on the provider's NAME_CONFLICT to make
a repeat write a no-op. Google Drive permits duplicate names and never
sends one, so every submission after the first landed a fresh copy.
Observed live: five copies after five uploads. providers/types.ts already
states the principle this violates -- collision detection belongs in
Firestore, not in a round trip to the provider.
psychds-ignore-claim.ts adds that Firestore-side collision detection: a
transaction on experiments/{id} that sets psychdsIgnoreWrittenAt the
first time, and returns false to every caller after. api-data.ts checks
the field on the experiment document it has already loaded for this
request -- zero extra reads for the overwhelming majority of submissions
-- and only runs the claim transaction when the field is absent, dropping
.psychds-ignore from derivedFiles before any of the several paths that
consume it (the direct upload and every queueDerivedFiles call, including
the compaction-hold one).
The claim is not released on an ordinary queue-and-retry failure -- the
retry worker will eventually land the file, and releasing early would
just race it into writing a second copy. It IS released when a write is
neither landed nor queued (queueDerivedFiles's own queueUpload throwing),
and when compaction's restoreIgnoreFile fails to put a borrowed slot's
copy back -- both are cases where nothing is left to retry the write, so
without a release nothing ever would again. finalization.ts has no
equivalent restore step: it deletes .psychds-ignore permanently as part
of merging the record, and a finalized experiment never submits again, so
there is nothing for a stale claim to block.
Known, accepted limitation, documented in the new module's header: an
experiment that already has the file but no claim -- every metadata-active
experiment that predates this -- gets one more redundant copy on its next
submission and then stops for good.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
At normal widths "19/09/2026, 18:33:48 GMT-4" dropped "GMT-4" onto a second line. whiteSpace="nowrap" on the Time column's header and cell text fixes it; the "What happened" column, which is prose, absorbs the flex it gives up. The table's existing horizontal-scroll container is untouched, for narrow screens. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… alone claimPsychdsIgnore runs a transaction on the experiment document from inside api-data.ts's request path, before the participant's raw data is uploaded. That document is hot -- every submission increments sessions on it -- so under a burst the transaction can abort, and an escaped throw would 500 a submission over a 24-byte marker file. It now never throws: if the claim cannot be decided the answer is 'write it', whose worst case is one duplicate copy, the behaviour this module replaced. Also restores .claude/skills/e2e-testbed and docs/e2e-testing.md to test's versions: end-to-end material is moving to the datapipe-testbed repo, and the timing facts this PR changes are updated there instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Move the e2e runbook out: end-to-end testing lives in datapipe-testbed
Recovered partials upload immediately; .psychds-ignore claimed once per experiment; rejections Time column no-wrap
Existing researchers returning to DataPipe after the September 2026 update need one page that says what's different, action-required changes first: renamed error codes, a removed body parameter, the OSF wind-down, and confirmed contact email, then what's new (providers, saving as you go, the failure queue, rejected submissions, metadata, finalizing, sign-in, privacy, the redesign). Listed first in the docs sidebar and pointed to from the docs overview page. Also fixes two references to the error-code rename that named the day it landed on this branch (16 September 2026) rather than the month the change ships to researchers, and links both to the new page. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Maintainer-requested exception to DESIGN.md §8's "modal as first thought" ban, for exactly one thing: telling a returning researcher that a large update shipped and pointing them at /docs/whats-changed. Chakra v3 Dialog, same conventions as components/ui/ConfirmDialog.js and components/account/SelectAuth.js's dialogs; Dialog.Title/ Dialog.Description give it a proper aria-labelledby/aria-describedby. Dismissal follows components/OsfSunsetBanner.js's pattern exactly: a versioned localStorage key, useSyncExternalStore with a server snapshot of "dismissed" so it never flashes open during hydration, and a session-local fallback so it still closes when storage is unavailable. The primary link, "Dismiss", the close control, and Esc/ backdrop all clear it for good. Mounted from pages/index.js only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs-nav.test.js: "What's changed" is DOCS_NAV's first entry, and every page's href resolves to a real file under pages/docs. whats-changed-page.test.jsx: the page renders and every internal /docs/... link on it resolves to a page that exists. whats-changed-modal.test.jsx: the dialog shows when nothing is stored, stays hidden once dismissed, renders nothing in the server snapshot, and each of the four ways to close it (the primary link, Dismiss, the close control, Escape) records the dismissal -- including when localStorage throws, where it still closes for the session. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The lead and the modal promised that existing experiments keep working 'exactly as before' and that nothing breaks a study in progress -- which the page's own first section contradicts, since renamed error codes can break a study's retry logic. What holds is that collection keeps running. And the contact email is not only for new accounts creating an experiment: ContactEmailGate blocks every dashboard route for any signed-in researcher without a usable address. Returning researchers are who this page is for, so tell them they will be asked for one when they next sign in, and that their experiments keep collecting whether or not they have. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The page still warned that nothing in the dashboard prevents unchecking both formats. ExperimentValidation.js refuses the last uncheck now; the server is unchanged, so an experiment already in that state still rejects everything. Say both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add a What's changed doc page and a one-time homepage modal
Shorter sentences, active voice, positive statements, parallel sub-headings that each say what happened, and no em dashes. The lead no longer repeats the page header. Three things were also wrong or unhelpful rather than merely wordy: - 'A light/dark toggle shipped briefly and was retired' -- it only ever existed on the test branch; production users never saw one. - 'A rejected submission was always gone' -- not so for a metadata failure, whose raw data is kept. Dropped the absolute. - 'It doesn't touch the lifetime count DataPipe keeps behind it' -- a number no researcher can see. Replaced with what they will notice: later refusals appear as they happen. The OSF 'what to do' now names the actual prompt (re-authorize OSF) instead of describing how an account 'owns its experiments some other way'. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
In Chakra v3 Dialog.CloseTrigger is not only behaviour: it is the dialog recipe's closeTrigger slot, positioned absolutely in the top corner. The footer's Dismiss was wrapped in a second one, so it was lifted out of the footer and drawn on top of the x. It is a plain button that calls dismiss() now, the way ConfirmDialog's Cancel is. jsdom does no layout, so the tests could not see the overlap. They can see its cause: exactly one close-trigger part may exist, and Dismiss must sit beside the primary link. That test fails against the old component. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The page opened with a checklist of things that might break, which is the right order for a changelog and the wrong one for an explanation: it gave the release no shape. One fact drives it -- OSF is closing the feature DataPipe was built on -- and the rest follows in order of size: Where your data goes OSF closing; three new providers How your data gets there saving as it runs; when an upload fails What DataPipe writes Psych-DS metadata; finalizing Your dashboard rejected submissions; the redesign Your account sign-in; contact email; privacy Changes to the API renamed codes; metadataOptions What you may need to do one checklist gathering every action item The small API changes are the ones that break code, so a callout at the top jumps to them and to the checklist, and each checklist item links back to the section that explains it. Tests pin the section order and check that every in-page link lands on an id that exists. The API docs' two links to this page now point at #error-codes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keep the What's changed modal's Dismiss button in the footer
What's changed: narrative order, and a copyedit
WaterButler reports file ids as "osfstorage/<id>", and listFiles and putFileOSF pass that id through verbatim. osfFilesLink already ends in "/providers/osfstorage/", so updateFile and downloadFile built ".../providers/osfstorage/osfstorage/<id>" and missed the file. main's metadata-process.ts stripped the prefix; the strip was lost in 4e889f3 when the metadata block moved to tracking a provider file ref. The effect on a metadata-active OSF experiment: the dataset_description.json update misses, the self-heal re-create hits OSF's 409, and the submission fails with 400 METADATA_ERROR -- every submission for experiments that already have the file, and from the second submission on for the rest. Tests used bare ids and a fake filesLink, so they could not see it. The new unit tests use the real upload-link shape; the new emulator cases 4b/5b feed prefixed ids through /api/data with OSF's 409 on re-create, and fail with 400 on the unfixed adapter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
providers-gdrive.test.js's section 7 expected downloadFile to GET filesLink + "osfstorage/111", i.e. the doubled-provider URL this branch fixes. It now expects the bare id. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fix OSF metadata updates: strip WaterButler's osfstorage/ id prefix
200 rested on the premise that main runs apidata at the default concurrency of 80 (about 1,600 slots), so 20 x 1 on this branch was a 98% cut. main actually sets concurrency: 1 on both apidata and apibase64, each at the global 20 -- 40 simultaneous submissions across the two endpoints. /api/base64 is now dispatched from inside apidata, so 40 keeps production's existing combined capacity instead of halving it. The ceiling costs nothing at idle but bounds what a flood of requests to this unauthenticated endpoint can spend; 200 gave up most of that bound for a lecture-hall burst that has never been observed. The comment now says to raise it from measured peak concurrency and 429 rate. Also fixes a stray "200" in the timeoutSeconds paragraph that should read 300, and pins the capacity test to exactly 40. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syncs
mainwith everything merged totestsince the last release sync (#149, April 2026) — 187 commits across PRs #150–#187.Highlights
Multi-backend provider migration (#154–#161, #164) — core architecture for sending data to providers beyond OSF, plus:
Required contact email (#184) — schema + Firestore rules, signup seeding, verification round trip, first-failure notification state machine, backfill migration.
Payload encryption at rest (#185) and mail delivery via SES trigger, replacing the deprecated extension.
Metadata — version-drift fix, sidecar files, discoverability, follow-ups (#150–#153).
Docs — full restructure with new nav and redirects (#186), multi-backend guides (#178), 32 MB limit documentation.
Design pass — account settings redesign (#179), light mode (#181), homepage cleanup (#182), logo animation (#180), spacing pass (#183), branded 404 (#42).
CI & deps — ESLint on every PR (#109), dependency refresh clearing all advisories, firebase-admin v14, pinned firebase-tools (#187).
🤖 Generated with Claude Code