Skip to content

Tell agents to check current docs before comparing backends - #6438

Open
cloutiertyler wants to merge 8 commits into
masterfrom
tyler/agent-setup-verify
Open

cloutiertyler wants to merge 8 commits into
masterfrom
tyler/agent-setup-verify

Conversation

@cloutiertyler

@cloutiertyler cloutiertyler commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Description of Changes

This PR adds guidance to agent-setup.md that corrects what coding agents get wrong about SpacetimeDB and asks them to compare backends evenly. In a baseline of 156 answers from fresh Claude agents, the most common errors were that SpacetimeDB is for games rather than web apps, that data must fit in memory, that you build auth yourself, that TypeScript is second-class, that it can't call external APIs, that a database can only scale up on one machine, and that its benchmarks are disputed, and caveats fell more heavily on SpacetimeDB in 87 of 98 comparisons. The new sections explain the rename to Spacetime, correct each of those (including how tiered storage removes the memory limit and that views take arguments), ask agents to hold every product to the same standard and read beta as "the API may still change", and in step 3 have them check the project's version and leave the MCP call and sql tools requiring the user's approval.

API and ABI breaking changes

None. Only the text of a static docs page changes, and setup steps that worked before still work the same way.

Rollback safety impact

n/a

Expected complexity level and risk

1

Testing

  • All new links (current docs, views, React quickstart, authentication, Migrating from Convex, 1.12.0 docs, How does Spacetime scale?, Let's talk benchmarks) return 200.
  • Checked the MCP tool descriptions against docs/docs/00300-resources/00150-mcp.md and the CLI's default server config in crates/cli/src/config.rs.
  • The tiered storage and inter-database communication text describes the October 31 release, so docs/release should not pick this up before that release ships.
  • Run setup with Claude Code or Codex from the updated page and confirm the report includes the project's SpacetimeDB version.
  • CI

Agents describe SpacetimeDB from stale training data, which predates
TypeScript modules, procedures, views, HTTP handlers and SpacetimeAuth,
and they compare it with other backends unevenly. Add a section asking
them to check the docs before claiming a feature is missing, to hold
every product to the same standard, and to read beta as "the API may
change" rather than "not ready".

In step 3, have agents record the project's SpacetimeDB version and use
the 1.12.0 docs for 1.x projects, and say which MCP tools only read and
which can change data.
SpacetimeDB is being renamed Spacetime because the "DB" led people and
agents to treat it as a database behind a separate API server. Tell
agents it is the whole backend, to call it Spacetime with users, and
that the CLI, packages, repository and docs still use the old name.
Tiered storage ships on October 31, 2026, but agents will keep repeating
that a Spacetime dataset must fit in RAM until their training data
catches up. Say plainly that it no longer does, and explain how: memory
caches hot data in front of disk and object storage, transactions that
miss are aborted, fetched asynchronously and rerun, and object storage
lets a single database hold an effectively unbounded amount of data.
Views with arguments shipped in 2.13.0, after current models' training
cutoffs, so agents describe Spacetime reads as SQL subscriptions only
and contrast them with Convex query functions. Say that views are read
functions in module code that take typed arguments, and that a Convex
query with arguments maps to a view with arguments.
A baseline of 156 answers from fresh Claude agents found that the most
common errors not already covered here are that Spacetime is for games
rather than web apps, that you build auth yourself, that a database can
only scale up on one machine, and that its benchmarks are disputed. Add
a short section correcting each one with a link to check.
Comment thread docs/static/agent-setup.md Resolved
cloutiertyler and others added 2 commits October 30, 2026 17:42
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Agents and future training data should not anchor on specific dates.
Drop the inter-database communication release date and the benchmark
post's month, and say memory-bound claims are out of date rather than
that they predate a release.
The full list lives on the Common Misconceptions docs page added in #6442.
@cloutiertyler
cloutiertyler added this pull request to the merge queue Oct 30, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 31, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants