bug: heartbeat after a bucket delete returns 500 (stale last_heartbeat cache)

aw-server-rust keeps the cached last heartbeat across a bucket delete, so the
next heartbeat that takes the merge path returns HTTP 500 and the event is not
stored.

Reproduction

Against the shipped v0.14.1 binary (activitywatch-v0.14.1-linux-x86_64.deb,
reports {"version":"v0.14.1 (rust)"}), on a throwaway database:

POST /api/0/buckets/probe            -> 200
POST /api/0/buckets/probe/heartbeat?pulsetime=2  (t+0) -> 200
POST /api/0/buckets/probe/heartbeat?pulsetime=2  (t+1) -> 200
POST /api/0/buckets/probe/heartbeat?pulsetime=2  (t+2) -> 200
DELETE /api/0/buckets/probe          -> 200
POST /api/0/buckets/probe            -> 200
POST /api/0/buckets/probe/heartbeat?pulsetime=2  (t+3) -> 500
{"message":"replace_last_event matched 0 rows for event_id 31 - cache/DB inconsistency"}

The last heartbeat falls inside the pre-delete event's end + pulsetime, so
Datastore::heartbeat() takes the merge branch (aw-datastore/src/datastore.rs),
merges into the cached event, and replace_last_event() is asked to update an
event id that was deleted with the bucket.

A heartbeat that lands outside the pulse window takes the insert branch and
succeeds, so the failure looks intermittent to a caller.

A minimal reproduction is also in ErikBjare/bob's
scripts/research/aw-clock-step-probe.py (probe_delete_then_heartbeat, run
with --http-base).

Cause

Command::InsertEvents invalidates the cache:

// aw-datastore/src/worker.rs:498
self.last_heartbeat.insert(bucketname.to_string(), None); // invalidate last_heartbeat cache

but Command::DeleteBucket (worker.rs:478-484) and Command::DeleteEventsById
(worker.rs:568-576) do not, so last_heartbeat keeps an id that no longer
exists.

aw-server (Python) already handles both cases by popping self.last_event in
delete_bucket() and delete_event(), which is what surfaced this: the same
sequence returns 200 there.

Suggested fix

Invalidate the cached last heartbeat in the delete paths too:

Command::DeleteBucket(bucketname) => {
    self.last_heartbeat.remove(&bucketname);
    // ... existing ds.delete_bucket() handling
}
Command::DeleteEventsById(bucketname, event_ids) => {
    self.last_heartbeat.insert(bucketname.clone(), None);
    // ...
}

I did not open a patch: this needs a Rust build and a regression test
(delete → recreate → heartbeat within pulsetime, expecting 200 on both the merge
and insert paths), and I could not compile it here.

Why it matters: aw-client's retry queue re-sends the rejected heartbeat, so the
user-visible effect is a burst of 500s plus a delayed/misordered first event after
a bucket delete, not silent data loss. Found while measuring clock-step behaviour
(ActivityWatch/aw-server#223 is the related Python-server repair).

Verified against current aw-server-rust master — this is already fixed upstream, and the reproduction is explained by release lag rather than a still-live bug.

Already fixed in `aw-server-rust`

PR #826, "fix(datastore): invalidate heartbeat cache on delete and rename", merged 2026-10-26 as fe353a6. It does exactly what the "Suggested fix" describes, plus two paths the report did not cover:

Command::DeleteBucket(bucketname) => {
    // A bucket re-created under this id must not merge into the
    // deleted bucket's cached last event.
    self.last_heartbeat.remove(&bucketname);
    ...
Command::DeleteEventsById(bucketname, event_ids) => {
    // The cached last event may be one of the deleted events; a
    // heartbeat merging into it would update a missing row. Drop
    // it even if the delete fails partway.
    self.last_heartbeat.remove(&bucketname);

It also drops the cached heartbeat on RenameBucket (both ids) and clears it after MigrateTestBucketNames, since a rename onto a cached id or a partially-failing migration reaches the same state.

Regression coverage landed with it, in aw-datastore/tests/datastore.rs:

test_heartbeat_after_deleting_cached_event
test_heartbeat_after_recreating_bucket
test_heartbeat_after_renaming_bucket_onto_cached_id

So no patch from me — a PR here would duplicate fe353a6.

Why the failure still reproduces

The reproduction ran against the shipped activitywatch-v0.14.1-linux-x86_64.deb. v0.14.1 was published 2026-10-10, and #826 landed on 2026-10-26, so that binary predates the fix. Nothing to repair in the release that was tested.

Remaining gap: the v0.14.2 pin will not carry it

This is the part worth acting on. ActivityWatch/activitywatch master currently pins the submodule at:

aw-server-rust -> fe558c4f106a70004a42bc30558c7438f8b66b4f   (2026-10-17)

fe558c4 predates fe353a6 (2026-10-26), so v0.14.2 as pinned today would ship without the #826 fix — the same 500s after a bucket delete would survive into the next desktop release. A one-commit pin bump to any aw-server-rust sha at or after fe353a6 (current master is 10b4b7f, 2026-11-01) is all that is needed.

I filed that as a follow-up rather than doing it here, since the pin bump and the tag ordering belong with the rest of the pre-tag work. Flagging it now because the tag has not been cut yet, so the window to include this is still open.

Thanks for the precise reproduction and the pointer to aw-server's Python handling of self.last_event — that comparison is what made the cause obvious.

Correction to my previous comment — the pin-bump suggestion in it was wrong, and I want to retract it before anyone acts on it.

I wrote that "a one-commit pin bump to any aw-server-rust sha at or after fe353a6 is all that is needed." Checking the ancestry rather than assuming shows that is not available:

5ae2e89  build(deps): bump rusqlite 0.30.0 -> 0.37.0 (#531)   2026-10-19
5ae2e89 ancestor of fe353a6?  YES
5ae2e89 ancestor of fe558c4?  NO

The rusqlite major bump is the 8th commit after the current pin, so every route from fe558c4 to fe353a6 carries it. There is no commit that picks up #826 (or #832/#834/#838) while leaving the bundled SQLite untouched. ActivityWatch's pre-tag policy treats that rusqlite change as needing its own verification near a release, so reaching the heartbeat fix is a v0.14.3-sized move via the scheduled aw-tauri submodule bump, not a one-commit pin correction.

The c8830548 surgical lock that worked for #822 does not generalise, precisely because #822 is the first commit past the pin while this batch is the twelfth.

What does not change:

The bug is real against the shipped v0.14.1 binary, exactly as reported.
It is already fixed on aw-server-rust master by fix(datastore): invalidate heartbeat cache on delete and rename #826 (fe353a6, 2026-10-26) with three regression tests, so no PR is needed here.
v0.14.2 will ship without the fix; that is now recorded as an explicit known issue rather than as a pin problem.

Net effect for this issue: nothing to change in aw-server-rust. The remaining action is on the ActivityWatch release side, where the 2026-10-26 batch is carried to v0.14.3 by the existing submodule-bump vehicle. Apologies for the churn — better to correct it here than have someone try the bump and discover the rusqlite dependency mid-release.

Navigation Menu

Repository navigation

bug: heartbeat after a bucket delete returns 500 (stale last_heartbeat cache) #856

Reproduction

Cause

Suggested fix

Already fixed in `aw-server-rust`

Why the failure still reproduces

Remaining gap: the v0.14.2 pin will not carry it

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

bug: heartbeat after a bucket delete returns 500 (stale last_heartbeat cache) #856

Description

Reproduction

Cause

Suggested fix

Activity

TimeToBuildBob commented on Nov 5, 2026

Already fixed in aw-server-rust

Why the failure still reproduces

Remaining gap: the v0.14.2 pin will not carry it

TimeToBuildBob commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Already fixed in `aw-server-rust`