Skip to content

Let a cloud result replace one the cache will never serve - #219

Draft
jwrosewell wants to merge 2 commits into
rcfrom
fix/cloud-cache-replace
Draft

jwrosewell wants to merge 2 commits into
rcfrom
fix/cloud-cache-replace

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

The problem

Since #208 the Cloud Request Engine's response is a lazy task, and a result is offered to the cache while its request is usually still running. Such a result is stored, and is served only once its request has finished, without failing, with a response that has no top-level fodid member.

If the stored request then fails, or answers with a fodid, the entry is never served. The cache the engine is given did not replace an existing entry (LruPutCache.Builder has updateExisting false, and CacheConfiguration(int) uses it). So LruCacheBase.add dropped the next result for the same Evidence, and every later request for that Evidence went to the cloud service while the dead entry stayed in the cache. Every answer was still right. The cost was cloud requests, and a fodid response left in the cache.

Turning on updateExisting showed a second fault. LruCacheBase.add made the replacement on a list chosen at random and spliced it into the old item's place, which is in the old item's list. With more than one list, the default on any machine with more than one processor, the lists were crossed after the first replacement, and later moves and evictions threw NullPointerException. No main code set updateExisting before this change, so the path had never been live.

The change

  1. LruCacheBase.add makes the replacement on the old item's list, so it takes the old item's place in the list it belongs to.
  2. CloudRequestEngineBuilder.configureEngine sets any cache built by a PutCacheBuilder to replace an existing entry, before the base class builds the engine's cache. That covers setCacheSize, the cloud pipeline builders (DeviceDetectionCloudPipelineBuilder, IPIntelligenceCloudPipelineBuilder, GeoLocationCloudPipelineBuilder, which pass new CacheConfiguration(resultsCacheSize) to setCache), and a CacheConfiguration made with your own LruPutCache.Builder. The builder in the configuration is set, not copied. A cache built by another kind of CacheBuilder is left as it is.
  3. Tests.
    • LruPutCacheTests.LruPutCache_ReplaceWithSeveralLists replaces forty items in a cache with four lists, evicts past its size, and checks that each list runs from its first item to its last through items that belong to it.
    • CloudRequestEngineLazyCacheTests.process_ResultAfterAFailureIsCached and process_ResultAfterAFodIdIsCached make three requests with the same Evidence. The first is held back until its process has returned, so its result is stored while it is running, and then fails or answers with a fodid. Each expects two data requests and the third answer served from the cache.
    • process_ReplacedEntryThenEvictions replaces a failed entry and then fills a cache of four with forty other requests, each expected to get its own response.

Red and green

Run locally on Windows with JDK 21 and Maven 3.9.16, with a clean build each time.

  • LruPutCache_ReplaceWithSeveralLists on rc's LruCacheBase: fails, "The list's last item is not the end of its chain".
  • process_ResultAfterAFailureIsCached and process_ResultAfterAFodIdIsCached on rc's CloudRequestEngineBuilder: both fail, the third request goes to the cloud service again (expected [failed, {...}, {...}] but was [failed, {...}, failed]).
  • process_ReplacedEntryThenEvictions passes on rc's LruCacheBase too, on a 12 processor machine, so it is an end-to-end check and not the red for the list fault.
  • With the change: LruPutCacheTests 8 of 8, CloudRequestEngineLazyCacheTests 9 of 9, CloudRequestEngineFodIdCacheTests 5 of 5, CloudRequestEngineCacheKeyCaseTests 2 of 2, CloudRequestEngineTests 48 of 48. Pipeline Caching, Engines, Cloud Request Engine and FiftyOne, and the Device Detection, IP Intelligence and Location cloud modules, with everything they build on, pass, with a resource key set and ValueCloudTests left out (that clone had no data files).
  • ci/check-environment-variable-names.ps1 passes.

A full-matrix runner build of rc with this change is run 44958153864.

Before you upgrade

  • A Cloud Request Engine whose cache is built by a PutCacheBuilder now replaces an entry when a new result is put for its key, including where you passed your own LruPutCache.Builder with setUpdateExisting(false). That builder object is changed, not copied.
  • An LruPutCache built with setUpdateExisting(true) now keeps its lists whole after a replacement. LruLoadingCache never replaces, so it is not affected.

What remains

  • The fodid response is still stored while its request runs, because what it holds is not known until it finishes. It is never served, and the next result for the key now replaces it. Storing only finished results would need the cache key made on the processing thread and the put made when the task ends, which needs a new method on the cache interfaces.

The Cloud Request Engine's result is stored while its request is still
running. If that request then fails, or answers with a 51Did, the entry
is never served. The cache the engine is given does not replace an
existing entry by default, so the next result for the same Evidence was
dropped, and every later request for that Evidence went to the cloud
service while the dead entry stayed in the cache.

The Cloud Request Engine's builder now sets a cache built by a
PutCacheBuilder to replace an existing entry, which covers setCacheSize
and the cloud pipeline builders' CacheConfiguration made with a size.
Two tests make three requests with the same Evidence, the first failing
or answering with a 51Did, and expect the third to be served from the
cache.
With updateExisting set, LruCacheBase.add made the replacement on a list
chosen at random and spliced it into the old item's place, which is in
the old item's list. With more than one list, the default on any machine
with more than one processor, the old list kept the replaced item as its
first or last, the new list's first or last pointed into the old list,
and the replacement named a list it was not in. Later moves and
evictions walked these crossed chains and threw NullPointerException.
No main code set updateExisting before the Cloud Request Engine's
builder did.

The replacement is now made on the old item's list. A new test replaces
forty items in a cache with four lists, evicts past its size, and checks
that each list runs from its first item to its last through items that
belong to it. The Cloud Request Engine's builder javadoc now says the
builder is set, not copied, and a further test fills a small cache after
a replaced entry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant