Skip to content

Limit the cloud host lookup to the timeout and log why processing failed - #123

Merged
ben51degrees merged 8 commits into
rcfrom
feature/cloud-host-lookup-within-timeout
Nov 19, 2026
Merged

ben51degrees merged 8 commits into
rcfrom
feature/cloud-host-lookup-within-timeout

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

In full build 43382362370 (head b96922b1 of #91), the job MacOS_X64_Java_17 failed in the geo-location cloud value tests. The first test waited 30.63 seconds, then all three threw, and the log does not say why.

[ERROR] ValueTests_Cloud_AvailableProperties(fiftyone.geolocation.data.tests.FiftyoneValueTests)  Time elapsed: 30.63 s  <<< ERROR!
fiftyone.pipeline.exceptions.AggregateException: Exception(s) occurred processing evidence.
	at fiftyone.geolocation.data.tests.FiftyoneValueTests.ValueTests_Cloud_AvailableProperties(FiftyoneValueTests.java:47)
[ERROR] ValueTests_Cloud_ValueTypes(fiftyone.geolocation.data.tests.FiftyoneValueTests)  Time elapsed: 0.003 s  <<< ERROR!
[ERROR] ValueTests_Cloud_TypedGetters(fiftyone.geolocation.data.tests.FiftyoneValueTests)  Time elapsed: 0.019 s  <<< ERROR!

The Cloud Request Engine's timeout is 2 seconds by default (Constants.java line 41).

Which wait was 30 seconds, and whose

The engine applies its timeout as the connect and read timeouts of each HttpURLConnection (CloudRequestEngineDefault.java lines 830 to 835). Neither covers the lookup of the host name. The connection looks the host up before its connect timeout starts, and the lookup waits for as long as the operating system's resolver does.

A probe on a macos-15-intel runner (run 43435285991), with Java 17.0.20.1 on macOS 15.7.9 as in the failing job, sent lookups for 51degrees.com to an address that never answers and timed them.

What was timed Result
Lookup of cloud.51degrees.com, resolver answering 40 ms
Request with connect and read timeouts of 2000 ms, resolver not answering failed after 30,453 ms with UnknownHostException
The same request straight after failed after 0 ms
A lookup 11 seconds later failed after 30,005 ms

So the 30 seconds is the macOS resolver giving up on a DNS server that does not answer. The JVM then keeps the failed lookup for 10 seconds, which is why the second and third tests failed in 3 ms and 19 ms. That is the shape of the failure.

The test class takes these times when the cloud answers, over the 14 full build attempts between 11 and 13 November (13 runs, one of them run twice). Only some of them ran the macOS jobs.

Runner Passing runs Shortest Longest
Ubuntu_Java_8 14 1.36 s 1.86 s
Ubuntu_Java_11 14 1.37 s 2.27 s
Ubuntu_Java_17 14 1.42 s 2.10 s
Ubuntu_Java_21 14 1.46 s 3.32 s
Ubuntu_ARM_Java_17 14 1.48 s 1.99 s
Windows_Java_8 14 1.55 s 3.13 s
Windows_Java_11 14 1.75 s 2.98 s
Windows_Java_17 14 1.72 s 3.33 s
Windows_Java_21 14 1.60 s 2.99 s
MacOS_Java_11 5 1.69 s 2.81 s
MacOS_Java_17 5 2.04 s 2.46 s
MacOS_Java_21 5 1.11 s 2.56 s
MacOS_X64_Java_11 5 2.09 s 3.94 s
MacOS_X64_Java_17 4 2.93 s 3.98 s
MacOS_X64_Java_21 5 2.77 s 3.48 s

No runner is near 30 seconds, so no limit needs to change. The wait was not the test's or the engine's, and it was longer than the engine's timeout says a request can take.

Why the cause was not in the log

  1. PipelineDefault.process throws an AggregateException that holds the element's exception only as a suppressed exception (PipelineDefault.java lines 495 to 504, AggregateException.java line 48). Surefire 2.22.2 trims a stack trace to the test's own frames, so the cause is not printed.
  2. FlowDataDefault.addError logged Error occurred during processing of ... without the exception (FlowDataDefault.java line 124).
  3. The location packages took SLF4J 1.7.36 (location/pom.xml line 66) with Logback 1.3.16, which binds only through the SLF4J 2 API. Their tests logged nothing at all, as the failing job printed.
    SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
    SLF4J: Defaulting to no-operation (NOP) logger implementation
    
    That also dropped the engine's warning when it is built, which carries the exception (CloudRequestEngineDefault.java line 364).

What changed

  1. HttpClientDefault looks up the host within the connection's connect timeout before it connects, through the new HostLookup in pipeline.engines. A lookup that takes longer throws SocketTimeoutException: Looking up the host 'cloud.51degrees.com' took longer than the timeout of 2000 ms. A lookup that succeeds leaves the address in the JVM's cache, where the connection then finds it. Callers waiting on the same host share one lookup, so a resolver that does not answer holds one thread, not one for each request. Nothing is looked up when the connect timeout is 0, which means no limit, or when a proxy makes the request, because the proxy looks up the host. The Cloud Request Engine uses HttpClientDefault unless it is given another client, so its timeout now covers the lookup.

    Relying on the JVM's cache has two limits, which HostLookup's comment also says. With the security property networkaddress.cache.ttl set to 0 nothing is cached, so the connection looks the host up again itself, with no limit, as before this change. And every request is looked up first, including one the connection would have sent on a kept-alive socket with no lookup at all. So once the cached address expires, 30 seconds after it was found by default, a resolver that does not answer now fails such a request within the timeout, where before the open socket would have served it. Avoiding that would need either a thread for each request's whole connect, which a resolver that does not answer would hold for 30 seconds each, or a way to hand the connection an address, which HttpURLConnection on Java 8 does not have.

    The default ProxySelector decides whether a request goes through a proxy. A subclass of HttpClientDefault whose connect opens the connection through a proxy of its own still has the host looked up here, which HttpClientDefault's comment says.

  2. FlowDataDefault.addError logs the exception with the error.

  3. The location packages take SLF4J 2.0.20 in place of 1.7.36 (location/pom.xml line 66), as the other five package trees already do, so Logback 1.3.16 binds and their logs are written.

  4. A failed cloud request leaves no response. Engines that override processEngine read the response without the check that CloudAspectEngineBase.processEngine makes, so each added NullPointerException: Cannot invoke "String.length()" because "s" is null after the real error, as the fault run 43437302629 on this branch showed once logging worked and before a check was added. rc has since added the same check to DeviceDetectionCloudEngine and GeoLocationCloudEngine, with the warning written out in each, and moved IPIntelligenceCloudEngine onto the shared processing, which reads the response only when there is one. So at this head DeviceDetectionCloudEngine and GeoLocationCloudEngine log the shared warning Constants.Messages.NoCloudResponse in rc's check, the developer example's SimpleCloudEngine gains the check, and IPIntelligenceCloudEngine is rc's, unchanged. Each logs one warning and processes nothing.

  5. CloudAspectEngineBase.processEngine logs the same NoCloudResponse warning in place of its own copy of the text.

How it was verified

New tests:

  • HostLookupTests, 8 tests. A resolver that does not answer times out within the limit with the host in the message, an unknown host keeps its cause, a timeout of 0 looks nothing up, a lookup after a failure looks up again, two callers at once share one lookup, a request through a proxy looks nothing up, and both request methods of HttpClientDefault fail within the connect timeout.
  • FlowDataTests.FlowData_AddError_LogsTheException.
  • NoResponse_ProcessesNothing in GeoLocationCloudEngineNullValueTests and in the Device Detection and IP Intelligence cloud ProductMissingTests, each checking that one warning is logged for each response.

Run against rc with only the tests and the new message constant added (run 43518934300), FlowData_AddError_LogsTheException fails with Argument(s) are different!, because the error is logged without the exception, and the three NoResponse_ProcessesNothing tests fail with NullPointerException: Cannot invoke "String.length()" because "s" is null. On this head (run 43518954543) all four pass, with the 8 HostLookupTests.

Fault runs on ubuntu-22.04 build the packages, run the geo-location cloud value tests with the cloud answering, then send every name lookup to an address that never answers, with a resolver timeout of 30 seconds as on macOS, and run them again.

rc (run 43437315713) this change (run 43439842768)
Cloud answering passed in 1.44 s passed in 1.55 s
Not answering, first test 30.19 s 4.23 s
Not answering, second and third tests 0.013 s and 0.014 s 4.03 s and 4.03 s
Cause in the log none, the SLF4J no-operation logger SocketTimeoutException: Looking up the host 'cloud.51degrees.com' took longer than the timeout of 2000 ms. in the engine's warning and in the processing error

Each test now takes 4 seconds under the fault, being the 2 second limit on the lookup when the engine is built and again when it processes. Both fault runs show as cancelled, because once the runner could not look up names it could not report back, but every step had finished and the job pages show their results.

Changes at this head

The previous head, a282b03, no longer merged into rc, which had added its own "no response, process nothing" check to the Device Detection and geo-location cloud engines and moved the IP Intelligence cloud engine onto the shared processing. One commit follows it.

  • e9852a1a merges rc 49bc431 in, as Base branch says, keeping rc's checks and having them log this draft's shared NoCloudResponse warning. The IP Intelligence NoResponse_ProcessesNothing test now processes a Flow Data through the engine's public process method, so it tests the shared processing's check, which is where the IP Intelligence engine now meets a missing response. The commit adds no test, so it has no red. Its green is the tests of what the draft touches, below.

Tests of what it touches

Run 44452837805 built e9852a1, with one temporary workflow commit on top of it (f192f7a) and nothing merged, on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21, and every job passed. In its Ubuntu Java 17 job, FlowDataTests passed 42 of 42, HostLookupTests 8 of 8, HttpClientTests 2 of 2, CloudAspectEngineBaseTests 13 of 13, the Device Detection and IP Intelligence cloud ProductMissingTests 8 of 8 each, GeoLocationCloudEngineNullValueTests 6 of 6, and the geo-location cloud value tests, FiftyoneValueTests, 3 of 3 in 1.48 s. macOS was left out of this build. The rc head is built with macOS before it merges into main.

An earlier full build 43441956524 of the head dd8d7356 passed in 14 of its 15 build and test jobs across its two attempts, the Intel macOS Java 11 job among them with FiftyoneValueTests in 3.21 s and HostLookupTests in 1.03 s. MacOS_X64_Java_21 failed in both attempts in the Selenium contract test that #124 has since fixed in rc, which does not touch this change.

Released tests

The released tests on main that exercise what this draft touches are FlowDataTests (39 on main), the Device Detection and IP Intelligence cloud ProductMissingTests (5 each on main) and GeoLocationCloudEngineNullValueTests (5 on main). The draft adds tests to each and changes none. GeoLocationCloudEngineNullValueTests gains a buildEngine(ILoggerFactory) overload, and the old buildEngine() calls it with the same logger factory as before, so its released tests run as released. All of them passed in the run above.

Base branch

rc 49bc431 is merged into the head with a merge commit, e9852a1a. #175 on rc had added the same check to the Device Detection and geo-location cloud engines with the warning written out in each, and moved the IP Intelligence cloud engine's reading of the response into processCloudEngine, which the shared processing calls only when there is a response. Three conflicts, resolved as follows.

  • DeviceDetectionCloudEngine.java. Both sides' imports are kept. Both sides' checks would have run one after the other, so one is kept, in rc's place, logging NoCloudResponse.
  • GeoLocationCloudEngine.java. rc's check is kept, logging NoCloudResponse.
  • IPIntelligenceCloudEngine.java is rc's file unchanged. The shared processing's check, which this draft moves to NoCloudResponse, logs the warning for it.

rc has since moved to 3bb6e71, and the draft merges with it with no conflict.

Before you upgrade

  • HttpClientDefault, which the Cloud Request Engine uses unless it is given another client, now fails a request with SocketTimeoutException when looking up the host takes longer than the connect timeout, which is the engine's TimeoutSeconds, 2 seconds by default. Before, it waited as long as the system resolver did, 30 seconds on macOS with a DNS server that does not answer. The specification's options table, line 538 at d79bbe3b, gives TimeoutSeconds as "The timeout to use when making requests to the cloud service", and the lookup is part of the request. The cost is that once the JVM's cached address expires, 30 seconds after it was found by default, a request that would have gone on a kept-alive socket fails within the timeout if the resolver does not answer. A user whose resolver is slower than the timeout sets a longer TimeoutSeconds.
  • The location packages depend on SLF4J 2.0.20 in place of 1.7.36, as every other 51Degrees package already does. An application that pinned SLF4J 1.7 for them moves to 2.0.
  • The warning a cloud engine logs when it has no response now reads "The did not process, as the Cloud Request Engine has no response. The Cloud Request Engine's error in the log says why." Anything that matched the old text in a log matches the new one.
  • FlowDataDefault.addError logs the exception with the error, so the log carries its stack trace.

What remains

  • DataUploaderHttp, ShareUsageElement and DidClient open their own connections, so their lookups still wait as long as the resolver does. None of them is the Cloud Request Engine.
  • DataUpdateServiceDefault connects through HttpClient.connect and then calls getResponseCode itself (lines 821 and 838 at 30e48867), not through postData or getResponseString, so the lookups of data file downloads have no limit either.
  • Why the runner's DNS server stopped answering during that job is outside this repository.

The Cloud Request Engine sets its timeout, 2 seconds by default, as the
connect and read timeouts of each HttpURLConnection. Neither covers the
lookup of the host, which waits as long as the system resolver does, so a
resolver that stops answering held a request for 30 seconds.

HttpClientDefault now looks up the host within the connection's connect
timeout before connecting. Callers waiting on the same host share one
lookup, and a request through a proxy is not looked up.

The error FlowData logs for a processing failure now carries the
exception. The location packages took SLF4J 1.7.36 with Logback 1.3.16,
which binds only through the SLF4J 2 API, so their tests logged nothing.
They now take SLF4J 2.0.20, as the other packages do.
A failed cloud request leaves no response. GeoLocationCloudEngine read
it anyway, so each failure added a NullPointerException from the JSON
parser after the Cloud Request Engine's own error. It now logs a warning
and processes nothing, as CloudAspectEngineBase does.
GeoLocationCloudEngine, DeviceDetectionCloudEngine, IPIntelligenceCloudEngine
and the developer example's SimpleCloudEngine override processEngine and
read the Cloud Request Engine's response without the check that
CloudAspectEngineBase makes. A failed cloud request leaves no response, so
each failure added a NullPointerException from the JSON parser after the
Cloud Request Engine's own error. Each now logs one warning and processes
nothing. The message is shared as Constants.Messages.NoCloudResponse.
HostLookup relies on the JVM's address cache, so its comment now says the
two limits that follow. With networkaddress.cache.ttl at 0 the connection
looks the host up again with no limit, and a request the connection would
send on a kept-alive socket is looked up too, so it fails once the cached
address expires and the resolver does not answer. HttpClientDefault says
that a subclass connecting through a proxy of its own is still looked up.

CloudAspectEngineBase logs Constants.Messages.NoCloudResponse, as the four
engines that override processEngine do, and the geo-location test checks
the warning as the Device Detection and IP Intelligence tests do.
rc's section 15 conformance change added the same "no response, process
nothing" check to the Device Detection and geo-location cloud engines, with
the warning written inline, and moved the IP Intelligence cloud engine onto
the shared Cloud Aspect Engine processing, which reads the response only
when there is one. Three conflicts, resolved as follows.

- DeviceDetectionCloudEngine.java. Both sides' imports are kept. Both
  sides' checks merged in one after the other, so one is kept, in rc's
  place, logging this branch's shared NoCloudResponse warning.
- GeoLocationCloudEngine.java. rc's check is kept, logging this branch's
  NoCloudResponse warning.
- IPIntelligenceCloudEngine.java is rc's file unchanged. Its
  processCloudEngine is only called with a response, and the shared
  processing's own check, which this branch already moves to
  NoCloudResponse, logs the warning.

This branch's new IP Intelligence test NoResponse_ProcessesNothing took
rc's test helper in the merge, which calls processCloudEngine directly and
so passed it the null response the shared processing never would. It now
processes a Flow Data through the engine's public process method, so the
shared processing's check is what it tests.

Run locally after the merge, every one passing: FlowDataTests 42,
HostLookupTests 8, CloudAspectEngineBaseTests 13, the Device Detection and
IP Intelligence ProductMissingTests 8 each, and
GeoLocationCloudEngineNullValueTests 6.
@ben51degrees
ben51degrees marked this pull request as ready for review November 19, 2026 17:19
@ben51degrees
ben51degrees merged commit 1364548 into rc Nov 19, 2026
4 checks passed
@ben51degrees
ben51degrees deleted the feature/cloud-host-lookup-within-timeout branch November 19, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants