Skip to content

grpc-js: Retry failed DNS resolutions until the channel is idle - #3133

Open
eisen0419 wants to merge 1 commit into
grpc:masterfrom
eisen0419:fix/dns-resolver-retry-on-failure
Open

eisen0419 wants to merge 1 commit into
grpc:masterfrom
eisen0419:fix/dns-resolver-retry-on-failure

Conversation

@eisen0419

Copy link
Copy Markdown

Fixes #3128

Summary

Retry failed DNS lookups on DnsResolver's existing BackoffTimeout while the channel remains active, and guard InternalChannel.getConnectivityState(true) against restarting resolution once a channel has entered SHUTDOWN.

 DnsResolver.startResolution -> lookup.then(..., err => ...)
   stopNextResolutionTimer()
-  this.listener(statusOrFromError(defaultResolutionError), ...)
+  healthStatus = this.listener(statusOrFromError(defaultResolutionError), ...)
+  onLookupFailure(healthStatus)
+    -> requestAnotherResolution() // reuse armed backoff timer or start next attempt once

 InternalChannel.getConnectivityState(tryToConnect)
-  if (tryToConnect)
+  if (tryToConnect && connectivityState !== ConnectivityState.SHUTDOWN)
     resolvingLoadBalancer.exitIdle()
  • PickFirstLoadBalancer accepts an error endpoint list by entering TRANSIENT_FAILURE and returning true, so onLookupFailure schedules a retry via requestAnotherResolution() even when healthStatus === true instead of calling handleHealthStatus(true) (which would stop and reset backoff).
  • Preserves #2098's per-attempt continueResolving = false reset so repeated updateResolution() calls share a single timer without multiplying lookups, stops retries on destroy() (both IDLE and close()), matches in-flight lookups by promise identity, and leaves unparseable dns: targets (dnsHostname === null) as one-shot errors.
  • Prevents synchronous SHUTDOWN connectivity watchers (such as Client.waitForReady) from calling exitIdle() during close() and restarting a destroyed resolver.

Evidence

  • Before: A channel created before its DNS name existed stopped resolving after initial lookup failures, leaving watchConnectivityState, Client.waitForReady, or a single queued wait-for-ready RPC stuck in TRANSIENT_FAILURE even after the DNS record appeared.
  • After: Failed DNS lookups retry with exponential backoff until the channel becomes READY, goes IDLE, or closes (41 passing tests across test-resolver, test-dns-resolver-retry, and client suites, including GRPC_NODE_USE_ALTERNATIVE_RESOLVER=true).

Merge Danger

Door: Two-way

Scoped to packages/grpc-js/src/resolver-dns.ts, packages/grpc-js/src/internal-channel.ts, and offline-stubbed DNS tests (ResolvingLoadBalancer is unchanged).

Blast Radius: Narrow

Intentionally updates the test expectation from #2105 so active channels retry failed DNS resolutions with bounded BackoffTimeout backoff (aligning with C-core's polling resolver and grpc-go's DNS watcher) while still halting at IDLE and SHUTDOWN.

A failed lookup reported the error and ignored the listener result, so
the backoff timer never scheduled another attempt. A channel created
before its DNS name existed stayed in TRANSIENT_FAILURE after the name
appeared.

Retry with the existing backoff, including when the load balancer
accepts the error by entering TRANSIENT_FAILURE. Repeated
updateResolution calls still share one timer. Retries stop when the
resolver is destroyed. getConnectivityState(true) does nothing after
SHUTDOWN, so a synchronous watcher such as Client.waitForReady cannot
restart a closed channel.

Fixes grpc#3128
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

grpc-js: DNS resolution stops retrying after failures, so a channel created before its name exists never connects

1 participant