Repository navigation
Detect Cloudflare block pages and pace the Best Matches feed - #21
Merged
Merged
Conversation
Cloudflare's block page is served at the requested URL, so _authenticated() treated it as a completed login: the scraper scrolled the block page and then reported a successful run with zero jobs. Detect the block page explicitly, fail fast with an actionable message, and check for it before and during scrolling. Open Best Matches directly instead of the login page on every run. With a persistent profile the login page only redirects to the dashboard, and a redirect followed by a second top-level navigation within a second is scored as automation. Accept the /nx/s/ session-scoped path so an authenticated session is not navigated again. Wait for feed tiles before sending input, jitter the scroll cadence, and stop once the feed stops moving. Read job URLs from the rendered job links with the opening UID as a fallback, and fail when no job links are found instead of reporting zero processed jobs. Log the HTTP status and Cloudflare headers of each document load so a refused run can be diagnosed from the response instead of inferred from page text.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the reported "You're blocked" behaviour, where the scraper navigated to Best Matches, treated Cloudflare's block page as a completed login, scrolled it and then reported a successful run with zero jobs.
auth.pyrecognises Cloudflare's block page (cf/"You have been blocked"/"Attention Required"), so_authenticated()no longer accepts it, andensure_not_blocked()raisesUpworkBlockedErrorwith an actionable message. Checks run before login, after navigating to Best Matches, and on every scroll step./nx/s/find-work/best-matchesis recognised as the session-scoped redirect target./jobs/...links, falling back to the opening UID; zero job links is now a hard failure instead of a silent success.cf-ray,cf-mitigatedandretry-afterof every document load, so a refused run can be diagnosed from the response rather than inferred.Related issue
No issue filed; reported in conversation with reproduction evidence.
Validation
make check.39 passed, Ruff clean,mypy upwork_scraperclean. Tests cover block-page detection, authentication state, login navigation order and fallback, response-header parsing, feed waiting, and both job URL extraction paths.Notes for reviewers
Blocking is decided by Cloudflare per request and varies between runs, so the change makes the failure explicit and diagnostic rather than attempting to defeat the scoring. The new log lines are the intended way to identify the trigger on a real blocked run.