Repository navigation
info provider for sparkfun.com - #1579
Merged
Merged
Conversation
SparkFun sells many of its own boards that no distributor API knows by their SKU, so parts bought there could only be filled in by hand. The shop is a Magento store with a public GraphQL endpoint, which gives the name, descriptions, categories, images, weight and prices including quantity discounts as structured data. The documents (schematics, hookup guides, eagle files, datasheets) and the feature list are only shown in tabs of the product page, so that page is read as well; if it can not be loaded, the documents linked in the description are still returned. Products can be searched by keyword or SKU. The details accept a SKU (DEV-13975), the bare number (13975, only an exact SKU match is taken as the shop search is fuzzy) or the URL key, so product URLs can be used too. The provider is enabled with PROVIDER_SPARKFUN_ENABLED or in the settings.
The provider reads the GraphQL endpoint and the product pages of the shop, which is no official API, so a few lookups in a row could look like a crawler and get the address blocked. All requests to sparkfun.com now go through one method, which keeps a configurable delay (PROVIDER_SPARKFUN_REQUEST_DELAY, default 10 seconds) to the previous request and sleeps if a request comes too early. The time of the next allowed request is kept in the cache (which the provider needs now), so the delay also holds between lookups and PHP processes. If the shop answers with HTTP 403, 429 or 503, the provider remembers that in the cache and sends no requests at all for one hour. Lookups fail with a message telling until when and why it is paused. A refused product page still does not fail the lookup, as the page is optional, but it starts the pause too.
phpstan (arrayFilter.strict) rejects array_filter without a callback; only arrays are product records anyway.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1579 +/- ##
============================================
+ Coverage 64.46% 64.67% +0.21%
- Complexity 10210 10295 +85
============================================
Files 762 763 +1
Lines 32711 32960 +249
============================================
+ Hits 21087 21317 +230
- Misses 11624 11643 +19 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an info provider for the SparkFun shop. It needs no key and is off until
PROVIDER_SPARKFUN_ENABLED=1is set.sparkfun.com is a Magento 2 shop with an open GraphQL endpoint, which the provider uses for everything it exposes; the documents and the "Features & Specs" list are only on the product page, so details read that too. SparkFun offers no official API for this, so it could break at any time.
What it does:
DEV-13975, matched exactly), a bare number (13975, accepted only if a SKU ends in that number), or a URL key, so both/products/13975and/<slug>.htmlURLs resolve.Politeness:
requestDelaysetting (PROVIDER_SPARKFUN_REQUEST_DELAY, 0-60 s, default 10) is the minimum time between any two requests (GraphQL and product pages). A request that comes too early waits; the next allowed time is kept in the cache so it holds across processes.Includes unit tests with mocked HTTP (22 tests), the settings class, translation strings and a docs section.
Notes for review: