Skip to content

info provider for sparkfun.com - #1579

Merged
jbtronics merged 7 commits into
Part-DB:masterfrom
samyk:provider-sparkfun
Oct 5, 2026
Merged

jbtronics merged 7 commits into
Part-DB:masterfrom
samyk:provider-sparkfun

Conversation

@samyk

@samyk samyk commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Adds an info provider for the SparkFun shop. It needs no key and is off until PROVIDER_SPARKFUN_ENABLED=1 is set.

sparkfun.com is a Magento 2 shop with an open GraphQL endpoint, which the provider uses for everything it exposes; the documents and the "Features & Specs" list are only on the product page, so details read that too. SparkFun offers no official API for this, so it could break at any time.

What it does:

  • Search: one GraphQL request, up to 20 results, with a product whose SKU matches the keyword exactly sorted first (the shop's search is fuzzy and can list a neighbouring SKU first).
  • Details: one GraphQL request plus one product page request. The ID can be a SKU (DEV-13975, matched exactly), a bare number (13975, accepted only if a SKU ends in that number), or a URL key, so both /products/13975 and /<slug>.html URLs resolve.
  • Returns name, short description, the description HTML plus the feature list as notes, category path, product URL, the main image and gallery (videos dropped), USD prices with quantity tiers (without tax), mass, and the documents linked on the product page (schematics, datasheets, hookup guides, Eagle files, GitHub). If the page fails to load, the documents linked in the description are still returned.
  • Manufacturer is "SparkFun"; the SKU is the provider ID and the MPN.

Politeness:

  • A requestDelay setting (PROVIDER_SPARKFUN_REQUEST_DELAY, 0-60 s, default 10) is the minimum time between any two requests (GraphQL and product pages). A request that comes too early waits; the next allowed time is kept in the cache so it holds across processes.
  • After an HTTP 403, 429 or 503 the provider sends nothing for one hour and lookups fail with a message saying until when it is paused.

Includes unit tests with mocked HTTP (22 tests), the settings class, translation strings and a docs section.

Notes for review:

  • Verified against the live site for DEV-13975 and BOB-12009.
  • Retired products are no longer in the shop's catalog, so they are not found.
  • The manufacturer is always "SparkFun", which is wrong for resold third-party products; I found no usable brand field.
  • The shop's weight field has no unit; the provider assumes pounds, which gives plausible masses for the products checked.
  • Parameters are usually empty: only feature bullets of the form "Name: value" become parameters, the rest of the feature list goes into the notes.
  • With the 10 s default a details lookup takes 10-20 s; say if you would prefer a lower default.

samyk added 4 commits October 3, 2026 14:16
SparkFun sells many of its own boards that no distributor API knows by
their SKU, so parts bought there could only be filled in by hand.

The shop is a Magento store with a public GraphQL endpoint, which gives
the name, descriptions, categories, images, weight and prices including
quantity discounts as structured data. The documents (schematics, hookup
guides, eagle files, datasheets) and the feature list are only shown in
tabs of the product page, so that page is read as well; if it can not be
loaded, the documents linked in the description are still returned.

Products can be searched by keyword or SKU. The details accept a SKU
(DEV-13975), the bare number (13975, only an exact SKU match is taken as
the shop search is fuzzy) or the URL key, so product URLs can be used
too. The provider is enabled with PROVIDER_SPARKFUN_ENABLED or in the
settings.
The provider reads the GraphQL endpoint and the product pages of the shop,
which is no official API, so a few lookups in a row could look like a
crawler and get the address blocked.

All requests to sparkfun.com now go through one method, which keeps a
configurable delay (PROVIDER_SPARKFUN_REQUEST_DELAY, default 10 seconds) to
the previous request and sleeps if a request comes too early. The time of
the next allowed request is kept in the cache (which the provider needs now),
so the delay also holds between lookups and PHP processes.

If the shop answers with HTTP 403, 429 or 503, the provider remembers that
in the cache and sends no requests at all for one hour. Lookups fail with a
message telling until when and why it is paused. A refused product page
still does not fail the lookup, as the page is optional, but it starts the
pause too.
phpstan (arrayFilter.strict) rejects array_filter without a callback; only
arrays are product records anyway.
@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.77912% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.67%. Comparing base (a7f2d49) to head (d7ffcc4).

Files with missing lines Patch % Lines
.../InfoProviderSystem/Providers/SparkFunProvider.php 94.77% 13 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #1579      +/-   ##
============================================
+ Coverage     64.46%   64.67%   +0.21%     
- Complexity    10210    10295      +85     
============================================
  Files           762      763       +1     
  Lines         32711    32960     +249     
============================================
+ Hits          21087    21317     +230     
- Misses        11624    11643      +19     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jbtronics
jbtronics merged commit 61044e6 into Part-DB:master Oct 5, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants