Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ repos:
entry: bash scripts/test-acq-bats
language: system
pass_filenames: false
files: '^(acq|acq\.backends/.*|scripts/(test-acq-lib\.sh|test-acq-bats|run-shellcheck|probe-msb-node-startup|test-probe-msb-node-startup\.py|fix-exec-bits|verify-backends)|test/bats/.*)$'
files: '^(acq|acq\.backends/.*|scripts/(test-acq-lib\.sh|test-acq-bats|bats-shard|run-shellcheck|probe-msb-node-startup|test-probe-msb-node-startup\.py|fix-exec-bits|verify-backends)|test/bats/.*)$'

- id: test-probe-msb-startup
name: MSB startup probe offline tests
Expand Down
8 changes: 8 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ This project operates under professional standards of conduct. All contributors:
| `install.sh` | One-line `curl \| sh` installer (auto-selects brew/npm/clone; direct `sh install.sh` still installs the default release tag from the repo, not the current checkout) |
| `scripts/rotate-apikey` | Rotate your USAi API key secret (`acq usai-rotate-api-key`) |
| `scripts/test-acq-bats` | Offline unit suite for acq (bats-core) |
| `scripts/bats-shard` | Picks the `test/bats` files for shard `I/N` (CI matrix; `ACQ_BATS_SHARD=I/N`) |
| `scripts/verify-backends` | Live end-to-end backend verification (needs Docker or KVM) |
| `.pre-commit-config.yaml` | Optional pre-commit hooks (secret detection, file hygiene) |
| `AGENTS.md` | Rules for working **on this quickstart repo** |
Expand Down Expand Up @@ -199,6 +200,13 @@ network), built on [bats-core](https://github.com/bats-core/bats-core) (ADR-0025
./scripts/test-acq-bats --timing test/bats/90-sbx-startup-kit.bats
```

To run only part of the suite (this is how the bash 3.2 CI job splits the work
across a matrix), set `ACQ_BATS_SHARD=I/N`, e.g. `ACQ_BATS_SHARD=2/3
./scripts/test-acq-bats`. `scripts/bats-shard` derives each shard from the
`.bats` files on disk (balanced by `@test` count; no checked-in table), so a new
test file joins a shard automatically, and an invalid spec or empty shard
fails closed with exit 2.

Timing mode reports the worker count and Bats' per-test elapsed time, including
setup and teardown. Explicit file/flag arguments still pass directly to Bats;
they do not implicitly enable parallelism. CI enables timing and verbose
Expand Down
93 changes: 93 additions & 0 deletions scripts/bats-shard
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
#!/usr/bin/env bash
#
# bats-shard — print the .bats files that belong to shard I of N.
#
# Used by scripts/test-acq-bats (ACQ_BATS_SHARD=I/N) so a CI matrix can split
# the offline suite across runners. Selection is derived from the files on disk
# every time — there is no checked-in table that can go stale: a new .bats file
# lands in a shard automatically.
#
# Algorithm (deterministic, balanced): weight each file by its number of @test
# blocks (minimum 1; a good, free proxy for wall time), sort by weight
# descending then path ascending (C locale), and greedily assign each file to
# the currently lightest shard (lowest index wins ties). Every file is assigned
# to exactly one shard. Adding or removing a file may move other files between
# shards, which is harmless: every run re-derives the full partition.
#
# Fails closed (exit 2) on a malformed spec, I > N, a missing directory, no
# .bats files, or an empty shard — an empty shard would otherwise pass vacuously.
#
# Usage:
# scripts/bats-shard I/N [DIR] # DIR defaults to test/bats; one path per line
#
# Portable to bash 3.2 / macOS (no mapfile, no associative arrays, POSIX awk).
set -uo pipefail

_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
_spec="${1:-}"
_bats_dir="${2:-${_dir}/../test/bats}"

case "$_spec" in
[1-9]*/[1-9]*) ;;
*) echo "bats-shard: spec must be I/N with positive integers (got '${_spec}')" >&2; exit 2 ;;
esac
_i="${_spec%%/*}"
_n="${_spec#*/}"
case "${_i}${_n}" in
*[!0-9]*) echo "bats-shard: spec must be I/N with positive integers (got '${_spec}')" >&2; exit 2 ;;
esac
if [ "${#_i}" -gt 4 ] || [ "${#_n}" -gt 4 ] || [ "$_i" -gt "$_n" ]; then
echo "bats-shard: need 1 <= I <= N <= 9999 (got '${_spec}')" >&2
exit 2
fi

if [ ! -d "$_bats_dir" ]; then
echo "bats-shard: not a directory: ${_bats_dir}" >&2
exit 2
fi
_bats_dir="$(cd "$_bats_dir" && pwd)"

export LC_ALL=C
_files=()
for _f in "${_bats_dir}"/*.bats; do
[ -f "$_f" ] && _files+=("$_f")
done
if [ "${#_files[@]}" -eq 0 ]; then
echo "bats-shard: no .bats files in ${_bats_dir}" >&2
exit 2
fi

_picked="$(awk -v shard="$_i" -v total="$_n" '
/^[ \t]*@test[ \t]/ { c[FILENAME]++ }
END {
nf = ARGC - 1
for (k = 1; k <= nf; k++) {
name[k] = ARGV[k] ""
w[k] = (c[ARGV[k]] > 0) ? c[ARGV[k]] : 1
ord[k] = k
}
for (a = 2; a <= nf; a++) {
cur = ord[a]
b = a - 1
while (b >= 1 && (w[ord[b]] < w[cur] || (w[ord[b]] == w[cur] && name[ord[b]] > name[cur]))) {
ord[b + 1] = ord[b]
b--
}
ord[b + 1] = cur
}
for (s = 1; s <= total; s++) load[s] = 0
for (a = 1; a <= nf; a++) {
k = ord[a]
best = 1
for (s = 2; s <= total; s++) if (load[s] < load[best]) best = s
load[best] += w[k]
if (best == shard) print name[k]
}
}
' "${_files[@]}")" || { echo "bats-shard: failed to read test files" >&2; exit 2; }

if [ -z "$_picked" ]; then
echo "bats-shard: shard ${_i}/${_n} is empty (only ${#_files[@]} .bats files); refusing to pass vacuously" >&2
exit 2
fi
printf '%s\n' "$_picked" | sort
39 changes: 34 additions & 5 deletions scripts/test-acq-bats
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,13 @@
# ./scripts/test-acq-bats test/bats/X.bats # a specific file (passed through)
# ACQ_BATS_JOBS=1 ./scripts/test-acq-bats # force serial
# ACQ_BATS_TIMING=1 ./scripts/test-acq-bats # per-test timings, also in fallback
# ACQ_BATS_SHARD=1/2 ./scripts/test-acq-bats # only shard 1 of 2 (see below)
#
# SHARDING: ACQ_BATS_SHARD=I/N runs only the files scripts/bats-shard assigns to
# shard I of N, so CI can split the suite across a job matrix. Selection is
# recomputed from test/bats/*.bats on every run (no checked-in table), so a new
# file joins a shard automatically; an invalid spec or empty shard fails closed
# (exit 2). Ignored when explicit file arguments are given.
set -uo pipefail

_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Expand Down Expand Up @@ -69,7 +76,27 @@ if [ "$#" -gt 0 ]; then
exec "$BATS" ${_timing[@]+"${_timing[@]}"} "$@"
fi

# Whole-suite run: opt into across-files parallelism when we can.
# Whole-suite run (optionally one shard): the targets are the whole directory,
# or the explicit file list scripts/bats-shard selects. A bad shard spec must
# stop the run here rather than silently running nothing or everything.
_targets=("${REPO_ROOT}/test/bats")
_shard_files=()
if [ -n "${ACQ_BATS_SHARD:-}" ]; then
_shard_out="$(bash "${_dir}/bats-shard" "$ACQ_BATS_SHARD")" || {
echo "test-acq-bats: invalid ACQ_BATS_SHARD='${ACQ_BATS_SHARD}'" >&2
exit 2
}
while IFS= read -r _f; do
_shard_files+=("$_f")
done <<EOF_SHARD
${_shard_out}
EOF_SHARD
_targets=("${_shard_files[@]}")
echo "test-acq-bats: shard ${ACQ_BATS_SHARD}: ${#_shard_files[@]} file(s)" >&2
printf 'test-acq-bats: %s\n' "${_shard_files[@]##*/}" >&2
fi

# Opt into across-files parallelism when we can.
# - ACQ_BATS_JOBS overrides the job count; =1 forces serial.
# - bats --jobs needs GNU parallel or shenwei356/rush on PATH; without one it
# errors out, so we only pass --jobs when a parallelizer is present.
Expand All @@ -85,13 +112,15 @@ if [ "${ACQ_BATS_TIMING:-0}" = 1 ]; then
echo "test-acq-bats: workers=$_jobs; per-test timings include setup and teardown" >&2
fi
if [ "$_jobs" -gt 1 ] 2>/dev/null && { command -v parallel >/dev/null 2>&1 || command -v rush >/dev/null 2>&1; }; then
exec "$BATS" ${_timing[@]+"${_timing[@]}"} --jobs "$_jobs" --no-parallelize-within-files "${REPO_ROOT}/test/bats"
exec "$BATS" ${_timing[@]+"${_timing[@]}"} --jobs "$_jobs" --no-parallelize-within-files "${_targets[@]}"
fi

if [ "$_jobs" -gt 1 ] 2>/dev/null && command -v xargs >/dev/null 2>&1 && printf 'x\0' | xargs -0 -n1 -P 1 true 2>/dev/null; then
_tmp="$(mktemp -d "${TMPDIR:-/tmp}/acq-bats.XXXXXX")"
trap 'rm -rf "$_tmp"' EXIT HUP INT TERM
printf '%s\0' "${REPO_ROOT}"/test/bats/*.bats |
_xargs_files=("${REPO_ROOT}"/test/bats/*.bats)
if [ "${#_shard_files[@]}" -gt 0 ]; then _xargs_files=("${_shard_files[@]}"); fi
printf '%s\0' "${_xargs_files[@]}" |
xargs -0 -n1 -P "$_jobs" sh -c '
_bats=$1 _outdir=$2 _file=$3
_name=$(basename "$_file")
Expand All @@ -105,7 +134,7 @@ if [ "$_jobs" -gt 1 ] 2>/dev/null && command -v xargs >/dev/null 2>&1 && printf
' sh "$BATS" "$_tmp"

_rc=0
for _file in "${REPO_ROOT}"/test/bats/*.bats; do
for _file in "${_xargs_files[@]}"; do
_out="${_tmp}/$(basename "$_file").out"
[ -f "$_out" ] && cat "$_out"
_status="${_out}.status"
Expand All @@ -116,4 +145,4 @@ if [ "$_jobs" -gt 1 ] 2>/dev/null && command -v xargs >/dev/null 2>&1 && printf
exit "$_rc"
fi

exec "$BATS" ${_timing[@]+"${_timing[@]}"} "${REPO_ROOT}/test/bats"
exec "$BATS" ${_timing[@]+"${_timing[@]}"} "${_targets[@]}"
53 changes: 53 additions & 0 deletions test/bats/01-test-runner.bats
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ setup() {
mkdir -p "$RUNNER_ROOT/scripts" "$RUNNER_ROOT/test/vendor/bats-core/bin" "$RUNNER_ROOT/test/bats"
cp "$REPO_ROOT/scripts/test-acq-bats" "$RUNNER_ROOT/scripts/test-acq-bats"
cp "$REPO_ROOT/scripts/run-shellcheck" "$RUNNER_ROOT/scripts/run-shellcheck"
cp "$REPO_ROOT/scripts/bats-shard" "$RUNNER_ROOT/scripts/bats-shard"
cat > "$RUNNER_ROOT/test/vendor/bats-core/bin/bats" <<'STUB'
#!/bin/sh
printf 'bats-arg:%s\n' "$@"
Expand Down Expand Up @@ -80,6 +81,58 @@ STUB
assert_line 'bats-arg:--timing'
}

@test "runner: ACQ_BATS_SHARD runs only the selected files (serial path)" {
printf '@test "x" {\n true\n}\n' > "$RUNNER_ROOT/test/bats/a.bats"
printf '@test "y" {\n true\n}\n@test "z" {\n true\n}\n' > "$RUNNER_ROOT/test/bats/b.bats"
run env ACQ_BATS_JOBS=1 ACQ_BATS_SHARD=1/2 bash "$RUNNER_ROOT/scripts/test-acq-bats"
assert_success
assert_line "bats-arg:$RUNNER_ROOT/test/bats/b.bats"
refute_line "bats-arg:$RUNNER_ROOT/test/bats/a.bats"
refute_line "bats-arg:$RUNNER_ROOT/test/bats"
run env ACQ_BATS_JOBS=1 ACQ_BATS_SHARD=2/2 bash "$RUNNER_ROOT/scripts/test-acq-bats"
assert_success
assert_line "bats-arg:$RUNNER_ROOT/test/bats/a.bats"
refute_line "bats-arg:$RUNNER_ROOT/test/bats/b.bats"
}

@test "runner: ACQ_BATS_SHARD xargs fallback runs only the selected files" {
printf '@test "x" {\n true\n}\n' > "$RUNNER_ROOT/test/bats/a.bats"
printf '@test "y" {\n true\n}\n@test "z" {\n true\n}\n' > "$RUNNER_ROOT/test/bats/b.bats"
run env ACQ_BATS_JOBS=2 ACQ_BATS_SHARD=1/2 bash -c '
command() {
case "$*" in
"-v parallel"|"-v rush") return 1 ;;
*) builtin command "$@" ;;
esac
}
export -f command
exec bash "$1/scripts/test-acq-bats"
' _ "$RUNNER_ROOT"
assert_success
assert_line "bats-arg:$RUNNER_ROOT/test/bats/b.bats"
refute_line "bats-arg:$RUNNER_ROOT/test/bats/a.bats"
}

@test "runner: invalid ACQ_BATS_SHARD fails closed before running any test" {
local spec
for spec in 0/2 3/2 x 1/2/3 9/9; do
run env ACQ_BATS_JOBS=1 ACQ_BATS_SHARD="$spec" bash "$RUNNER_ROOT/scripts/test-acq-bats"
assert_failure 2
refute_output --partial 'bats-arg:'
done
}

@test "runner: explicit file arguments ignore ACQ_BATS_SHARD" {
run env ACQ_BATS_SHARD=1/2 bash "$RUNNER_ROOT/scripts/test-acq-bats" 'only.bats'
assert_success
assert_line 'bats-arg:only.bats'
}

@test "runner: an ACQ_BATS_SHARD exported by the CI leg does not leak into runner tests" {
# acq_setup_stubs (this test's setup) must have cleared it.
[ -z "${ACQ_BATS_SHARD:-}" ]
}

@test "shellcheck runner: validates workers and preserves a failing batch" {
cat > "$STUBDIR/shellcheck" <<'STUB'
#!/bin/sh
Expand Down
Loading
Loading