Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
22ac3af
[#154] Added Google Drive database fetch and push scripts and the 'vo…
AlexSkrypnyk Nov 10, 2026
50a1c56
[#154] Fixed mock expectations in the Google Drive and refresh tests.
AlexSkrypnyk Nov 10, 2026
7fe9d34
[#154] Documented pushing the sanitized database to a Google Drive or…
AlexSkrypnyk Nov 10, 2026
64fd757
Updated snapshots.
AlexSkrypnyk Nov 10, 2026
86d1b97
[#154] Fetched the dump from the upstream source in 'vortex-fetch-db'…
AlexSkrypnyk Nov 10, 2026
facf98e
[#154] Narrowed the fetch-db upstream source failure assertion.
AlexSkrypnyk Nov 10, 2026
08e24e3
[#154] Regenerated the variables reference.
AlexSkrypnyk Nov 10, 2026
d10d3de
Updated snapshots.
AlexSkrypnyk Nov 10, 2026
d0e8c9f
Addressed code review: defaulted the Google Drive and S3 push file na…
AlexSkrypnyk Nov 11, 2026
6234c31
Addressed code review: regenerated the variables reference.
AlexSkrypnyk Nov 11, 2026
8c6abc1
Replaced the Google Drive push with an FTP push that keeps the curren…
AlexSkrypnyk Nov 11, 2026
61f8c8a
Renamed the S3 push test loop variable that clashed with the BATS sta…
AlexSkrypnyk Nov 11, 2026
f97a387
Wired the FTP and S3 dump store push into CI and the installer, and d…
AlexSkrypnyk Nov 11, 2026
102eeac
Updated snapshots.
AlexSkrypnyk Nov 11, 2026
fc70a27
Pushed the latest dump under the remote file name with an optional su…
AlexSkrypnyk Nov 11, 2026
2ea03ee
Updated the database storage docs and the generated CI configs for th…
AlexSkrypnyk Nov 11, 2026
d99cece
Updated snapshots.
AlexSkrypnyk Nov 11, 2026
60571a5
Addressed code review: sent an empty length on the S3 copy request an…
AlexSkrypnyk Nov 11, 2026
8f233b5
Addressed code review: documented setting separate push and fetch fil…
AlexSkrypnyk Nov 11, 2026
beba57c
Addressed code review: documented renaming the dump with 'VORTEX_DB_F…
AlexSkrypnyk Nov 11, 2026
7cfda42
Reworded the suffixed S3 fetch note to pass the docs spellcheck.
AlexSkrypnyk Nov 11, 2026
a6fa78d
Added a playground script to fetch the database dump from FTP.
AlexSkrypnyk Nov 11, 2026
93b713c
Added the 'VORTEX_FETCH_DB_FTP_DIR' remote directory for the FTP fetc…
AlexSkrypnyk Nov 11, 2026
93b64a3
Added 'VORTEX_PUSH_DB_DESTINATIONS' to push the database dump to seve…
AlexSkrypnyk Nov 11, 2026
148bc9f
Documented the FTP directory and pushing to several destinations.
AlexSkrypnyk Nov 11, 2026
11ac925
Updated snapshots.
AlexSkrypnyk Nov 11, 2026
4cf30a2
Reworded the timestamped copy note in '.env' so the configured timezo…
AlexSkrypnyk Nov 11, 2026
305f54d
Updated snapshots.
AlexSkrypnyk Nov 11, 2026
2de4355
Pushed Vortex's own demo databases to the destinations in the 'VORTEX…
AlexSkrypnyk Nov 15, 2026
0f30d1b
Used 'filename' in the database storage docs to pass the prose check.
AlexSkrypnyk Nov 15, 2026
06af396
Updated snapshots.
AlexSkrypnyk Nov 15, 2026
4587ad8
Re-triggered CI to clear a flaky check.
AlexSkrypnyk Nov 15, 2026
5e5e42f
[#3340] Added 'VORTEX_FETCH_DB_STORE' to fetch the database from a sa…
AlexSkrypnyk Nov 16, 2026
3daeddb
[#3340] Documented fetching from a sanitized store, refreshing it on …
AlexSkrypnyk Nov 16, 2026
e524c57
[#3340] Added the 'Database store' and 'Migration database store' ins…
AlexSkrypnyk Nov 16, 2026
7f51fd4
Updated snapshots.
AlexSkrypnyk Nov 16, 2026
2059378
Updated snapshots.
AlexSkrypnyk Nov 16, 2026
1cc0ae6
[#3340] Split the deploy job condition over 2 lines to stay within th…
AlexSkrypnyk Nov 16, 2026
8c38365
Updated snapshots.
AlexSkrypnyk Nov 16, 2026
dbc3fa1
[#3340] Re-recorded the installer video with the database store prompt.
AlexSkrypnyk Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,15 @@
#; and will be removed during installation or update of Vortex.
version: '2.1'

#;< !PROVISION_TYPE_PROFILE
parameters:
# Refresh the database as on the nightly run, without building or deploying.
# Set it when triggering a pipeline from the CircleCI UI or API.
refresh_db:
type: boolean
default: false
#;> !PROVISION_TYPE_PROFILE

aliases:
#;< !PROVISION_TYPE_PROFILE
# SSH key fingerprint to fetch the database.
Expand Down Expand Up @@ -363,6 +372,27 @@ jobs:
no_output_timeout: 30m
#;> DB_FETCH_SOURCE_CONTAINER_REGISTRY

# Push the freshly-exported sanitized database dump to the push
# destinations. No-op unless a fresh DB was fetched (the fetch semaphore) and
# the push was requested (VORTEX_PUSH_DB_PROCEED, in the script).
- run:
name: Push DB
command: |
[ ! -f /tmp/fetch-db-fresh ] && echo "==> No freshly fetched database. DB push will not proceed." && exit 0
./vendor/bin/vortex-push-db
no_output_timeout: 30m

#;< MIGRATION
# Push the freshly-fetched migration database dump to its push destinations,
# next to the primary dump. No-op unless the push was requested.
- run:
name: Push migration DB
command: |
[ ! -f /tmp/fetch-db-fresh ] && echo "==> No freshly fetched database. DB push will not proceed." && exit 0
VORTEX_DB_INDEX=2 ./vendor/bin/vortex-push-db
no_output_timeout: 30m
#;> MIGRATION

- save_cache:
# Save cache per default branch and the timestamp.
# The cache will not be saved if it already exists.
Expand All @@ -387,6 +417,17 @@ jobs:
# Deploy container image.
VORTEX_EXPORT_DB_CONTAINER_REGISTRY_PUSH_PROCEED: 1
#;> DB_IN_IMAGE
# Push the sanitized database dump to the database store and destinations.
VORTEX_PUSH_DB_PROCEED: 1
#;< VORTEX_DEV
# Tell this provider's demo database dumps apart on the test FTP server
# by directory, with a timestamped copy of each push. The destinations
# and the FTP server come from the VORTEX_PUSH_DB_DESTINATIONS,
# VORTEX_PUSH_DB2_DESTINATIONS and VORTEX_PUSH_DB_FTP_* project
# variables.
VORTEX_PUSH_DB_FTP_DIR: circleci
VORTEX_PUSH_DB_TIMESTAMPED_COPY: 1
#;> VORTEX_DEV
# Do not build the Drupal front-end.
VORTEX_FRONTEND_BUILD_SKIP: 1
#;> !PROVISION_TYPE_PROFILE
Expand Down Expand Up @@ -759,6 +800,10 @@ jobs:
workflows:
# Commit workflow. Runs for every commit push to the remote repository.
commit:
#;< !PROVISION_TYPE_PROFILE
when:
not: << pipeline.parameters.refresh_db >>
#;> !PROVISION_TYPE_PROFILE
jobs:
#;< !PROVISION_TYPE_PROFILE
- database:
Expand Down Expand Up @@ -859,4 +904,11 @@ workflows:
- develop
jobs:
- database-nightly

# Database refresh workflow runs the nightly database job on demand, when a
# pipeline is triggered with the "refresh_db" parameter.
refresh-db:
when: << pipeline.parameters.refresh_db >>
jobs:
- database-nightly
#;> !PROVISION_TYPE_PROFILE
21 changes: 21 additions & 0 deletions .circleci/vortex-test-common.yml
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,27 @@ jobs:
no_output_timeout: 30m
#;> DB_FETCH_SOURCE_CONTAINER_REGISTRY

# Push the freshly-exported sanitized database dump to the push
# destinations. No-op unless a fresh DB was fetched (the fetch semaphore) and
# the push was requested (VORTEX_PUSH_DB_PROCEED, in the script).
- run:
name: Push DB
command: |
[ ! -f /tmp/fetch-db-fresh ] && echo "==> No freshly fetched database. DB push will not proceed." && exit 0
./vendor/bin/vortex-push-db
no_output_timeout: 30m

#;< MIGRATION
# Push the freshly-fetched migration database dump to its push destinations,
# next to the primary dump. No-op unless the push was requested.
- run:
name: Push migration DB
command: |
[ ! -f /tmp/fetch-db-fresh ] && echo "==> No freshly fetched database. DB push will not proceed." && exit 0
VORTEX_DB_INDEX=2 ./vendor/bin/vortex-push-db
no_output_timeout: 30m
#;> MIGRATION

- save_cache:
# Save cache per default branch and the timestamp.
# The cache will not be saved if it already exists.
Expand Down
40 changes: 40 additions & 0 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,11 @@ VORTEX_FETCH_DB_FTP_HOST=
# Database dump FTP port.
VORTEX_FETCH_DB_FTP_PORT=21

# Database dump FTP directory.
#
# Leave empty for the root directory of the FTP account.
VORTEX_FETCH_DB_FTP_DIR=

# Database dump FTP file name.
VORTEX_FETCH_DB_FTP_FILE=db.sql
#;> DB_FETCH_SOURCE_FTP
Expand All @@ -258,6 +263,24 @@ VORTEX_FETCH_DB_S3_BUCKET=
VORTEX_FETCH_DB_S3_REGION=ap-southeast-2
#;> DB_FETCH_SOURCE_S3

# Database store of the sanitized database dump file: ftp or s3.
#
# When set, the database is fetched from the store instead of the source. The
# nightly CI job fetches from the source, sanitizes the database and pushes
# the dump to the store. The store reads the FTP or S3 variables above. Leave
# empty to fetch from the source.
VORTEX_FETCH_DB_STORE=

# Destinations the nightly CI job pushes the sanitized database dump to
# besides the database store: ftp, s3 or both, such as "ftp,s3". Each
# destination reads its own variables, such as VORTEX_PUSH_DB_FTP_HOST for ftp.
# The database is always fetched from 1 place.
# VORTEX_PUSH_DB_DESTINATIONS=ftp,s3

# Also push a copy of each dump named with the time of the push, such as
# db_20261111-180000.sql, to keep a history of the pushed dumps.
# VORTEX_PUSH_DB_TIMESTAMPED_COPY=1

# Environment to fetch the database from.
#
# Applies to hosting environments.
Expand Down Expand Up @@ -319,6 +342,11 @@ VORTEX_FETCH_DB2_FTP_HOST=
# Second database dump FTP port.
VORTEX_FETCH_DB2_FTP_PORT=21

# Second database dump FTP directory.
#
# Leave empty for the root directory of the FTP account.
VORTEX_FETCH_DB2_FTP_DIR=

# Second database dump FTP file name.
VORTEX_FETCH_DB2_FTP_FILE=db2.sql
#;> MIGRATION_DB_FETCH_SOURCE_FTP
Expand All @@ -333,6 +361,18 @@ VORTEX_FETCH_DB2_S3_BUCKET=
VORTEX_FETCH_DB2_S3_REGION=ap-southeast-2
#;> MIGRATION_DB_FETCH_SOURCE_S3

# Database store of the second database dump file: ftp or s3.
#
# When set, the second database is fetched from the store instead of the
# source. The nightly CI job fetches from the source and pushes the dump, as
# fetched, to the store, next to the primary dump. Leave empty to fetch from
# the source.
VORTEX_FETCH_DB2_STORE=

# Destinations the nightly CI job pushes the second database dump to, as
# fetched, besides the database store: ftp, s3 or both.
# VORTEX_PUSH_DB2_DESTINATIONS=ftp,s3

# Environment to fetch the second database from.
#
# Applies to hosting environments.
Expand Down
70 changes: 66 additions & 4 deletions .github/workflows/build-test-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,13 @@ on:
description: 'Enable terminal session for CI jobs'
required: false
default: false
#;< !PROVISION_TYPE_PROFILE
refresh_db:
type: boolean
description: 'Refresh the database only'
required: false
default: false
#;> !PROVISION_TYPE_PROFILE
#;< !PROVISION_TYPE_PROFILE
schedule:
- cron: '0 18 * * *'
Expand All @@ -84,7 +91,7 @@ jobs:
lint:
runs-on: ubuntu-latest
#;< !PROVISION_TYPE_PROFILE
if: ${{ !inputs.deploy_target && github.event_name != 'schedule' && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')) }}
if: ${{ !inputs.deploy_target && !inputs.refresh_db && github.event_name != 'schedule' && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')) }}
#;> !PROVISION_TYPE_PROFILE

container:
Expand Down Expand Up @@ -225,6 +232,38 @@ jobs:
VORTEX_CI_DB_CACHE_FALLBACK: "yes"
# Which branch to use as a source of DB caches.
VORTEX_CI_DB_CACHE_BRANCH: "develop"
#;< DB_FETCH_SOURCE_FTP
VORTEX_FETCH_DB_FTP_USER: ${{ secrets.VORTEX_FETCH_DB_FTP_USER }}
VORTEX_FETCH_DB_FTP_PASS: ${{ secrets.VORTEX_FETCH_DB_FTP_PASS }}
#;> DB_FETCH_SOURCE_FTP
#;< DB_FETCH_SOURCE_S3
VORTEX_FETCH_DB_S3_ACCESS_KEY: ${{ secrets.VORTEX_FETCH_DB_S3_ACCESS_KEY }}
VORTEX_FETCH_DB_S3_SECRET_KEY: ${{ secrets.VORTEX_FETCH_DB_S3_SECRET_KEY }}
#;> DB_FETCH_SOURCE_S3
#;< MIGRATION_DB_FETCH_SOURCE_FTP
VORTEX_FETCH_DB2_FTP_USER: ${{ secrets.VORTEX_FETCH_DB2_FTP_USER }}
VORTEX_FETCH_DB2_FTP_PASS: ${{ secrets.VORTEX_FETCH_DB2_FTP_PASS }}
#;> MIGRATION_DB_FETCH_SOURCE_FTP
#;< MIGRATION_DB_FETCH_SOURCE_S3
VORTEX_FETCH_DB2_S3_ACCESS_KEY: ${{ secrets.VORTEX_FETCH_DB2_S3_ACCESS_KEY }}
VORTEX_FETCH_DB2_S3_SECRET_KEY: ${{ secrets.VORTEX_FETCH_DB2_S3_SECRET_KEY }}
#;> MIGRATION_DB_FETCH_SOURCE_S3
#;< VORTEX_DEV
# Push destinations of the demo databases on the nightly run. Empty
# repository variables skip the push.
VORTEX_PUSH_DB_DESTINATIONS: ${{ vars.VORTEX_PUSH_DB_DESTINATIONS }}
VORTEX_PUSH_DB2_DESTINATIONS: ${{ vars.VORTEX_PUSH_DB2_DESTINATIONS }}
# FTP server that receives both demo databases, in a directory that
# tells this provider's dumps apart, with a timestamped copy of each
# push.
VORTEX_PUSH_DB_FTP_HOST: ${{ secrets.VORTEX_PUSH_DB_FTP_HOST }}
VORTEX_PUSH_DB_FTP_PORT: ${{ secrets.VORTEX_PUSH_DB_FTP_PORT }}
VORTEX_PUSH_DB_FTP_USER: ${{ secrets.VORTEX_PUSH_DB_FTP_USER }}
VORTEX_PUSH_DB_FTP_PASS: ${{ secrets.VORTEX_PUSH_DB_FTP_PASS }}
VORTEX_PUSH_DB_FTP_DIR: ${{ vars.VORTEX_PUSH_DB_FTP_DIR || 'gha' }}
VORTEX_PUSH_DB_FTP_FILE: ${{ vars.VORTEX_PUSH_DB_FTP_FILE || 'db.sql' }}
VORTEX_PUSH_DB_TIMESTAMPED_COPY: 1
#;> VORTEX_DEV

steps:
# Frees disk space by removing preinstalled toolchains unused by this project. Steps run in a container, so the host is reached via the Docker socket.
Expand Down Expand Up @@ -270,7 +309,7 @@ jobs:
run: ./scripts/vortex-tooling.sh

- name: Adjust variables for a scheduled run
if: github.event_name == 'schedule'
if: github.event_name == 'schedule' || inputs.refresh_db
run: |
{
echo "VORTEX_CI_DB_CACHE_FALLBACK=no"
Expand All @@ -281,6 +320,8 @@ jobs:
# Deploy container image.
echo "VORTEX_EXPORT_DB_CONTAINER_REGISTRY_PUSH_PROCEED=1"
#;> DB_IN_IMAGE
# Push the sanitized database dump to the database store and destinations.
echo "VORTEX_PUSH_DB_PROCEED=1"
} >> "$GITHUB_ENV"

- name: Create cache keys files for database caching
Expand Down Expand Up @@ -343,6 +384,25 @@ jobs:
timeout-minutes: 30
#;> DB_FETCH_SOURCE_CONTAINER_REGISTRY

# Push the freshly-exported sanitized database dump to the push
# destinations. No-op unless a fresh DB was fetched (the fetch semaphore) and
# the push was requested (VORTEX_PUSH_DB_PROCEED, in the script).
- name: Push DB
run: |
if [ ! -f /tmp/fetch-db-fresh ]; then echo "==> No freshly fetched database. DB push will not proceed."; exit 0; fi
./vendor/bin/vortex-push-db
timeout-minutes: 30

#;< MIGRATION
# Push the freshly-fetched migration database dump to its push destinations,
# next to the primary dump. No-op unless the push was requested.
- name: Push migration DB
run: |
if [ ! -f /tmp/fetch-db-fresh ]; then echo "==> No freshly fetched database. DB push will not proceed."; exit 0; fi
VORTEX_DB_INDEX=2 ./vendor/bin/vortex-push-db
timeout-minutes: 30
#;> MIGRATION

# Save cache per default branch and the timestamp.
# The cache will not be saved if it already exists.
# Note that the cache fallback flag is enabled for this case in order
Expand All @@ -359,7 +419,7 @@ jobs:
runs-on: ubuntu-latest
#;< !PROVISION_TYPE_PROFILE
needs: database
if: ${{ !inputs.deploy_target && github.event_name != 'schedule' && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')) }}
if: ${{ !inputs.deploy_target && !inputs.refresh_db && github.event_name != 'schedule' && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')) }}
#;> !PROVISION_TYPE_PROFILE

permissions:
Expand Down Expand Up @@ -717,7 +777,9 @@ jobs:
runs-on: ubuntu-latest
needs: [build, lint]
#;< !PROVISION_TYPE_PROFILE
if: ${{ !cancelled() && (inputs.deploy_target || (success() && github.event_name != 'schedule' && !startsWith(github.head_ref || github.ref_name, 'deps/') && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')))) }}
if: >-
${{ !cancelled() && (inputs.deploy_target || (success() && !inputs.refresh_db && github.event_name != 'schedule'
&& !startsWith(github.head_ref || github.ref_name, 'deps/') && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')))) }}
#;> !PROVISION_TYPE_PROFILE

permissions:
Expand Down
Loading
Loading