codespaces

Description:

This repo contains all the environmental tools/dependencies to deploy the entire glueops platform. Tools include but are not limited to: terraform, helm, kubectl, devbox/nix, etc. We primarily use this repository in all of our codespaces as well as github actions. Ref: https://github.com/GlueOps/glueops/blob/%F0%9F%9A%80%F0%9F%92%8E%F0%9F%99%8C%F0%9F%9A%80/.devcontainer/devcontainer.json#L5

DevBox

Since we have devbox installed any packages/tools that aren't installed (e.g. python) can easily be fetched from https://www.nixhub.io/

Here is a quick getting started with devbox: https://jetify.com/docs/devbox/quickstart/#create-a-development-environment

devbox init
devbox add python@3.10
devbox shell
python --version

The codespace VM: boot, recovery and dev

A codespace VM runs one container, codespace, and keeps it running by itself. Nobody has to log in after a reboot, a crash or an out-of-memory kill.

• The container restarts itself (--restart unless-stopped). Its main process is the editor: code serve-web on 127.0.0.1:8000 when the Slack bot wrote a connection token, otherwise a VS Code tunnel (code tunnel) that you log in to with a GitHub device code.
• codespace-ensure (codespace-ensure.service, at boot and every minute from codespace-ensure.timer) starts the container if it's stopped, and creates it from the image tag in /etc/glueops/codespace-tag if it's missing. A container you stop with docker stop is started again within a minute.
• codespace-tunnel.service runs the sish reverse tunnel for the browser editor and reconnects whenever it drops.
• Home (/home/vscode) and /nix are named volumes (codespace-home, codespace-nix), so a recreated container keeps your dotfiles, logins and devbox packages. The image build fills them, so a new codespace starts quickly; after dev reset the first start copies them again and takes a few minutes. /workspaces/glueops is on the VM's disk. None of this survives deleting the VM: commit and push.

When the editor process restarts (after an OOM kill, for example), the browser tab says it can't reconnect: reload the page. Terminals and anything else running inside the container are gone after a restart.

Run dev (as any user) to check on it:

dev          repair if needed, show status, open the `dev` tmux session (window `logs`, window `shell`)
dev status   status only; exit 0 if healthy, 1 if not
dev logs     follow the codespace's logs (the `logs` window runs this; Ctrl-C there is harmless)
dev reset    delete the container AND its home and /nix volumes, then create it again

dev status shows the device code when a VS Code tunnel is waiting for its first login (it's also in docker logs codespace). After that first login the tunnel reconnects by itself. Low-level logs: sudo journalctl -u codespace-ensure -u codespace-tunnel. The secrets in /etc/glueops/codespace.env are passed to the container as environment variables, so they show in docker inspect, as before.

CDE bootstrap: cde-init / CDE_SETUP_SCRIPT

When codespace-ensure creates or starts the container, it runs a one-time bootstrap (cde-boot) driven by the CDE_SETUP_SCRIPT environment variable (supplied by the Slack bot via /etc/glueops/codespace.env). The value selects what runs:

• unset / empty (the default — the Slack bot leaves it commented out) — runs cde-init: authenticates GitHub from GITHUB_TOKEN (non-interactive; runs gh auth setup-git so git/private-repo clones work, and sets your git identity), clones GLUEOPS_CDE_CLONE_REPO into /workspaces/glueops/<repo> (public repos need no token), and seeds gluekube_ssh (AutoGlue) prod/nonprod profiles from GLUEKUBE_SSH_AUTOGLUE_{PROD,NONPROD}_{URL,TOKEN} — a profile is seeded only when both its URL and token are set.
• a command — e.g. curl setup.example.com | zsh runs instead of cde-init.
• base64:<encoded> — a base64-encoded script, for complex/multi-line setup (<your script> | base64 -w0). A decode failure is refused (never runs a partial script).
• true (or any no-op) — skip setup entirely.

Every step is best-effort and non-fatal; unconfigured inputs are skipped. It runs once per home volume: the sentinel ~/.cde-init-done lives in home, so a recreated container doesn't run it again. The sentinel is written only when setup exits 0, so a hard failure (or a bad base64: value) is tried again the next time codespace-ensure creates or starts the container, rather than silently marked done; output is captured to ~/.cde-init.log. Set CDE_INIT_FORCE=1, or just run cde-init / cde-boot again, to re-run. The tools live in .devcontainer/tools/ (cde-init.sh, cde-boot.sh) and are installed to /usr/local/bin by the image build.

Because the default GitHub auth is token-based, running yolo afterward is a smooth no-op (it detects the existing auth and skips the interactive browser login).

captain_utils: upgrading a cluster

captain_utils is the menu an operator drives a cluster upgrade from, run at the captain repo root. Its components are chosen in order:

captain_utils → crds              # the layer-0 CRD bundle (GlueOps/platform-crds)
captain_utils → argocd
captain_utils → crds              # again — see below
captain_utils → glueops-platform

crds runs before ArgoCD and again after it. The bundle is the only installer of platform CRDs — no ArgoCD Application renders one — so the CRDs have to exist before ArgoCD is deployed. The second run recreates anything the argocd release removed on the way past (docs-argocd ≥ 0.20.0 drops the Gate CRD it used to own); on a cluster that is already past that, it reports no changes.

The step applies with kubectl apply --server-side --force-conflicts --field-manager=glueops-platform-crds and nothing else. It never runs kubectl replace on a CRD: that is a whole-object PUT, and it erased metadata.finalizers on a CRD that was mid-deletion, destroying the type and orphaning every object of it.

Before applying, it prints what the run will do — content changes, any ArgoCD tracking metadata it will strip, and whether anything is Terminating — then shows the diff and asks. Declining changes nothing.

It never writes without a yes. When nothing differs it says so and offers a re-apply rather than doing one, so the second run above normally reports in sync, nothing applied and stops. Answering yes is still useful when there is tracking metadata to strip or field ownership to consolidate — neither of which a diff can show. Without a terminal it skips quietly and exits 0; CRDS_AUTO_CONFIRM=yes answers yes, as it does everywhere else.

crds check          # read-only: 0 in sync with the pinned bundle, 1 drift, 2 failure. Mutates nothing.

A cluster is on the bundle once its captain repo pins platform_crds_version in VERSIONS/glueops.yaml. Repos that predate the pin keep the legacy path, where the argocd step installs ArgoCD's own CRDs.

The command lives at .devcontainer/libexec/captain_utils/crds, installed to /usr/local/libexec/captain_utils/ and deliberately off the PATH; its file header carries the full usage and the contract it is held to. Tests: hack/test-captain-utils.sh (offline, seconds) and CRDS_TEST_KIND=1 hack/test-captain-utils.sh for the cluster test. (hack/test.sh runs every app's suite, including gluekube_ssh's, which requires python3, and the VM suite hack/test-codespace-vm.sh with CODESPACE_VM_TEST_DOCKER=1, which needs docker.)

Releasing:

• Please stick to semver standards when dropping a new tag.
• Once you publish a release a new image will be built and uploaded to GHCR.io: https://github.com/GlueOps/codespaces/pkgs/container/codespaces
• Please tag the release as a pre-release this will prevent it from being pulled by production users whie it's still building. Once the AWS/QMEU/etc. pipelines have finished, and after you have tested the pre-release go ahead and update the release to latest release. Here is a quick video on promoting from pre-release to the latest release

Note: Due to the connection between the packer workflows and the Docker build/publish workflow, it's not possible to cut a release from any branch other than main. Some limitations in GitHub prevent the actions checkout step in the packer workflows from accurately determining the parent commit SHA, which defaults to main. Therefore, if you're testing changes outside of the Dockerfile, this method may not provide accurate results. One potential solution is to combine all the workflows, but this would result in a runtime of an hour or more.

QA / Testing:

To test a pre-release you will first to do the following:

1. Make sure a nonprod/stage provisioner API functioning is working and has a node for you to test with.
2. In the nonprod slack channel for the developer workspaces (e.g. #testing-developer-workspaces) run !vm and you should see the pre-release tag.
3. Select the pre-release tag. The VM starts the codespace from that tag by itself; log in and run dev to see it.
4. To try a different image tag on that VM: export ENVIRONMENT=nonprod, then dev reset and pick the tag. This deletes the container and its home and /nix volumes.
5. Before promoting the pre-release, run hack/vm-recovery-test.sh root@<vm> against a nonprod VM created this way (one with a token and one without; on the one without, finish the device login first: the code is in dev status): it kills the editor, the tunnel and the container, restarts docker and tailscaled, and reboots, and checks the codespace comes back each time and the reboot keeps the VM's SSH host keys. Run it on a libvirt or Proxmox VM: they attach the cloud-init seed only at first boot, so only there does a reboot show whether cloud-init re-initialises the VM. Also check the first boot: the URL from Slack should work a few minutes after the VM is created, without logging in.

Local Dev / Running packer locally:

It's best to just reference the github workflows under .github/workflows the packer workflows for each respective cloud start with packer-*. For each respective cloud you will notice env variables are being passed into a github action step. To do this locally, you will need to create credentials for the respective cloud and then export the applicable environment variables before running the packer build command. The qemu build compresses the image with zstd, so it needs qemu-img 5.1 or newer (as does anything that reads the image). Its build VM defaults to 12 GB of RAM and 4 vCPUs (-var vm_memory=... -var vm_cpus=... to change; CI uses 24 GB / 8).

Break Glass Setup

If you are having issues spinning up a VM using our automation, just create a VM/Server with a provider of your choice and run these commands:

# Optional, export GLUEOPS_CODESPACES_CONTAINER_TAG if you want to specify a specific version. Otherwise the latest release will be grabbed/pulled. Note: if the latest release was just published then it may not be available for download yet and the bootstrap may fail
# export GLUEOPS_CODESPACES_CONTAINER_TAG=v0.129.0 #update tag to latest version found here: https://github.com/GlueOps/codespaces/releases
curl -sL setup.glueops.dev | bash
sudo tailscale up --ssh
sudo reboot   # the codespace starts on boot; or run `dev` to start it now

Note: as of April 2025 we have been using Debian 12 as our base Operating System. It's possible Debian 13 or newer work but we haven't tested it yet.