Skip to content

About

debug repo for federated learning

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

MS_FL_DEBUG — federated flip-detection debug task

A stripped-down copy of the UCD ↔ Padova federated OCT classifier (MS_NF_FL), used to debug the federation itself. The task is trivially learnable, so a run that fails or stalls points at the pipeline rather than the data. The task:

  • Each site draws 150 B-scans from its own data. The MS/CTRL labels are ignored.
  • 75 of the 150 are flipped vertically.
  • RETFound ViT-L is trained to predict 0 = upright, 1 = flipped.

The federation works exactly like the MS classifier:

  1. Start the server.
  2. It waits for both sites to connect.
  3. It runs 10 rounds × 3 local epochs of FedAvg.

The federation runs over HTTPS (TLS) on port 9092. Clients only make outbound connections, and only weights and scalar metrics leave a site. Tokens and weights are encrypted in transit, and each client checks the server's certificate before sending anything. See TLS certificates.

Expected result: val accuracy and AUC near 1.0 at both sites within a few rounds.

Data at each site

--data-path is searched recursively for images (.png/.jpg/.tif/...). The files are sorted, and 150 are drawn with a fixed seed. Of those, 75 are flipped. The split is:

split upright flipped
train 50 50
val 25 25

The selection is the same on every run. Images are decoded once at startup and kept in memory. No caching step is needed. UniPD images get the usual 500 px left crop before the resize to 224×448.

Installation (both sites)

git clone <repo-url> MS_FL_DEBUG
cd MS_FL_DEBUG
pip install -r requirements.txt

Use the same requirements.txt at both sites. timm in particular must match, because the server and the clients compare their state_dict layouts at join.

Running

1. Server (UCD)

Create the TLS certificates once (see TLS certificates). Set the site tokens in pyproject.toml under [tool.fl.sites.<id>], then run:

bash run_trial.sh

The server loads RETFound from retfound-weights, listens on port 9092, waits for ucd and unipd to connect, then drives the rounds. You can override any [tool.fl.config] value on the command line, for example --run-config "num-server-rounds=10 local-epochs=3 trial-tag='x'".

2. Clients (one per site)

UCD:

bash run_ucd_client.sh

Padova: edit run_unipd_client.sh (data path, token, optional weights/save paths), then run:

bash run_unipd_client.sh

The client flags are the same as in the MS classifier, minus --cache-path:

FL_TOKEN=<site token> python fl_client.py \
  --site unipd \
  --data-path /path/to/oct/images \
  --server https://140.226.4.75:9092 \
  --ca-cert certs/ca.crt \
  --trial-tag flip_debug \
  --gpu-index 0
  • --site must match a [tool.fl.sites.<id>] on the server, and the token must match that site's token.
  • --ca-cert defaults to certs/ca.crt in the repo. It is how the client verifies that it is talking to the real server.
  • --weights-path is optional. It is only fingerprinted against the server's checkpoint and never used to initialise.
  • Clients need at least 24 GB of host RAM, since each 1.2 GB weight exchange transiently holds about 3.6 GB.

TLS certificates

All traffic is TLS 1.2 or newer. The server will not start without a certificate, and clients will not connect over plain http://. The bearer tokens are still checked, as a second factor.

UCD creates a private certificate authority (CA) and a server certificate once, on the server machine, from the repo root:

mkdir -p certs && cd certs

# 1. private CA (ca.key stays on the server, never shared or committed)
openssl req -x509 -newkey rsa:4096 -sha256 -days 730 -nodes \
  -keyout ca.key -out ca.crt -subj "/CN=MS_FL_DEBUG CA"

# 2. server key + certificate signed by the CA. The subjectAltName must list
#    every address clients use in --server.
openssl req -newkey rsa:4096 -sha256 -nodes \
  -keyout server.key -out server.csr -subj "/CN=140.226.4.75"
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 730 -sha256 -out server.crt \
  -extfile <(printf "subjectAltName=IP:140.226.4.75,IP:127.0.0.1\nextendedKeyUsage=serverAuth")
  • server.crt and server.key are what pyproject.toml points at (tls-cert and tls-key).
  • ca.crt is the only file Padova needs. It is public. Either commit it as certs/ca.crt or send it to them, and they put it in their certs/ folder.
  • *.key files are in .gitignore. Never commit or send them.

To check from Padova that the server is reachable and presents the right certificate:

openssl s_client -connect 140.226.4.75:9092 -CAfile certs/ca.crt </dev/null | grep -E "Verify return|Protocol"

Optional: mutual TLS. This requires each site to also hold a certificate.

  1. Sign a certificate for each site with the same CA, using extendedKeyUsage=clientAuth.
  2. Uncomment tls-client-ca = "certs/ca.crt" in pyproject.toml.
  3. Have each client pass --client-cert <site>.crt --client-key <site>.key.

Outputs

On the server, under results/<trial-tag>/ and runs/fl_server/<trial-tag>/ (TensorBoard), the files are the same as in the MS classifier:

  • metrics_global.csv, metrics_per_site.csv
  • loss_curves.png, auc_curves.png
  • model_federated_final.pth, model_best_val_auc.pth
  • final_metrics.json, log.log, failure.log (written on abort)

On each client, under runs/fl_clients/<site>/<trial-tag>/:

  • TensorBoard scalars
  • model_federated_final.pth (or the file at --save-path)
  • failure.log (written on abort)

The run is fail-fast. Any client error, heartbeat drop, or round timeout aborts the whole run, and the reason names the site.

Files

File Role
fl_server.py Server: HTTP endpoints, FedAvg, orchestration, logging, checkpoints
fl_client.py Client: pulls weights, trains/evaluates locally, pushes results
fl_common.py Wire format (.npz + manifest validation), FedAvg
model.py RETFound ViT-L binary classifier (build_model)
data.py 150-image selection, vertical flip, transforms (FlipDataset)
train.py Local train/evaluate (AdamW + layer-wise LR decay), AUC, plots
run_server.py Entry point for the server
*.sh Launch scripts

About

debug repo for federated learning

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages