System Design

System Design

External context, the deliberate two-pipeline decomposition, component responsibilities, and the layered configuration that drives every experiment.

Vesuvius Segmentation — System Context An architecture diagram generated by Archify. Competition Data · CT volumes, sparse labels · Architecture component · TIF stacks Competition Data CT volumes, sparse labels TIF stacks Preprocessing · convert, normalize, patch · Architecture component Preprocessing convert, normalize, patch Custom Pipeline · dual-head DynUNet · Architecture component · src/approach/unetbasic Custom Pipeline dual-head DynUNet src/approach/unetbasic nnU-Net v2 Baseline · plan, train, predict · Architecture component · scripts/nnunetbaseline nnU-Net v2 Baseline plan, train, predict scripts/nnunetbaseline Weights & Biases · runs, metrics, configs · Architecture component Weights & Biases runs, metrics, configs Run Artifacts · checkpoints, predictions · Architecture component Run Artifacts checkpoints, predictions Competition Metrics · TopoScore, VoiScore · Architecture component · installed package Competition Metrics TopoScore, VoiScore installed package Scoring Lab · post-process and score · Architecture component · exploration/scoring Scoring Lab post-process and score exploration/scoring Kaggle Submission · scored result · Architecture component Kaggle Submission scored result Inference Package · republished to Kaggle · Architecture component · kaggle_datasets/inference Inference Package republished to Kaggle kaggle_datasets/inference TIF stacks normalized patches preprocessed dataset runs and metrics checkpoints, predictions checkpoints, predictions probability + distance maps metric definitions scored predictions packaged inference code published dataset Legend Backend Database Cloud External
The system context and component map. Two training pipelines share one dataset contract and one evaluation surface; experiment tracking, the Kaggle dataset registry and the documentation portal are the only external touchpoints.

1. System context

The repository is a self-contained research system: it trains on local CT volumes, writes all artifacts to a local output tree, and reports metrics locally. Nothing in the critical path requires a network service. What it does touch is four external surfaces:

External surfaceDirectionPurpose
Raw CT scans Input 3D TIF volumes, plus voxel-size information, converted to .npy for training.
Weights & Biases Out Experiment tracking: metrics, loss curves, run configuration and predictions.
Kaggle Both Competition data and leaderboard on one side; a published inference-code dataset and submitted predictions on the other.
Topological metrics package Input The reference implementation of the competition metric, supplied as an installable package rather than vendored into this repo.

2. The two-pipeline decomposition

There are two end-to-end routes from raw scan to scored prediction. This is a deliberate structure, not an accident of history, and it is worth being explicit about why both exist.

 Custom pipelinennU-Net v2 pipeline
Core idea Hand-designed architecture and losses aimed specifically at thin-manifold topology. A well-tuned general-purpose framework applied to the same dataset.
Model Dual-head 3D DynUNet: binary mask head + signed-distance head. Residual-encoder 3D U-Net, configured from an explicit plans file.
Training driver Hugging Face Trainer with torchrun for DDP. nnU-Net’s own trainer with its native pipeline commands.
Preprocessing MONAI transform chain, composed in the dataset. nnU-Net fingerprint-driven preprocessing derived from the dataset properties.
Inference Sliding-window inference producing probability and distance arrays. Partitioned multi-GPU prediction across numbered parts, then TIFF export.
Primary strength Direct control over the topology / overlap trade-off via the loss suite. Robust default performance with very little per-experiment configuration.
Main entry point src/approach/unetbasic/train.py scripts/nnunetbaseline/train_nnunetv2.py
Why keep both. The custom pipeline is where architecture and loss ideas are tested, because every design decision is visible and adjustable. The nnU-Net pipeline provides a strong, low-effort baseline and a second opinion: when a custom experiment beats it, the gain is attributable to the modelling change rather than to incidental training details. Both write predictions in the same shape, so both are scored by the same code.

3. Component responsibilities

ComponentResponsibility
src/approach/unetbasic/train.py Training orchestration: resolves configuration, builds datasets, instantiates the model and loss, applies the loss-weight curriculum each epoch, wires logging and checkpointing, and runs distributed training.
src/approach/unetbasic/data/dataset.py The data contract. Implements the custom MONAI transforms — foreground extraction, valid-mask construction, signed and unsigned distance transforms — and assembles the train and validation transform pipelines.
src/approach/unetbasic/models/unet.py Network definitions: a straightforward single-head U-Net and the dual-head topology-preserving variant that also predicts a signed distance field.
src/approach/unetbasic/utils/losses.py The loss suite: thin-manifold, topology-preserving and anti-bifurcation losses, each assembled from weighted components.
src/approach/unetbasic/utils/loss_weight_schedule.py The curriculum: interpolates each loss component between a start and end weight across a configured epoch range.
src/approach/unetbasic/inference.py Standalone sliding-window inference from a trained checkpoint, writing probability and distance arrays per case.
src/approach/unetbasic/metrics/kaggle_metrics.py The competition metric implementation and the post-processing operations applied before scoring, with multiprocess orchestration over cases.
scripts/nnunetbaseline/train_nnunetv2.py The complete nnU-Net lifecycle: dataset preparation, preprocessing, training, prediction, TIFF export and submission generation.
exploration/scoring/ The offline scoring lab: post-processing strategy families, probability-space operations, and the harness that scores a prediction directory.
configs/ Shared Hydra configuration: global paths and runtime settings, data defaults, and tracking settings, composed with a per-experiment YAML.
kaggle_datasets/inference/ The slice of the codebase published as a Kaggle dataset so that submission-time inference runs in the competition notebook environment.

4. Configuration-driven experiments

Both pipelines are configured declaratively; experiment variation is expressed as data, not as code branches. The composition follows a strict precedence order, so a new experiment only has to state what it changes: 

configs/globals.yaml                     shared paths and runtime settings
configs/data/base.yaml                   competition data paths
configs/wandb/wandb.yaml                 experiment-tracking project and defaults
        ↓ composed with
src/approach/<approach>/configs/experiments/<experiment>.yaml
        = resolved configuration for one run

The resolved configuration is the single input to a run, which is why the model, the loss weights, the augmentation set and the training loop are all reproducible from one YAML file. The resolved copy is persisted alongside the run outputs, and Project Structure documents the override conventions in detail.

Approach subpackages An approach is a self-contained directory under src/approach/ holding its own model, dataset, training entry point and experiment YAMLs. This is the extension point for new ideas: a new approach is added by creating a new sibling directory, without modifying existing ones. Pipelines describes what the current approach contains.