Pipelines

Both training pipelines end to end: experiment launch, the curriculum-driven custom training loop with distributed execution and sliding-window validation, and the complete nnU-Net v2 lifecycle.

1. The custom MONAI pipeline

This is the repository’s primary pipeline: a dual-head 3D network trained through a Hugging Face Trainer, driven by layered configuration, with a loss-weight curriculum applied every epoch. It is the pipeline to read first if you intend to change how the model learns.

1.1 Experiment launch and startup sequence

A run is identified by a pair: an approach (the subpackage under src/approach/) and an experiment (an YAML file under that approach’s configs/experiments/). Startup resolves those into one concrete configuration and then performs an ordered sequence of side effects before training begins:

Because step 4 persists the resolved configuration next to the outputs, two experiments can be diffed from their output directories alone — no need to reconstruct which YAML overrides were in play. The output layout is described in Deployment.

1.2 Distributed training

Multi-GPU training is the default path rather than an advanced option. The entry point is expected to be launched under a distributed launcher, which supplies the rank and world size through the standard environment; the code reads those to decide sharding and to make logging and checkpointing rank-aware so only one process writes to disk or reports metrics. A single-device run is the degenerate case of the same code path.

1.3 The training loop

The loop itself is deliberately conventional — the design effort went into the data contract and the objective, not into reimplementing an optimiser. What is worth noting is the set of behaviours layered onto it:

1.4 Why a loss-weight curriculum

The loss suite can contain terms that are only meaningful once the model produces a recognisable surface. A topological term evaluated on garbage output gives a gradient that points in an unhelpful direction, so the curriculum starts the objective simple and ramps complexity in. The schedule is a tool, not an opinion: a component with the same start and end weight is simply constant, and a component that starts at zero is introduced gradually.

1.5 Validation and progress metrics

Validation is a sliding-window pass over the held-out cases, using a region of interest matching the training patch size and a configured window overlap, with overlapping window predictions averaged to suppress seams. This is the same inference machinery used for standalone prediction, which means the number reported during training is the number the next stage will see.

Alongside the loss, the trainer computes per-sample statistics from the logits at each validation round and derives a small set of aggregate metrics. These are progress indicators computed from the validation tensors — they track the shape of the learning curve during a run. The authoritative competition score is computed separately from written prediction volumes, as described in Evaluation & Metrics.

2. The nnU-Net v2 pipeline

The second pipeline applies nnU-Net v2 to the same dataset. It is a first-class path, not a fallback: it provides a strong reference point and an independent implementation of the same task, which makes custom-pipeline gains interpretable.

2.1 Dataset preparation

nnU-Net expects a specific on-disk dataset layout with a JSON descriptor declaring modalities, labels and file endings. The repository’s driver translates its own data layout into that structure, generating the descriptor programmatically so label semantics stay in one place rather than being duplicated in a hand-written JSON file. A separate spacing sidecar carries voxel-size information, because the physical resolution is not encoded in the TIFF files themselves.

2.2 Fingerprint-driven preprocessing

Preprocessing is planned automatically. nnU-Net analyses the dataset to compute an intensity and geometry fingerprint, then derives the preprocessing pipeline and the network configuration from it. In this project that yields isotropic spacing, a median input volume close to 320 × 314 × 314 with 128 × 128 × 128 training patches, and a residual-encoder 3D U-Net plan — the plan used in practice is checked in as a reference under docs/.

The checked-in plans file docs/plans.json is an nnU-Net plans file rather than documentation. It is kept in the repository so the exact plan a reported result was produced with can be inspected without re-running fingerprinting, and it is referenced from Models & Losses and Deployment.

2.3 Training

Training runs through nnU-Net’s own trainer rather than the Hugging Face Trainer used by the custom pipeline. The driver wraps the framework’s commands, exposing dataset preparation, preprocessing, training, prediction and export as callable steps, and composes them into a full end-to-end pipeline that can also be run one stage at a time. Stage-at-a-time execution is what makes the pipeline practical to debug: preprocessing is a long-running step that should not be repeated because a training hyperparameter changed.

2.4 Partitioned multi-GPU inference

Inference over the whole test set is spread across devices rather than run on one. The inference driver launches one predictor subprocess per GPU, each restricted to a single visible device and given its own partition index and partition count. The split is over the case list rather than over space: every subprocess writes into the same predictions directory, and once they have all finished the driver converts the combined output.

inference driver
 ├─ GPU 0  nnUNetv2_predict  -part_id 0 -num_parts N   → case subset 0
 ├─ GPU 1  nnUNetv2_predict  -part_id 1 -num_parts N   → case subset 1
 └─ ...
 shared predictions directory  →  per-case label volumes  →  TIFF export

Because the framework expects a specific results-directory layout, the driver builds a symlinked structure pointing at the trained folds, so existing checkpoints can be used without moving or copying them. The wrapper also forces the channels-last-3D memory format, which is a meaningful throughput win for 3D convolutions on the target hardware.

2.5 Export and submission generation

Predicted volumes are converted back to TIFF for submission. The driver separates the conversion step from the generation step so a failed export can be retried without re-running inference. The inference code itself is also packaged and published as a Kaggle dataset, so the same prediction logic runs in the competition notebook environment; that packaging flow is covered in Deployment.

3. Choosing between the pipelines