GotthardGandClaude Fable 5.1
4698dbeaf9
CI / ledger-and-tools (push) Successful in 17s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
29 lines
1.3 KiB
Markdown
29 lines
1.3 KiB
Markdown
# Training on the GPU cluster (template, not yet filled in)
|
|
|
|
`train.sbatch` runs one ledger run on the `mx` partition. It needs these facts,
|
|
which replace the `__PLACEHOLDERS__` in the script:
|
|
|
|
| Placeholder | What | Value |
|
|
| --- | --- | --- |
|
|
| `__PARTITION__` | SLURM partition | `mx` |
|
|
| `__ACCOUNT__` | SLURM account to charge | |
|
|
| `__WALLTIME__` | wall-time limit for 400 epochs of yolo26n at 640 px | |
|
|
| `__SHARED_FS__` | shared filesystem visible from login and compute nodes, holds the repo checkout | |
|
|
| `__MODULES__` / `__VENV__` / `__CONTAINER__` | how Python, CUDA and ultralytics are provided on compute nodes (one of the three) | |
|
|
| `__DEPLOY_KEY__` | SSH key on the cluster allowed to push to this repo, if the job reports back itself | |
|
|
|
|
Still to confirm before the first job: the cluster and login node names; whether
|
|
compute nodes reach gitea.psi.ch (git + LFS) and PyPI, or whether a container has to
|
|
carry everything; the GPU type on the partition; who submits (a service account or
|
|
a person). Until then the reference runs happen on the DGX Spark with the same
|
|
`tools/train_run.py`.
|
|
|
|
Workflow once filled in:
|
|
|
|
```bash
|
|
python tools/ledger.py new --dataset dataset/<name> --tag <tag> --base yolo26n-seg.pt
|
|
git commit -am "run <id>: exported" && git push
|
|
sbatch slurm/train.sbatch <id>
|
|
python tools/ledger.py show
|
|
```
|