diff --git a/CMakeLists.txt b/CMakeLists.txt index 57c83a8ef..79801b719 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -47,11 +47,16 @@ ENDIF() # came out differently from the same source and data. Off, every x86-64 level computes the same bits # (measured: identical p.mtz on nine battery sets, no measurable time cost), and aarch64, where FMA is # always there and GCC contracts by default, follows the same arithmetic. MSVC does not contract under -# its default /fp:precise. Host code of .cu sources gets it through nvcc; device code is not affected. +# its default /fp:precise. Host code of .cu sources gets it through nvcc. IF (NOT MSVC) ADD_COMPILE_OPTIONS($<$:-ffp-contract=off>) ADD_COMPILE_OPTIONS($<$:-Xcompiler=-ffp-contract=off>) ENDIF() +# The same for device code. nvcc contracts by default and does it separately for every architecture +# it compiles for, so a T4 (sm_75) and an RTX 5080 (sm_120) ran the same kernels to different bits +# and a run that hit a knife edge came out differently on the two machines. Off, the architectures +# compute the same bits (measured: identical p.hkl from sm_75 and sm_120 builds, no measurable time cost). +ADD_COMPILE_OPTIONS($<$:--fmad=false>) SET(JFJOCH_WRITER_ONLY OFF CACHE BOOL "Compile HDF5 writer only") SET(JFJOCH_INSTALL_DRIVER_SOURCE OFF CACHE BOOL "Install kernel driver source (ignored if building writer only; necessary for RPM building)") diff --git a/docs/JFJOCH_WRITER.md b/docs/JFJOCH_WRITER.md index 3bd9f8823..1ebca2e6e 100644 --- a/docs/JFJOCH_WRITER.md +++ b/docs/JFJOCH_WRITER.md @@ -21,7 +21,8 @@ Options: -d | --root_dir= Root directory for file writing (-R is a deprecated alias) -r | --zmq_repub_port= ZeroMQ port for PUSH socket to republish images -f | --zmq_file_port= ZeroMQ port for PUB socket for notifications on finalized files --w | --rcv_watermark= Receiving ZeroMQ socket watermark (default = 100) +-w | --rcv_watermark= Receiving ZeroMQ socket watermark in messages (default = 10) +-m | --rcv_ring_mb= Receive ring buffer in MB; holds images until written and republished (default = 2048) -W | --repub_watermark= Republish ZeroMQ socket watermark (default = 1000) -b | --rcv_buffer= Receiving TCP socket buffer in bytes (only with -T; disables autotuning, capped by net.core.rmem_max) -v | --verbose Verbose output @@ -32,6 +33,33 @@ for example: jfjoch_writer -d /data tcp://dcu-address:5400 ``` +## Memory +Every message the writer receives is placed in a ring buffer of a fixed size (`-m`, default 2048 MB, +per writer process - `-j 8` is eight rings) and stays there until it has been written to HDF5 and, if +republish is on, republished. The ring is the writer's memory budget for buffered images, and the +peak use is printed at the end of every run. Size it as the number of images you can afford to have +in flight times their compressed size: at 2048 MB that is some 250-500 JUNGFRAU 4M images but only +50-100 JUNGFRAU 9M images, so a 9M site will want more. + +When the ring is full the writer stops reading from the socket and the sender has to wait. How long +it is prepared to wait depends on the stream: + +- **ZeroMQ.** The socket queue fills up to its watermark (`-w`, a count of messages, kept small for + that reason) and the broker's send blocks for one second. After that the broker drops the image, + logs the fact, and sends without waiting for the rest of the run, so every later image that finds + the queue full is dropped too. A writer that stalls for longer than its ring covers therefore + loses images on ZeroMQ; the ring has to be large enough for the stalls of the filesystem it writes + to. (Before the ring the same stall was absorbed by queues of about 1500 messages, whatever their + size - a budget nobody had chosen.) +- **TCP/IP.** The broker sees the writer's BUSY heartbeats and waits, giving up only after a minute + in which not a single image was written. + +Republish holds its own images in the same ring. Because the ring is reclaimed in arrival order, an +image kept for a slow republish consumer would hold back everything received after it, so republish +may hold at most a quarter of the ring; beyond that images are simply not republished, exactly as +when its queue is full. A single message larger than the whole ring - a START or END message of a +very long run - is held outside it. + ## Status and cancellation When a data collection is finalized, each writer reports its outcome back to `jfjoch_broker` over the writer notification socket — a ZeroMQ address the broker passes in the START message diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index 6a38593f8..5b4b945cd 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -192,6 +192,8 @@ One page per job, so the answer needed is near the top of a short page: anisotropy section. - [Advanced usage](RUGNUX_ADVANCED.md) — reference data and the indexing ambiguity, model validation, re-merging, and the full command-line option tables. +- [Models and reference data](RUGNUX_MODEL_AND_REFERENCE.md) — every model and reference option, + and decision by decision what a model can and cannot change. - [Detector calibration](RUGNUX_CALIBRATION.md) — the geometry from a calibrant's powder rings (`--mode calibration`). - [CPU/GPU data analysis](CPU_DATA_ANALYSIS.md) — the algorithms behind all of it. diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 719b903dd..4bcf8edd9 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -147,6 +147,9 @@ determined on, CCP4 style (`k,l,h` is h'=k, k'=l, l'=h); `h,k,l` where nothing m ## Validating against a model (`rugnux --model`) +Which decisions a model can influence, and which it cannot, is set out decision by decision in +[Models and reference data](RUGNUX_MODEL_AND_REFERENCE.md). + Given an atomic model of the same structure, `--model model.pdb` scales the model structure factors to the merged amplitudes — fitting a flat bulk-solvent contribution and an overall anisotropic *B* — and reports **R-work / R-free** and the mean 2mFo−DFc density at the atom centres. diff --git a/docs/RUGNUX_MODEL_AND_REFERENCE.md b/docs/RUGNUX_MODEL_AND_REFERENCE.md new file mode 100644 index 000000000..1cc62e8c2 --- /dev/null +++ b/docs/RUGNUX_MODEL_AND_REFERENCE.md @@ -0,0 +1,333 @@ +# Models and reference data + +What Rugnux does with outside knowledge — an atomic model, a predicted model, a reference dataset, a +cell or a space group you already know — and, decision by decision, what that knowledge is and is not +allowed to change. The first section is the short version; the tables after it are the precise one; +the rest walks through a run step by step. + +```{contents} On this page +:local: +:depth: 2 +``` + +## In short + +- **Rugnux needs nothing but the images.** Indexing, the cell, the space group, the resolution cut, + scaling, merging and the error model all come from the data. Everything on this page is optional. +- **A model (`--model`, `--mr`) does not change how your data are processed.** The images are + indexed, integrated, scaled and merged the same way with or without it, and the space group is + determined before the model is used. The merged intensities and their sigmas in + `.mtz` are the same numbers either way. +- **The model can only relabel, and only if the data accept it.** It is first tested against + copies of itself placed in random orientations. A model that fits no better than those changes + nothing at all: the reflection files are byte for byte what a run without a model writes. A model + that does fit may change three *labels* in `.mtz`: the **hand** of the space group + (P4₁2₁2 against P4₃2₁2, which the intensities cannot tell apart), the **indexing** where the + lattice allows two equally valid ones, and the **axes** the files are written on. No + intensity is changed, no reflection is added or removed, and no other space group is ever taken. +- **A different space group chosen by the model goes to a separate file.** If the model prefers + another space group, or a twinned subgroup, that merge is written *beside* the answer as + `_mr.mtz`/`.cif`, with the model as placed there. `.mtz` stays the data's own + answer. +- **A reference dataset (`-z`) and a space group or cell you give (`-S`, `-C`) are different:** they + are taken as facts about the crystal. They fix the space group and the cell before processing, + so with them the space group is yours, not the data's. +- **One exception to "processed the same":** on **serial stills** given `--model` together with + `-C` and `-S`, the model is used during integration to choose, crystal by crystal, between the two + equally valid indexings — the same job a reference dataset does there (see + [Serial stills](#serial-stills-the-one-place-a-model-acts-before-the-merge)). + +## Every option that brings in a model or reference data + +| Option | What you give | What Rugnux uses it for | Can it change `.mtz`? | +|---|---|---|---| +| *(none)* | — | Everything from the data | — | +| `--model ` | An atomic model, PDB or mmCIF, gzipped or not, recognised by content: an experimental structure or a prediction | Validation after the merge (R-work/R-free, CC(model, data), maps, the placed model); a rigid-body placement checked against a null, with molecular replacement where the rigid body does not place it, and straight away for a prediction | Labels only (hand, indexing, axes), and only where the model fits. Other space groups → `_mr.*` | +| `--mr ` | The same kinds of model | As `--model`, but molecular replacement straight away, without the rigid body. `--mode mx` only, not together with `--model` | Hand label only. Other space groups → `_mr.*` | +| PAE file beside a prediction | `AF--F1-predicted_aligned_error_v.json` next to `AF--F1-model_v.cif` (read automatically, no option) | Cuts the prediction into rigid domains for molecular replacement | No (a domain alone can only confirm the answer's group) | +| `-z, --reference ` (`--reference-mtz`) | Merged data of the same crystal form: an MTZ, or a PDB structure-factor mmCIF (`-sf.cif`) | Fixes the space group (in its own setting) and the cell (softly), puts the merge on its axes, resolves the indexing ambiguity, hands over its R-free set, reports CCref on stills. Never a scale anchor | **Yes**: the space group, the axes, the indexing and the free set are the reference's | +| `--reference-column