3 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	5f838bdb26	Broker: refuse a run the writer cannot write before arming the detector ... A writer refusing to overwrite an existing file was reported badly, and the earlier fix only covered half of it. The commit message of a6a715703 states the assumption that broke: "SendStartMessage precedes every std::async in the receiver constructor". That is true of JFJochReceiverFPGA, where the start message goes out from the constructor body, and false of JFJochReceiverLite, where it goes out from MeasurementThread - launched by std::async from the constructor, which waits only on data_analysis_started. On the DECTRIS path the writer's refusal therefore cannot reach services.Start(); it arrives from services.Stop(), i.e. the second try-block of the state machine's MeasurementThread, which never touched start_exception. /wait_till_done answered 200 for a run that never wrote anything. Two changes, in front of the problem and behind it. In front: a PREFLIGHT frame (TCP protocol version 3 -> 4), sent to every writer before the receiver is built and before the detector is armed. It carries the same CBOR start message a START does and asks one question: could this run be written? The writer runs FileWriter::Preflight - the first three statements of the FileWriter constructor, stopping short of the only one that creates anything - and answers with an ACK. Shared code, so a pre-flight that passes cannot be contradicted by the start behind it. A refusal is not fatal: nothing was started, the connection stays usable, and above all no series is left armed waiting for a trigger that will not come. Both ends already reject a frame of another protocol version, so the bump costs no compatibility. The check now covers the data files, not only the master. A stale _data_000001.h5 used to sail through and fail when the file was renamed into place at the end of the run, after the whole dataset had been collected. The writer holding the master file checks for all of them - they share a directory by construction, the master linking them by relative name - so the siblings do not race each other statting files the others are creating. Behind: /wait_till_done maps an Idle state to 200 or 500 by the severity of the end-of-run message, so a collection that lost packets, or whose writer could not write, is no longer indistinguishable from a good one to a script. A cancelled collection is not affected - MeasurementThread checks the cancel before it looks at the completeness of the data, and reports it with Info severity - and a warning is still a 200. start_exception is untouched. The PREFLIGHT payload is built from FillMessage alone, which is what keeps it small: the per-pixel arrays are added by SendStartMessage and are not in it. Measured on a JUNGFRAU 9M, 1.4 kB against 543 kB for the start message. It is deliberately the whole of FillMessage rather than the fields the check reads today - a hand-picked subset would answer for a different set of file names the day the check learns to read one more field, and would do it silently. Three defects found on the way: TCPImagePuller republished a PREFLIGHT as a real start message, announcing to downstream consumers a run that was not going to happen. TCPStreamPusher::run_number was a plain uint64_t written by the control thread and read by every ACK thread; Preflight adds a second writer to it. StreamWriter::CollectImages reprocessed the last frame whenever WaitForImage returned false - run is only tested at the top of the loop - so an abort wrote the last image a second time. Pre-existing; the new branch made it visible. Verified against a running broker on the local configuration: a colliding master file and a colliding data file are both refused by /start, /wait_until_running and /wait_till_done with the writer's own message, and the log shows the receiver was never started for either; a cancelled collection still answers 200. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WmryXe8ASbNi632sUMfsa	2026-08-27 12:20:29 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	64002f1e29	v1.0.0-rc.129 (#36) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files) * jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction * jfjoch_broker: Image pusher statistics are accessible via the REST interface * jfjoch_writer: Supports TCP image socket and for these auto-forking option Reviewed-on: #36 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-03-05 22:13:12 +01:00