179 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	0e67a7b055	Merge branch 'd-dual-reference' into pool-C	2026-09-20 18:45:19 +02:00
leonarski_fandClaude Opus 5	20326af6b6	docs: the rc.172 changelog ... Four lines for 118 commits: the broker GPU-memory leak that cancelled collections, the rotation scaling loop now running to convergence (which moves merged intensities and every decision read off them), the per-frame sweep placement that makes gapped and angle-wrapped CCD/CBF series read correctly, and the two diagnostic files a rotation run now writes without being asked. Left out deliberately, as developer-facing or niche: -A no longer changing what a rotation run decides, --report-resolution, the R-free holohedry keying, the mmCIF twin loop, and every internal refactor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:19 +02:00
leonarski_fandClaude Opus 5	0c91749a41	Battery: let a row accept more than one reference ... A handful of open-arm rows disagree with their deposition on a knife edge that no test available to us settles. They were handled three different ways - silently overridden to our answer, marked unscored, or left failing - and none of the three says what is true: either answer is acceptable as long as the program picks one of them. A manifest row can now list `ref_alternatives`. Each entry replaces the reference fields it names - a space group, a cell, or both - and the row passes if the answer matches any of its references, the deposition included. `ref` keeps the deposited values verbatim in every case. Every alternative must carry `why`: an accepted alternative with no stated reason raises rather than passing, so the mechanism cannot be used to launder a failure. The report keeps these rows visible rather than folding them into the passes: a summary column counting them, their own segment in the verdict bars, and a section naming each row, what we read, what was deposited and the reason both are accepted. Five rows use it. Four are symmetry: a tetragonal row where the refinement test is split and its spread exceeds the effect, and three trigonal rows where we read a higher point group - one where the evidence favours our answer, one where our own twin-immune test favours the deposition, one unresolved in either direction. The fifth is a cell: a real tNCS supercell whose (0,1/2,1/2) sublattice is what was deposited, both being correct descriptions of the same lattice. The documentation frames all of them as open questions, not as errors in a deposition, and states the limit: a merohedral twin at exactly one half and true higher symmetry predict identical intensities, so no test can close them even in principle. `test_score.py` covers the new path: both answers accepted, the other hand of an alternative, a third answer still failing, the cell case, and the missing-justification schema error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:19 +02:00
leonarski_f	281da9d73c	Merge branch 'd-diagnostic-output' into pool-C	2026-09-20 18:45:18 +02:00
leonarski_f	ab0b5e18d3	Merge commit '7620dc647' into d-dual-reference	2026-09-20 18:45:18 +02:00
leonarski_fandClaude Opus 5	4e4de57aae	Rugnux: two diagnostic outputs a person can check the run by eye with ... Written on every rotation run, next to the reflections, with no flag to switch them on: <prefix>_detector.jpg - the mean projection the beam-stop pre-scan built, which is the very image the shadow detection tested, coloured with the viewer's own default map from common/ColorScale.h: white-to-indigo for the counts, grey for the module and chip gaps, coral for the detected shadow, magenta for the rest of the pixel mask. A shadow claimed where the background is unbroken, or a dark patch left uncoloured, is then visible at a glance instead of being a pixel count in the log. The projection is the one the pre-scan already holds, so no frame is read twice; the picture is rendered there and held as JPEG bytes until the canonical pass writes it, because on a two-pass run the pre-scan belongs to the superseded pass. <prefix>_plot.txt - one row per processed image, whitespace-separated with a single '#' legend line over its own columns, so gnuplot plots it as it stands: image, spindle angle, background, resolution, spot count, scale, sigma_M, CC to merge, and whether the image's observations reached the merged data. A quantity nothing measured is written nan, so the columns never shift and no row is ever left out. sigma_M is emitted as the pipeline produces it - a per-image Gaussian rocking width, smoothed in frame order and filled with the run median where a frame could not be fitted - and the documentation says so rather than presenting it as a crystal property. <prefix>_image.dat stays: the in-tree comparison scripts read it and it carries the smoothing wedge, which the new file does not. The docs name _plot.txt as the one to script against. Measured on a 1800-image rotation sweep: 29.6/30.4 s before against 29.8/30.5 s after, and the merged reflections are byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:18 +02:00
leonarski_fandClaude Opus 5	2f453e018f	Record the battery rows where our reduction and a deposition disagree ... Five open-arm rows do not reproduce their deposited space group or cell. Four are scoring rows that disagreed with a published entry with nothing written down about why, which is not a state a reader should have to discover from the manifest. A tetragonal row whose deposited asymmetric unit holds two chains related by the very two-fold the higher group adds, to 0.16 A CA RMSD - coordinate error at its resolution - is overridden to the higher group. Merging there costs 0.0006 in R_meas for 1.75x the multiplicity and correlates better with the deposited model, and an independent reference implementation reads the same point group on our own P1 merge. The refinement comparison is recorded as SPLIT: one purpose-built program makes the higher group better at half the parameters and calls the deposited assignment incorrect, an independent restrained refinement on a symmetry-consistent free set makes the lower group better, and the gap between refinement protocols exceeds the difference being measured. One trigonal row becomes unscored rather than overridden: the evidence points at a higher point group but the deposited copies differ by more than coordinate error, and a purpose-built program settles on a different supergroup whose operators these data do not support. The set runs; it is not scored. Two trigonal rows had already been overridden, by editing the reference itself, with no reason recorded. The deposition is restored verbatim to ref and the change moved into ref_override, which leaves the effective reference and every verdict unchanged. One of them carries strong centric-zone evidence; the other is marked OPEN, because the one test that separates real symmetry from pseudo-symmetry reads against the promotion the row is scored on, and may belong unscored instead. The documentation frames all of these as open questions rather than as errors in the PDB, and states the limit that applies to every one of them: a merohedral twin at a fraction of exactly one half and a crystal that genuinely has the higher symmetry predict identical intensities, so no test available to us can show that an operator holds exactly rather than very nearly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:18 +02:00
leonarski_fandClaude Opus 5	df4be47c84	rugnux: decide the beam-centre axis harmonic in the one direction nothing else repairs ... The beam-centre check indexes a second first pass at the measured centre and, where the two centres return cells related by an integer volume factor, says so and decides nothing. That abstention is right in general - the validation frame count leans to the sub-cell, since a cell twice as long must place every spot twice as accurately to score the same - but it leaves one direction unrecoverable. Where the MEASURED centre holds the LARGER cell, the run starts on the sub-cell and stays there: the second pass re-finds the larger cell at the refined geometry and its supercell guard refuses it for being larger than pass 1's. The run then merges a halved axis at full indexing rate, with good merging statistics and nothing to say it is wrong. Decide that one direction, on the pooled spots rather than on the frames: the excess each lattice puts on itself over its own wrong-spindle null, which is the measurement the failed-pass branch of the same check already adopts a centre on, and the one statistic that does not credit a denser lattice for accidental hits. The opposite direction is deliberately left alone - a second pass is free to drop to the smaller cell and does, on every corpus crystal that reaches this branch that way. Measured on a tetragonal crystal whose header centre is 3.6 px out across the spindle, more than the geometry absorbs: the file's centre gives a halved a axis (43/60 frames, 25.4% of the pooled spots on the lattice) and the measured one the deposited cell (60/60, 43.3%). Before, the run merged the halved cell and the deposited model refined to R-free 0.575; after, it merges the deposited lattice to 1.62 A with R-free 0.254 against the deposition's 0.203. Five other corpus crystals reach the same branch with the measured centre on the SMALLER cell; all five are unchanged (space group, cell and R-free identical), as are two crystals whose halved or doubled axis comes from elsewhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:18 +02:00
leonarski_fandClaude Opus 5	2fdc4d3bdf	Post-refinement: the distance hypothesis is decided at the fit, on the excitation residual ... Whether a sweep determines its detector distance at all was decided by running the canonical pass twice - once at the post-refined distance, once at the header's - and keeping whichever realised the lower held-out residual, the header's on a tie. The first-stage test that sent a run down that arm compared the free and held fits on the POOLED held-out residual, and pooling is what made it uninformative: the positional family cannot see the distance along the degenerate direction, so it dilutes the one family that can, while the unpaired standard error of a heavy-tailed mean of squares is 2-25 % of the mean against in-fit differences of 0.1-1 %. The test therefore said "cannot tell" on four fifths of the fitted sweeps and the arm ran on most of a corpus, buying by prior what it could not measure. The two hypotheses are now compared on the one residual family that can tell them apart. The excitation residual never involves the detector, so it is blind to the distance itself; what it sees is the cell scale, and a held fit at a wrong header distance is forced into a wrong cell scale by the spot positions, which the rocking angles then refuse - measured on a sweep whose header was 1.4 % long, the held fit's held-out excitation residual is seventeen times the free fit's. Where freeing the distance lowers that residual by more than its own standard error the free fit is committed as before; where it does not, the held fit is committed - header distance, refined beam, cell, orientation and axis - and the walk and the commit run held. The positional residual is deliberately not consulted: its in-fit gain along the degenerate direction is the one re-integration erases. The question is asked only while the run is still at the file's distance. A run that has walked off the header has already refuted that hypothesis by re-integrating, and asking it again at every round stalls a walk short of its fixed point, because the excitation standard error at the walk's tail is outlier-dominated (measured: a walk stopped 0.4 % early, seven passes, cell 0.65 % off against 0.26 %, ISa 10.6 against 14.0). So there is no arm, and with it go the two probe passes that measured it and the canonical pass the losing arm used to cost: the decision costs two Ceres solves. On thirteen rotation sweeps covering both verdicts, every decision the arm took by evidence or by its tie rule is reproduced at the fit, except where the excitation family sees what the pooled test could not and the free distance - the better cell against an external reference - is taken instead; merged intensities are unchanged where the verdict is. POSTREFINE_DISTANCE_HELD now means "the committed fit held the header distance", and is cleared where a later geometry walk left it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:18 +02:00
leonarski_f	508fcfb400	Merge branch 'c-screw-undetermined' into pool-C	2026-09-20 18:45:17 +02:00
leonarski_fandClaude Opus 5	7a6df3c24a	Space group: say which axis a screw could not be decided on ... A screw axis whose row the sweep never recorded - it lies in the spindle's blind cone, or outside the resolution range - is not a group the data refused, it is a question nobody asked. The search already offered the whole set in SPACE_GROUP_ALTERNATIVES, but the per-zone screw table that would say it in words is printed for the SELECTED candidate only, and the selected candidate in exactly this case is the one with no screw zones, so the run's account of the open axis was a blank. The search now names the axes on which two SELECTED candidates disagree about whether the row carries screw absences at all, with why the row could not be judged (never recorded / no control class). The report writes the axes as SPACE_GROUP_SCREW_UNDETERMINED= beside SPACE_GROUP_ALTERNATIVES and explains them in prose; the adoption logs a warning naming the axis and the set. An enantiomorphic or origin-ambiguous pair predicts the same absences on every row and is not named here - that ambiguity is the hand, or the origin. Nothing about the decision moves: the group adopted, the alternatives and the written .mtz/.cif/.hkl are exactly as before, because a reflection file cannot hold "maybe a screw". The battery scorer mirrors its existing "hand only" rule: a set differing from its reference only by a screw the run reports as undeterminable, with the reference among the groups it offered, scores unscored/screw_undetermined instead of a sym_screw failure. All three conditions are necessary, so a screw called wrongly where the row WAS measured stays a failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:17 +02:00
leonarski_fandClaude Fable 5.1	40874c2c2d	Space-group search: the twin-immune zone is read on a normalisation the control certifies ... The zone verdict that arbitrates a refused promotion read its zone absolutely, against the centric and acentric Wilson expectations, and an absolute reading is only as good as the normalisation under it. On two refused 622 promotions - a 6/m crystal and a 312 one, whose added operators disagreed at 4.7x and 5.2x the parent's H and merged to an R_meas of 0.34 under the higher group - every class read centric: a 67 A^2 anisotropy, isotropically normalised in bins of 100, spread one shell's expected intensity over a factor of 15 between its directions, and the acentric control read 1.00 against its 0.74. The zone read the same as the control, +0.10 nats per reflection, and +124 nats over 947 reflections rescued a twin law. Two things change. The anisotropy is fitted on the acentric reflections of the shells read (ln I = c + s^T Q s, by least squares) and its deviatoric part taken out of every intensity before anything is normalised; that alone brings the controls of the crystals measured to 0.73-0.77. What no normalisation removes, the control then certifies: an acentric population reads -0.130 nats per reflection when the normalisation is right, a twinned one reads below that, so whatever the control reads above it is the normalisation's - anisotropy, a pseudo-translation, a pseudo-centring, noise all inflate every class towards centric alike - and the zone, normalised the same way, carries the same per reflection; the calibrated evidence has it taken off, and that is what the verdict reads. The two rescued twin laws now read -130 and -78 nats and stay refused, with the law named; the genuine promotions measured read +137 to +580 (a pseudo-centred orthorhombic crystal whose control reads 0.99 still +334); the trigonal and hexagonal partial twins -129 to -420 as before. The report prints the control's excess and the calibrated evidence beside the raw one. The metric-lattice re-ask carried a second copy of the two-arm rule without the zone: a P31 partial twin whose 32 the main rule had refused on the zone was promoted by that ask's Lorentz-filtered arm. It now puts the same verdict to a refusal the other arm would outvote. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:17 +02:00
leonarski_f	4721a91e2f	Merge branch 'c-core' into pool-C	2026-09-20 18:45:17 +02:00
leonarski_fandClaude Fable 5.1	36c24b6e5e	rugnux: a distance the data cannot place stays at the header ... The joint post-refinement is asked twice, with the distance free and with it held at the header. Where freeing it lowers the fit's held-out residual by more than that residual's own standard error, the free fit is committed exactly as before. Where it does not, the fit cannot tell - its own gain need not survive re-integration - so both hypotheses are carried: the canonical pass runs at the free geometry as before and once more at the header distance, and the run keeps the one whose RE-INTEGRATED held-out residual is lower, the header's on a tie. The report says which (POSTREFINE_DISTANCE_HELD), the pass decision names both realised residuals, and the log prints the free fit's formal sd(distance) and its correlation with the cell lengths, from the covariance of the solution. Why: at a detector far enough away that no reflection reaches more than a few degrees of 2theta, a longer distance and a larger cell move every spot the same way to first order (the difference is of order sin^2 theta of the spot's position - 0.4 px rms per per cent over a 2M detector at 820 mm, against 2-3 px at usual distances), so the fit finds a distance/cell pair that fits its own observations a little better than the header, commits it, and the pass re-integrated there asks for the next pair: a walk along the degenerate direction the realised residual never ratifies. XDS leaves the distance out of IDXREF by default and its documentation says to remove it from CORRECT where it drifts on low-resolution data; DIALS fixes the wavelength for the same reason. Here the data are asked instead of a rule. Measured: a sweep at 820 mm reaching 2theta ~ 9 deg no longer walks to 846 mm with the cell 3.3 % too large (pool-B stopped it at 825.6 mm, still +1.1 %); it keeps 820 mm and its cell agrees with the reference at that distance to 0.2 %, R_meas 101 % -> 70 %, two passes fewer. 9yl4 keeps 600 mm (cell 0.5 % -> 0.1 % from the deposition), 7mzt keeps 400 mm (R_meas 169 % -> 145 %, CC1/2 0.961 -> 0.986), 5lzl keeps 689 mm on a tie (cell -0.3 % instead of +0.25 %, ISa 13.9 -> 11.6). Merged output byte-identical on a lysozyme reference sweep at 110 mm (the free distance wins its realised comparison), 8egn and 8pqd (decisive in-fit gains, no second arm) and 8qq7 (the quality guard had already reverted it); one extra canonical pass wherever the two arms are run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:17 +02:00
leonarski_fandClaude Opus 5	8bc4c6e8a6	Merge pool-B into rc172 ... Pooled run B: twin report-only evidence and free-R on the lattice holohedry, information-weighted resolution cutoff with weighted outlier median and ISa rework, sparse-lattice integration and beam-centre arbiter, GPU/CPU collapsed scale upload, geometry walk on the realised residual, --model speed-ups with FFTW structure factors, memory reductions for large-cell sweeps, -A decisions Friedel-merged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:17 +02:00
leonarski_fandClaude Fable 5.1	61afff891b	Rotation scaling: the per-frame scale loop runs to convergence, gauge pinned, plain fit ... The alternating per-frame scaling used to run a fixed three rounds of a Cauchy-reweighted fit against a reference that included half a sweep's rocking-curve tails. Three rounds left a short sweep merged in P1 far from its answer (a 90 deg tetragonal sweep refused its 422 with the P1 scales anti-correlated with the converged ones), and more rounds did not help: the fit had no fixed point. Two things made it walk. The objective is invariant under G -> cG with the reference -> reference/c, so every round moved every scale by a constant factor; and the robust loss, iterated against a reference refitted each round, drops the strong reflections of a frame whose scale is off by a third (ten-sigma residuals) and lets the weak ones carry it further off - measured on a 360 deg sweep the scales shrank 10-30% per round for thirty rounds and the H ratio of a genuine 222 read 21x. Now the loop pins its gauge every iteration (G divided by the precision-weighted typical frame scale G_ref = sum G^3 / sum G^2, one definition shared with the CC1/2 weight), fits the plain weighted least-squares slope with the weights the reference uses and only on the observations the reference is built from (the partiality floor), and stops when the rms |log(G_new/G_old)| over the frames falls below 1e-3. With the same weights on both sides the alternating fit is exact coordinate descent on one objective and cannot climb; measured, the 360 deg sweep settles in 19 rounds and the 90 deg one in 30-50, each pass's partials and fulls loops alike. --scaling-iterations is now the cap (default 100). A loop that reaches it is logged, the report prints SCALING_ITERATIONS and raises SCALING_NOT_CONVERGED, and the correction surfaces run to the same tolerance under their own cap. On the GPU the loop runs one iteration per call so the pin and the step test read the same numbers as on the host; the fulls' reset is split out of ScaleFulls. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:17 +02:00
leonarski_fandClaude Opus 5	c63f9973bd	v1.0.0-rc.172 ... Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-20 18:45:04 +02:00
leonarski_f	6a21d453ba	macOS groundwork: what a static read says will stop an Apple Clang / libc++ build ... None of this has been built on a Mac - there is none yet. It is the list a read-only audit of the viewer/rugnux subtree produced, plus a serial -fsyntax-only pass of every reachable .cpp with clang 16 + libc++ on Linux, which found exactly one error (the first item). - JFJochDatasetInfoChartView: std::vector<fftwf_complex> does not compile with libc++, whose construct_at refuses an array element type (float[2]). Use std::vector<std::complex<float>> and the reinterpret_cast every other FFTW call site already uses. - libcurl: GSSAPI off on Linux, and neither TLS nor GSSAPI on macOS. The viewer never sets CURLOPT_HTTPAUTH, so Negotiate was dead weight that cost a krb5-devel build dependency; on macOS curl's FindGSS refuses the system Heimdal outright, and with Secure Transport gone from curl (8.15) TLS would mean a Homebrew OpenSSL - the only host library a Mac build would need. Linux keeps OpenSSL. CURL_USE_GSSAPI is forced OFF rather than left unset so an existing build tree drops its cached ON. - libjpeg-turbo ExternalProject: CMAKE_SYSTEM_NAME/PROCESSOR were forwarded unconditionally, which puts even a native sub-build into cross-compiling mode, and CMAKE_OSX_ARCHITECTURES / SYSROOT / DEPLOYMENT_TARGET were not forwarded at all. Now the same rule the zlib-ng sub-build follows. - ShadowAccumulatorGPU.cu was added on the JFJOCH_USE_CUDA option (default ON) instead of JFJOCH_CUDA_AVAILABLE like every other .cu, so a machine without nvcc got a CUDA source in a target with no CUDA language. - CMAKE_OSX_DEPLOYMENT_TARGET defaults to 12.0 (overridable). Left unset, CMake takes the build machine's OS version and the .dmg starts nowhere older. - Standard headers that were only arriving transitively (<chrono>, <cmath>, <limits>, <cstring>, <atomic>, <thread>, <string>); newer libc++ releases keep removing such transitive includes. Checked: the seven changed sources pass clang 16 + libc++ -fsyntax-only. The CMake changes are not configured or built. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015eAE2K7i5JGDwgwifiCfuA (cherry picked from commit 5a7282759a)	2026-09-20 18:45:04 +02:00
leonarski_fandClaude Fable 5.1	7547df77de	rugnux: --report-resolution, a second statistics table at a reference range ... Comparing a run with another program's table has meant running rugnux AT that program's resolution range (--scaling-high-resolution), which is a different run: the range moves the cut, the space-group decision and everything after them, so the comparison buys itself a different answer. --report-resolution <dmin>[,<dmax>] instead leaves the run alone and adds a second table to section 3 of the report - the REFRES_* keys and a shell table - binned from the same merged reflections over the range given, with the completeness denominator enumerated over that range and the shells in equal steps of 1/d^2 so they read row for row against a CORRECT.LP at the same range. Report-only: the merged files and every decision are byte-identical with and without it. The table holds only what the run kept. Where the reference range is finer than the run's own limit, the shells past it are printed as not merged (with their possible count) rather than as zeros, REFRES_SHELLS_PAST_LIMIT counts them so a consumer can tell "not merged" from a measured zero, REFRES_ COMPLETENESS counts their reflections as missing, and the other overall numbers are over the shells the run reached; nothing is read from the observations the run judged to carry no signal. REFRES_ISA is the error model refitted on the reflections of the table alone, in XDS's convention (rotation only; the stills model is fitted over the whole range already). On the rotation path the statistics block of MergeAndStats becomes a lambda over a shell grid, called once for the run's own grid and once for the reference one; the reference call floors every observation-level count at the cut by group d, the rule the erase applied. The stills MergeStats takes a declared range, whose bounds are the grid's whether or not any reflection reaches them. Both --mode mx and --mode scale report it, the viewer's command line echoes it, and the docs describe the keys. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-20 18:45:03 +02:00
leonarski_fandClaude Fable 5.1	060a89cd7d	CUDA: allocation streams are handed back, not leaked; a handled pool failure leaves no error ... A long-running broker began cancelling every data collection with Device decoding failed (CUDA (GPU) error (out of memory)), falling back to host decompression CUDA (GPU) error (out of memory) while nvidia-smi showed the cards less than a fifth full. Two defects, both in CUDAMemHelpers.h and both from the pooled allocator (cudaMallocAsync) that came with rc.162. The leak. cuda_allocation_stream() kept one stream per (thread, device) in a thread_local map of raw cudaStream_t and never destroyed them. That was written against rugnux, where the worker threads live as long as the process. The broker starts fresh std::async threads for every data collection - 16 or 64 of them - so every collection left that many streams behind. Measured: 0.56 MB of device memory per leaked stream, linear to 4928 streams, never returned, with nothing on the host side growing. The RAII wrapper (CudaStream) was there but not used at this site, and using it as-is - a stream destroyed when its thread exits - would not have been safe: ShadowFinder builds its GPU accumulator on a throw-away std::async thread and frees it from another thread long after, and that free is ordered on the allocating thread's stream. So the streams are still never destroyed, but a thread now only borrows one: CudaStream objects live in a process-wide per-device idle list, a thread takes one on first use and hands it back when it exits. Their number is bounded by the threads that were ever alive at once instead of by the threads ever started. The list itself is deliberately leaked, so that nothing calls into CUDA during static destruction. Replaying the broker's pattern against the real header, 60 collections of 64 threads: before 3840 streams, 260 -> 2424 MB of device memory after 64 streams, 260 -> 358 MB The stale error. Every helper here throws a named message, yet the log carried the raw CUDA string, so the failure came through a cuda_err() and not from an allocation. CudaDevicePtr falls back to cudaMalloc when cudaMallocAsync fails, silently - but the failed call stays behind as the thread's last error, and the cudaGetLastError() that follows the next kernel launch reports it. The buffers were all allocated; the frame was lost anyway, once on the device-decode route (caught, hence the warning) and once more on the host fallback (fatal). The pooled attempt failing, and a stream that cannot be created, are both handled by falling back, so both now clear the error they leave. What finite resource the production cards ran out of at under 4 GB used was not established - no cap on the number of streams was found up to 4928 on the card this was measured on. The leak is the only thing on this path that grows with uptime. tests/CUDAMemHelpersTest.cpp: later threads end up on the same stream, concurrent threads on different ones, a buffer is freed cleanly after its allocating thread has exited (and another has borrowed its stream), and a pool that cannot serve a request leaves no error behind - the last by capping a memory pool at 4 MB so that the pooled attempt fails and the fallback succeeds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-20 18:45:03 +02:00
leonarski_fandClaude Opus 5	bb0648ae87	tools/battery: run every variant of a set back to back ... The battery is bound by reading images from disk, so instead of one battery run per setting each set now runs in all its variants in a row while its images are still in the page cache: open arm bare, then model (rugnux --model <deposited coordinates>) XDS arms bare, then xds (XDS's resolution range forced, -A where XDS was anomalous) bare (plain rugnux <input>) runs first on every arm, so its time always carries the set's disk read whichever variants are selected; each row records first_read and the report's timing table says which variant's times are cold. --variants runs a subset; --unforced is gone (--variants bare). Rows, work dirs (work/<arm>/<set>/<variant>/), compare and the report are per variant; compare pairs (arm, set, variant) on the variants both runs have on an arm, and a schema-1 run is read as one variant. The model variant takes the coordinates and published R-free from model_check's RCSB cache (site key pdb_cache) and records rugnux's R-free/R-work and the ratio; the REFMAC check is now opt-in (--model-check) and its keys moved to refmac_*. results_schema 2. model_sweep.py is retired: the model variant replaces it (its --spot/--scaling-low-resolution 50 were rugnux's defaults, so the command is the same). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-20 18:45:03 +02:00
leonarski_fandClaude Opus 5	4b0c33a473	rugnux: the geometry walk continues only while re-integration realises a gain ... The rotation geometry walk kept re-integrating while the post-refinement at the adopted geometry still committed a move, up to four rounds. It never ended on its own: every walking run hit the cap, and with the cap lifted to twelve every one hit twelve. Each round's fit predicted a held-out gain the re-integration at its geometry then did not realise - the realised residual stayed flat from the first round, or passed a minimum and rose again while the distance crept on along the distance/cell-scale degeneracy - so the cap decided where on the drift the run stopped. A round is now kept only when the held-out residual measured by the pass re-integrated at its geometry falls below the best round's by more than the standard error of the difference. The first round that does not ends the walk, and the run goes back to the best round (one more pass, since the canonical files were written by the last). The walk also stops when re-indexing pushes the fitted cell back on the same length in the same direction on two rounds running: the fit keeps asking for a cell nothing ratifies. The cap is 8; a walk still paying at the cap keeps its last (= best) round, sets geometry_not_converged, and the report flags GEOMETRY_NOT_CONVERGED. Either way the kept round is judged against the header-geometry pass by RefinedPassIsWorse, as before. PASS_DECISION lists distance and realised residual per round and why the walk ended. Runs whose post-refinement does not commit a move of more than one step do not walk and are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-20 18:45:03 +02:00
leonarski_fandClaude Opus 5	3d62388467	Twin-immune zone evidence for the operators a promotion adds (report-only) ... Under a twin law T a reflection whose twin mate is itself (up to the true group and Friedel) is untouched at every twin fraction. For each index-2 subgroup H of the adopted point group, those are the reflections centric in the group and acentric in H: centric if the operators the group adds over H are real, acentric if they are a twin law or a pseudo-symmetry - the one intensity statistic that still separates the two at fraction 0.5, where every operator statistic reads "real". Read on the P1 cross-check merge: epsilon-1 reflections in shells with <I/sigma> >= 5 (noise inflates every class towards centric), each class normalised against its own mean in bins of ~100 reflections and within the two phase classes of a detected pseudo-translation, Wilson outliers above E^2 = 20 dropped. Reported per subgroup: the added operators, n, <|E^2-1|> +- SE read absolutely against 0.968 / 0.736, and the centric-over- acentric Wilson log-likelihood in nats with both densities convolved with each reflection's error (flooring E^2 at its sigma instead read a genuine 1.2 A lysozyme zone as acentric), with the acentric control beside it. Log, report prose and TWIN_ZONE_n keys. Nothing reads it back; no decision changes. On the reference sets: 6toc P4222, all three subgroups centric (1.04-1.07, +253 to +534 nats); 6iu9 P3121 over P31 acentric (0.789, -294 nats); 5j23 R32 over R3 acentric (0.805, -304 nats, tNCS-class normalised); lysozyme P41212 centric (0.90, +476 to +1297); a P21 myoglobin centric (0.930, +135). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-20 18:45:03 +02:00
leonarski_fandClaude Opus 5	54fcb3cbcb	R-free flags keyed on the lattice holohedry ... The free-set hash was keyed on the Laue ASU of the merging group, so a twin law - a lattice symmetry the crystal lacks - put nearly every free reflection's twin mate in the working set (measured: 97-98% of the free-touching twin pairs mixed, on the main P3121 output of 6iu9 and on every law of the 6toc P1 cross-check), and each file of one crystal carried a different free set (P1 vs merged agreed on 88-92% of reflections). With the cell given, the key is now the reflection's orbit, Friedel mate included, under the lattice holohedry: the metric point group of the cell (gemmi Le Page two-folds, 3 deg obliquity, lattice of the cell's own basis vectors so every file gets the same group; this contains the merging group). Twin mates share a flag, and the merged MTZ/mmCIF, the P1 cross-check and any subgroup re-merge carry one free set (nested where the small-data floor lifts the fraction of one file more than another). This is phenix.refine's default (use_lattice_symmetry). Where the cell does not carry the merging group, the merging group's key is used as before. The small-data floor still counts reflections of the merging group, so the free fraction is unchanged. Reference free sets are untouched. On 6toc, 6iu9, 5j23: mixed twin pairs 0 for every law; P1 and merged flags agree on 100% (6iu9, 5j23) and nested on 6toc; free fraction 0.050-0.051 as before (6toc merged 0.086, floor unchanged). Intensities and space groups identical to rc171. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-20 18:45:03 +02:00
leonarski_fandClaude Opus 5	99123f33df	Twinning: normalised L-test, and one verdict read in every Laue class ... The L-test now divides each intensity by its resolution-shell mean before forming pairs, over the shells whose <I/sigma> reaches 1 (the floor the second moment already used). Two index steps are not the same resolution, and on a small cell with a steep fall-off the raw pairs read <|L|> up to 0.08 high: an untwinned crystal 0.568 and a partial twin 0.427 where phenix.xtriage reads 0.486 and 0.360. Normalised, the four reference merges (6toc, 6iu9, 5j23, a lysozyme) agree with xtriage within 0.02 (0.487, 0.376, 0.374, 0.480). Selection stays by shell, never by the reflection's own I/sigma, which biases <|L|> down. Equal-count shells were tried for the second moment and widen its gap to xtriage, so the shells are kept. The L-test is no longer switched off in holohedral Laue classes. Merging I(h) with I(Th) under a false operator gives (I(h)+I(Th))/2 for every twin fraction, the perfect-twin distribution, so <|L|> < 0.42 there is reported as an adopted operator averaging unequal intensities (promotion suspect, or a twin law absorbed into the point group) - a warning, not a veto. Over 42 holohedral merges of the corpus the genuine ones read 0.436-0.513, the over-promoted H3 twin 0.374, and one ~490 A-axis crystal 0.365. The verdict is one line (TwinningVerdictLine) used by the stats text, the report prose, the summary row, the warning and the viewer; a new TWINNING_VERDICT key names it. The twin fraction comes from the statistic that carries the verdict (the L-test unless the call rests on the second moment alone), is not quoted from a second moment under a detected pseudo-translation, and is not quoted at all on a merge under a suspect operator. The pre-search numbers (measured on the P1 search merge) are user-visible, are measured with that merge's own pseudo-translation declared, and leave out the reflections the lattice centring extinguishes - an R lattice in its hexagonal cell otherwise pairs present with absent reflections (5j23: 0.633 with them, 0.40 without). Report-only: no space-group decision, merge or intensity changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-20 18:45:03 +02:00
leonarski_fandClaude Opus 5	e8fd659e21	Retire the scattered battery scripts in favour of tools/battery ... - rugnux_vs_xds.py: deleted. The inhouse and private arms of tools/battery run the same comparison against CORRECT.LP, from a fixed manifest instead of a directory scan, with persisted runs and a standard report. - lyso_test/anomalous_scoreboard.sh: deleted. A one-dataset SHELXC/ANODE scoreboard with hard-coded personal paths, superseded by rugnux_anomalous.py. - rugnux_vs_pdb.py -> tools/battery/model_sweep.py. The rugnux --model sweep is not superseded, but it now takes its inputs from the open arm's manifest and the battery site config, so both tools process the same files. - docs/TESTS.md and rugnux_stills_ab.py point at the battery. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-20 18:45:02 +02:00
leonarski_f	30b6800289	v1.0.0-rc.171 (#81) ... * Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-17 14:42:52 +02:00
leonarski_f	cb5a2f032a	v1.0.0-rc.170 (#80) ... * Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked. * rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results. * `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry. * `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel. Reviewed-on: #80 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-16 18:17:46 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	77bc0cfe52	v1.0.0-rc.168 (#78) ... * rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`. Reviewed-on: #78 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-10 13:51:16 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_f	61c603b274	v1.0.0-rc.163 (#73) ... * Packaging: each package installs its license notices under a directory of its own - `share/doc/jfjoch_broker`, `share/doc/jfjoch_writer`, `share/doc/jfjoch_viewer`, `share/doc/jfjoch_driver_dkms` - instead of all of them into the shared `share/doc/jfjoch`, so `jfjoch` and `jfjoch-writer` can be upgraded one at a time instead of only together. Reviewed-on: #73 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 13:38:29 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	6136f858af	v1.0.0-rc.154 (#64) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability) Reviewed-on: #64 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-25 18:12:00 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	c49bd2ac3b	v1.0.0-rc.152 (#62) ... * jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5) * jfjoch_viewer: Adjust Q bounds for azimuthal integration * jfjoch_azint: Add tool to do quick azimuthal integration Reviewed-on: #62	2026-06-17 20:36:24 +02:00
leonarski_f	ef52dac2ee	v1.0.0-rc.151 (#61) ... * jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname Reviewed-on: #61 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-16 14:13:29 +02:00
leonarski_f	90e804acd7	v1.0.0-rc.150 (#60) ... * jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin. * jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins. Reviewed-on: #60	2026-06-15 20:24:15 +02:00
leonarski_f	ea575f790a	v1.0.0-rc.149 (#59) ... * XDS plugin: Fix HDF5 mutex to run on multiple processors Reviewed-on: #59	2026-06-13 21:27:41 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00