abb61e27081e1972fdfecf45cc4c8d20dcc0af4d
1144
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  leonarski_fandClaude Opus 5
|
abb61e2708 |
Unmerged export: carry the sort key with the part here too
SumRockingEvents assembles rocking events for the unmerged MTZ through the same indirect comparator the anisotropy diagnostic had, on a larger array - it has neither the scaled-image test nor the usable() filter in front of it. It is latent, since --export-unmerged is off by default, but it is the same defect and it is a mechanical fix. Same treatment: the key travels with the part, plus an exact reserve. p_unmerged.mtz, p_unmerged_partials.mtz and p.hkl are md5-identical across eight runs on two crystals. With the export on it is worth about 0.9 s on a seventeen-million-partial run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
0e6ece3c38 |
Anisotropy: assemble only the reflections the floor actually reads
BuildObservationCells strides whole ASU clusters down to MAX_CLUSTERS = 20000 and keeps 1.4% of what it is handed - so the assembly before it sorted and summed every integrated partial in the sweep to throw 98.6% of the result away on the next line. The comment there says the striding "costs nothing and bounds the work"; it bounded it after the expensive part. Sample first instead: keep a partial iff a splitmix64 hash of its ASU key falls in the low one part in 2^n. Whole ASU clusters are mandatory rather than convenient - the cluster-robust sandwich sums one score per cluster, and the forbidden-direction signal IS the contrast between mates, so a raw-hkl sample would collapse that onto the independent covariance and inflate the floor. The hash rather than the key matters too: HKLKey::pack puts h in the low bits, so masking the key directly would select lattice planes instead of a sample. The rate comes from the partial count, not from merged.size(): the merged list is resolution-cut, and taking the rate from it made the first version never fire at all on the crystal it was written for. One part in 2^n of the partials is one unique reflection in 2^n, so the count already computed for the reserve is the right one. DELTA_B is unchanged by construction, not by luck. AnalyzeAnisotropy touches the unmerged observations in exactly two places - an empty test and MeasureSystematicFloor - so the tensor, the shape verdict and the directional limits come from the merged list alone. Over the battery: 0 verdict flips of 38; DELTA_B, DELTA_B_LINEAR, SHAPE, FREE_DIRECTIONS and the D_MIN keys differ on none of the 38, byte-identical rather than within noise; every reported key identical on 26; the mmCIF diff is exactly _reflns.jfjoch_aniso_floor and _significance; no caution appears or disappears anywhere. The spread those two keys move within was measured rather than assumed, and re-running cannot measure it because the code is deterministic. The shipped code already samples - by striding cluster order - so moving the phase of its own stride gives the sampling spread it has always carried. Against that control the change moves the floor less: ratio range 0.803-1.481 against 0.497-1.343, geomean 1.031 against 0.941. The gate therefore did not get easier to trip - the floor's geomean moves +3.1% here against -5.9% for the phase control, the crystal nearest the gate is not sampled at all and is bit-identical, and the two sampled crystals nearest the gate both move away from it. N_OBSERVATIONS goes up, not down, because a smaller pool lands the downstream integer stride slightly above the cluster cap. The ASU reduction is 14-45 ns a call in isolation, which nearly sank the design; in place it is free, because it overlaps the memory stalls of the Reflection stream it walks. The 24 crystals below the sampling cap run the shipped algorithm plus the reduction and cost +1.79 s in total, median +0.04 s. The block that introduced this measures 8.07 s before and 1.97 s after, which is 5.74 s off the largest crystal (-12.3%) and -11.9 s of anisotropy time over the battery. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
62f0f6472d |
Changelog: say what the anisotropy sort actually cost
It was about eleven seconds on the largest sweep, not minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
e332aaf28c |
Space-group search: read the presence cut on counting significance
present_i_over_sigma is meant to say "this reflection is really there".
The merged I/sigma is not that quantity. The merge carries the error
model's intensity-proportional term, sigma^2 = a*sigma0^2 + (b*I)^2, so
I/sigma saturates - at ISa*sqrt(n) for a reflection observed n times, and
at ISa exactly for one observed once. Above that knee it stops rising
with the intensity.
Measured over the battery's 76 search merges: the median I/sigma of the
top E^2 decile sits at 0.81 of ISa, and on the weakest merge the decile
profile reads 1.58 1.61 1.63 1.64 1.64 1.65 1.65 against an ISa of 1.70 -
flat to three digits, log-log slope against E^2 of 0.008. Reflections an
order of magnitude apart in real intensity read the same number. So one
constant demands between 2.16 and 11.91 in counting significance
depending on the crystal, and the fraction of a merge clearing it runs
from 3.1% to 98.8%.
Convert it once instead. For a reflection observed once
sigma_counting^2 = sigma^2 - (b*I)^2, so I/sigma_counting >= T is exactly
t >= T/sqrt(1+(T/ISa)^2). That cut lies strictly below ISa for every ISa,
so it is always reachable by the reflection the ceiling binds hardest,
and it is within 1% of T on any merge with ISa >= 21, so a healthy merge
is left where it was. Multiplicity is taken as 1 deliberately rather than
estimated, for the same reason.
The quantile fallback goes with it. Its comment claimed to be inert on
the battery; it fires on 4 of 76 arms, and on one of them it relaxes the
cut BELOW the nominal value because that merge is bimodal. A cap on a
statistic is not a substitute for measuring the right statistic.
Two comments in the tree still described a merged sigma floored at b|I|,
removed in
|
||
|
leonarski_fandClaude Opus 5
|
278f8e3d73 |
Anisotropy: carry the sort key with the part, as the merge's ingest does
ScaledObservations assembles every integrated partial into rocking events before the tensor is fitted, and sorted them through a pointer. On the largest sweep that is 29.1 million parts spread over about 2.5 GB of 88-byte records against a 32 MB L3, and the comparator dereferenced one of those pointers on every one of roughly 660 million comparisons. Single-threaded, once per run, at the end of a run whose image loop uses sixteen threads and a GPU - and for a diagnostic that corrects no intensity and removes no reflection. It was 8% of one run, 12%, 15% and 16% of others, and 19% of the largest. The fix is the idiom this repository already uses for this exact sort: RotationScaleMerge's ingest sorts the same partials on the same key through a narrow record and its header says why. ScaledObservations was the one place that read the key back through the pointer instead. The key now travels with the part. Twelve lines, no bit-packing, and a reserve of the upper bound so the growth chain stops copying a few hundred megabytes on a large sweep. Ties are real and had to be handled rather than argued away: the predictor emits both Ewald intersections and near the blind region both land on the same frame, so one crystal has 7424 tied adjacent pairs, 3778 of them in reverse arrival order. The comparator keeps the same four values in the same order, so introsort - whose every branch is decided by comparator outcomes at positions and never by element size - makes the same comparisons and the same swaps and leaves the same permutation. Checked rather than asserted: a hash of the sorted permutation of all 29.1 million parts, a hash of the returned vector's raw bytes, and the md5 of the whole .cif all match on eight crystals, each first run twice on the baseline as a determinism control. The sort is 1.7-2.1x faster and the function 1.5-2.0x, which is 6.7 s of single-threaded critical path over those eight crystals and 10.8% of total wall clock over six timed warm and interleaved. A wider element costs 7% in the grouping pass that follows. A 24-byte variant that indexes back instead was built and is marginally faster to sort and slower overall, so the plainest version is also the fastest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
c4f4f7e28f |
Stills: run the background ring to 14 px instead of 12
The reported sigma carries the background ring mean's own error as (dI/dbkg)^2 * bkg/n_ring, and the stills ring holds only 244 pixels at r3 = 12 against the rotation default's 408 - so that term is a larger share of the variance here than anywhere else in the program. r3 = 14 takes the ring to 416. What normally stops a ring being widened is crowding, since a neighbour inside the annulus is excluded from it. Stills barely crowd: neighbour occlusion is 0.000% at both radii on six of the seven datasets scored and 0.721% -> 0.678% on the densest, against most reflections having a neighbour within 13 px on rotation. Rotation is left alone at 4,6,13, where the ring already wins by a factor of several. Measured with the new stills harness over six datasets and 72 shells, 10000 images each, resolution range pinned so both arms share shell edges, against a control floor that is bit-identical in every column: <I/sigma> +1.23% median over the 56 signal-bearing shells (44/56) R_meas -0.37 points (39/56) CC1/2 flat (29/56) obs flat R_meas is the number to read here. <I/sigma> improving when the ring is widened is half mechanical - a better background estimate lowers sigma by construction - whereas R_meas has no sigma in it, so it moves only if the intensities themselves got better. A prediction from ring pixel counts alone, made before the run, was 1.6-2.7%; landing at 1.2% with the profile-fit weighting in the way is the mechanism behaving as expected. The larger and more consistent effect is the opposite of the risk this was checked against: reflections discarded for want of a background ring fall from 2.53-2.88% to 1.87-2.25% on all seven datasets, because a wider ring reaches past the module gap that starved the narrow one. Against it: CC1/2 and observation count are flat, and the gain reverses below <I/sigma> of about 4 - median -0.4% and -0.6% in the two weakest bands. Roughly a fifth of that is background curvature, which --background-radial recovers; the rest is not explained. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
39a0b067a4 |
Add a stills validation harness, the analogue of rugnux_vs_xds.py
rugnux_vs_xds.py gates every rotation change in this project. Nothing gates a stills change, so until now one could not be validated at all - which is why the background-ring width below had to be measured before it could be argued about. The rotation harness does not transfer. A stills dataset is one file of 100k-500k images with no per-crystal reference, the space group is known in advance and never in question, and what a change moves is the merge. So this asks a different question - did this make stills data better or worse - by running rugnux over the same fixed image subset under two or more argument sets and tabulating the merge per resolution shell. Three things it does on purpose. It pins the resolution range per dataset so both arms share shell edges: without that the automatic cut-off moves between arms and the columns are not the same shells, which happened on two of seven datasets. It fixes and prints the image subset, contiguous rather than strided because a stride turns one sequential read into ten thousand seeks. And --repeat measures the control floor, which on six of seven datasets is bit-identical in every column of every shell. It reads the stable interfaces - the report's KEY= value lines and the mmCIF _reflns block and _reflns_shell. loop. The one console-log number, how many predicted reflections lost their background ring, is labelled as such; it is there because it is the only place the price of a wider ring is counted. A CrystFEL stream can be scored alongside as an optional reference, keyed on the global image serial number - Event: //N restarts at zero in every file and silently multiplies the count. No dataset list is shipped. A dataset directory name identifies a sample, and so does a run label or a pump-probe parameter in a filename, so the table committed here is a template carrying the reasoning and no paths. A real one belongs in a config kept outside the repository, which is why --config is required and has no default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
d987bd6178 |
Space-group search: record what the E^2 shell count does on a small merge
The shell count is order.size() / 100 in integer arithmetic, so a merge of fewer than two hundred reflections gets a single shell and E^2 becomes I divided by one global mean. That is harmless for the thing E^2 was introduced for - the operator correlation is a Pearson coefficient and is invariant to a common scale, so it lands on exactly the value raw I would have given. It is not harmless for the overlap cap, which then rejects reflections by their intensity against the whole merge rather than against their own resolution shell, and so cuts the low-resolution end and spares the high. No behaviour change; a merge that small has already failed for other reasons. This is here so the next reader does not have to rediscover that the normalisation quietly switches off at the bottom of the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
00c27584cb |
Spot width: stop doing obviously redundant work in the pixel loops
None of this is worth its speedup - the routine is 0.5-0.8% of the pre-scan's cycles on two independent profiles, and the whole change is 0.9 ms per data set. It is worth having because the code was doing work that has no reason to exist. The isolation grid was a vector<vector<uint32_t>> over ISOLATION_PX cells: on a 4148x4362 detector that is 23244 std::vector objects constructed, heap-allocated and destroyed per image for a structure that is read once. It is now a counting sort - one offset array, one index array. This was the largest single item and it is not a pixel loop. The encircled-flux curve added every pixel into every bin at or beyond its own radius, about 7.5 adds per pixel, which sums the same aperture R_MAX/2 times over. Each pixel now lands in the one bin its radius falls in and the curve is the running total. The bins are double where the running totals were float, so the result is more accurate, not merely faster. No square roots remain in the pixel loops. The isolation and beam gates compare squared distances, and the radial bin is a table lookup over the 197 squared distances the aperture can produce. That is the same bin, not an approximation: floor(sqrt(floor(y))) == floor(sqrt(y)) for every real y >= 0, because k*k is an integer, so the table recovers ceil(sqrt(rc2)) exactly once the perfect-square case is separated out. The disks are also walked as disks rather than as their bounding boxes - constexpr per-row half-widths - which takes the background pass from 1681 to 1257 pixel visits per spot. Verified over the rotation battery: r80 identical to nine significant figures on 37 of 37 measurable crystals, the 38th unmeasurable in both arms, and the chosen r1 identical on 38 of 38. The pass feeds nothing but r1 into the run, so merged output is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
1bb4cecbd1 |
Spot width: measure on a growing sample and stop once it settles
The pre-scan measured the spot width on all 60 frames of its sample every
time. The radius that measurement feeds is a three-way choice - R1ForWidth
moves only where 2*r80 crosses 4.5 and 5.5 - and 29 of 37 measurable
crystals sit more than 0.29 px clear of both switches, so most of the
sample is spent confirming a bucket the first few frames already picked.
Measure instead on a growing share of it - every eighth frame, then every
fourth, every second, all of it - and stop at the first tier whose answer
has settled: within 0.40 px of what the smaller sample said AND 0.25 px
clear of both switches. Both conditions are load-bearing. Clearance alone
loses the crystal whose r80 is 0.07 px from a switch, because its small
samples read across it; the step test alone lets a sample settle on a
switch and stop there. Both bounds sit interior to a two-dimensional
region that is right on 38 of 38, and every candidate was re-scored at all
eight phase offsets of the tier ladder.
The tiers are sized in worker-rounds rather than frames: a first tier of
five frames occupies eight workers as long as one of eight does. A {12,4,
2,1} ladder measures 11% fewer frames than {8,4,2,1} and is 11% slower.
Over the rotation battery this reads 1215 of 2280 frames - 21 crystals
stop at 15, 4 at 30, 13 still run all 60 - and the chosen r1 is identical
on 38 of 38, as is the beam-stop mask the reordered loop also touches.
The pass feeds nothing else into the run, so every merged intensity is
unchanged by construction and no battery is required. Cost over the
battery goes 15.6 s to 10.1 s, median 0.18 s to 0.14 s per data set.
Note for anyone optimising this further: the measurement itself is 0.54%
of the pre-scan's cycles. The cost is the decode, preprocessing and spot
finding each frame needs before it - 64% of the pass - because the
beam-stop projection decodes on the GPU and never materialises the frame
on the host. Cutting frames is the only lever short of harvesting the
width from the two-pass run's first GPU pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P
|
||
|
leonarski_fandClaude Opus 5
|
7e3ab8150d |
Integration: widen the radius for the final pass only
|
||
|
leonarski_fandClaude Opus 5
|
64957d5a5f |
Space-group search: record what min_pairs_per_operator is actually worth
Comment only; the constant stays at 20.
|
||
|
leonarski_fandClaude Opus 5
|
f6bfa54624 |
Space-group search: score operators on normalised intensities
Symmetry operators were scored by Pearson correlation on raw merged intensities. Both members of a symmetry pair sit at the same |s|, so the resolution fall-off is variance the two arms share exactly, and it inflates the correlation of true and false operators alike. The clearest demonstration is the test this commit adds: a synthetic data set with no symmetry at all - a radial fall-off times an independent per- reflection factor - is assigned point group 432 by the shipped code, with all 23 rotations confirmed at 0.632 to 0.656. The existing suite passes identically before and after, because nothing covered this. The new case fails 64 of its 100 assertions on the old scoring and passes on the new. On real data the same effect had the gate leaking: on one cubic crystal three pseudo-symmetric operators scored 0.506 to 0.517, above the 0.5 bound, so they were confirmed and 432 had to be refused further downstream by the twin-law and systematic-b guards. Scored on E-squared they read 0.283 to 0.298 and exactly the eleven genuine rotations of 23 are confirmed. The normalisation has to be over the reflections the correlation actually pairs. Reusing the existing normalised array is worse than doing nothing: it is normalised over the set the absence tests use, whose surviving fraction is itself resolution-dependent, and the coupling to the resolution cut rises from 0.086 to 0.262 against a raw baseline of 0.086. Normalised over the paired set it falls to 0.023. The twin-law H statistic keeps its own vectors on raw intensities. It shares the pair arrays with the correlation, and normalising in place moves it by up to 12% against a bound whose window is 5.5% wide. Verified rather than assumed: two instrumented binaries print the same H to twelve significant figures while the correlation differs. min_operator_cc goes 0.5 to 0.30. Normalised correlations run lower, and the observed window on rugnux's own search merges is 0.298 to 0.351; 0.35 is too high, because one crystal's weakest genuine operator reads 0.351. The headroom between a crystal's weakest true operator and its own measured false -operator floor widens on 14 of 14 crystals, median 0.430 to 0.619, and the worst operational margin goes from 0.031 to 0.051. Battery, twice, against a baseline reproducible to zero: the space group is identical on all 38 crystals, per-shell merging is 0 better and 0 worse across all 380 shells, and the merged mmCIF is byte-identical on 38 of 38 - on the 12 crystals whose integration radius now adapts as well as the 25 that do not. The arm is live rather than inert: all 313 operator correlations move while every pair count and every H value stays bit-identical, and on one cubic crystal three operators that raw intensities confirmed are rejected, with the space group unchanged. Following Padilla and Yeates (2003) Acta Cryst. D59, 1124-1130 for why a resolution-normalised statistic is the right one for a symmetry test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
4e6eb18d93 |
Integration: fall back to the fixed radius on a pattern too dense for it
|
||
|
leonarski_fandClaude Opus 5
|
0da593b32a |
Anisotropy: say what was measured, and stop crying wolf
Four reporting changes and a restored changelog line. The gate boundaries are unchanged - see below - and no merged number moves: .mtz and .hkl are byte-identical on two crystals, and REPORT_VERSION stays 3 because everything here is an added key or prose. A directional diffraction limit that is really the edge of the measured data was printed as if it were the crystal's. One crystal reported 1.09 / 0.99 / 0.99 A against a 0.994 A cutoff, so two of its three limits were the cutoff. A censored limit now prints with a "<" and sets ANISOTROPY_D_MIN_CENSORED. It fires on 4 of 28 crystals, and on two of those all three limits are censored, which means their reported directional spread was binning noise. The caution about a too-high symmetry assignment fired on every crystal with one deviatoric degree of freedom - every tetragonal, trigonal and hexagonal case, 17 of 38. A caution that fires on 45% of runs is noise. It is now conditional on the signature that actually indicates it, a large symmetry-forbidden signal with a gate that established nothing, and fires on 2 of 38: the crystal already on record as space-group unstable, and one other. The forbidden-direction z is now reported so the reader can see it; its battery median is 6.5. The verdict line quoted deltaB_linear while the headline reported deltaB, and the two differ - deltaB_linear is the larger on 12 of 31 crystals. It now writes out the arithmetic, says which number the gate tests, and says which to act on. The observation count the floor was measured on is reported, and the mmCIF verdict item carries its vocabulary like its neighbours. On the boundaries: they were calibrated against XDS-derived unmerged data, while production measures the floor from rugnux's own observations, and those disagree by 0.12x to 16.2x with 7 of 29 verdicts flipping. Re-measured on production input - by merging each cubic crystal in proper subgroups of its own Laue class, where the true anisotropy is exactly zero and every deltaB that comes back is manufactured - the false-positive rate is 24% at 2.0, 10% at 3.5 and 5% at 5.0. Each sits inside the published band read as its worst-case figure rather than its average. 3.5 maximises power minus false-positive rate across the grid; moving to 4.0 costs a real detection and reduces the false-positive rate by nothing. So the boundaries stay, and the "strong" band at 5.0 is a 5% statement on real data rather than the 0.5% its derivation suggested. The changelog entry for the diagnostic itself was committed with only its first line, having been extracted by a script that took one line of a four-line bullet. Restored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
48008e1447 |
rugnux: set the integration radius from the crystal's own spot width
On rotation data the signal radius is now r1 = clamp(round(2*r80), 4, 6),
where r80 is the 80% encircled-flux radius of the crystal's own spots. The
background ring keeps its area (r3 = sqrt(r2^2 + 133)), so r1 = 4 is the
shipped default bit for bit and 26 of the 38 battery crystals come out
byte-identical.
The width had to be measured somewhere new. rugnux already has one -
shell_sigma2[].tan - but it is a second moment taken inside the r1 disk it
would be setting, and it saturates at r1/2, so feeding it back measures the
cap and not the crystal. SpotWidth instead measures encircled flux over an
aperture fixed for the whole file (14 px, normalised at 8), in the pre-scan,
from spots the finder already produces on the frames the beam-stop projection
already reads. It touches no integrator output and runs before the first
integration pass, so there is no loop, it costs no extra frame reads, and
both passes - including the space-group search, which runs in pass 1 - see
the same radius. A default run pays a median 1.9 s.
k = 2 is not fitted. For a Gaussian r80 = 1.794 sigma, so r1 = 2*r80 is
3.59 sigma, where the truncated second moment recovers 0.990 of sigma^2. The
new test checks the estimator returns 1.794 sigma on a known Gaussian.
Battery, 38 crystals, both arms run twice: the space group is identical on
all 38 and 35 agree with the reference in both arms. Per shell on the 12
crystals the rule moves, 6 win and 4 tie, with mean per-shell <I/sigma> up
30.6, 24.6, 15.8, 9.1, 8.0 and 5.1 per cent and R_meas down as much as 23.8.
Runtime is neutral - 19m13s against 22m00s warm.
One crystal is a real cost and is named in docs/RUGNUX.md with its
workaround: an I222 case that is simultaneously the widest-spot and among the
highest-mosaicity in the set loses 28.5% of its observations at unchanged
completeness, because at r1 = 6 its predicted reflection density leaves the
background ring too few clean pixels. No cheap guard separates it - its
predicted spacing is mid-table, larger than five crystals that survive r1 =
12 - and the guard that would, on the measured drop rate out of pass 1, needs
a diagnostic channel out of both integration engines and is not yet
validated.
This depends on
|
||
|
leonarski_fandClaude Opus 5
|
db908c6ec1 |
_process.h5: record the change of basis, and the space group
A _process.h5 was internally inconsistent whenever the space group re-seated the lattice. The per-image reflections and lattices go to file as each image is processed, but the conventional setting is only chosen afterwards, so the file kept pre-reindex indices beside a post-reindex cell. Measured on an affected dataset, max|d_file - d(hkl, stored cell)| was 18.6 A. --mode scale then compared each frame's lattice against the stored cell, found none of 1800 acceptable, rejected every observation and died in the merge with "resolution calculation failed". /entry/MX/reindexMatrix now carries M with hkl_cell = M . hkl_written, and the reader applies it, so everything it hands out is in the setting of /entry/sample/unit_cell. Absent means the identity, so a file written before this reads exactly as before. On the affected dataset M comes out [[1,1,0],[0,1,1],[1,0,1]], det 2 - the primitive-to-body-centred basis its volume ratio implied - and the same measure falls from 18.6 A to 1.8e-5. Writing the reflections in the final setting instead was rejected: the per-image writer is shared with the broker, which streams and cannot buffer a run; and h,k,l, predicted_x/y and the per-image lattice are one consistent statement about one image, which retro-editing the indices would silently break. Two things turned up while fixing it. There are three re-seat sites, not one - the space-group search's own centred-lattice test re-seats too, and logged nothing - so the matrix composes over all of them. And the space group itself was never written: it was set only on the arm that searches, while a two-pass rotation run reuses pass 1's group and takes the other arm, so the canonical file carried a cell but no group and --mode scale merged in P1. --mode scale now reproduces --mode mx on the affected dataset: same space group and cell, 39329 unique reflections both, 100% of reflections common, CC 0.99985, sum|dI|/sum|I| = 0.0081. The residual is three reflections in 11.09 M crossing an ice-band edge, because mx carries the integrator's d and scale recomputes it from the cell. An unaffected dataset is byte-identical in .mtz, .hkl and .cif. An older affected file still cannot merge - M is not recoverable from it - but now says so in 51 s, naming both cells and the -S/-C override to use, instead of failing inside the merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
8fa1ab7b05 |
CI: make the artifact checks capable of failing, and of passing
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 7m27s
Build Packages / build:windows:nocuda (push) Successful in 15m41s
Build Packages / build:rugnux:windows (push) Successful in 16m37s
Build Packages / build:windows:cuda (push) Successful in 19m33s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 16m47s
Build Packages / build:viewer-tgz:cpu (push) Successful in 19m39s
Build Packages / build:viewer-tgz:cuda (push) Successful in 21m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 22m3s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 26m46s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 28m19s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m58s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 21m29s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 26m25s
Build Packages / build:rpm (rocky9) (push) Successful in 22m46s
Build Packages / build:rpm (rocky8) (push) Successful in 28m2s
Build Packages / Generate python client (push) Successful in 42s
Build Packages / Build documentation (push) Successful in 1m13s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m33s
Build Packages / XDS test (durin plugin) (push) Successful in 11m44s
Build Packages / DIALS test (push) Successful in 25m14s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 27m29s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 11m9s
Build Packages / XDS test (neggia plugin) (push) Successful in 9m44s
Build Packages / Unit tests (push) Successful in 1h25m11s
The rugnux tarball built and packaged correctly; the step that verifies it did not. Two shell mistakes, both mine, both of the same family. `! find /tmp/rgx -name 'libcufft*'` can never pass: find exits 0 whether or not it matched anything, so the negation always fails. That is what broke the x86_64 job after a successful build. The aarch64 checks were worse, in the way that matters. Piping a large producer into `grep -q` kills the producer with SIGPIPE as soon as grep exits on its first match, and `set -o pipefail` promotes that to the pipeline's status: on `cuobjdump --list-elf | grep -q sm_90` over a 225 MB binary it fails a check that should pass, and on `! ... | grep -q "x86-64"` the leading `!` inverts it into a PASS -- so an x86 file leaking into an aarch64 tarball, the exact thing that check exists to catch, would have been reported as clean. Both now capture each producer's output to a file and grep the file, with explicit if/exit rather than exit-status negation. Verified by running the checks verbatim against the real artifacts rather than only reading them: both tarballs pass, and the checks were confirmed to FAIL when they should -- an x86_64 file planted in the extracted tree is caught, and an architecture absent from the fatbin is reported missing. The toolchain file now comes from the checkout rather than /opt/cross in the image. It describes how to build this source, so it belongs with the source: baked into the image, the Eigen fix in the previous commit could not reach CI without rebuilding and re-pushing the image, and appeared to have no effect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h |
||
|
leonarski_fandClaude Opus 5
|
3175e65400 |
Space-group search: say what it tested, and what it could not
Three reporting changes. None of them touches a decision: the merged .mtz, .hkl, .cif and _image.dat are byte-identical before and after on two test crystals, and the space group and cell are unchanged. REPORT_VERSION moves to 3 and one warning appears where the second change is working. The alternatives line was misleading. "Best space group: C2 or P21 or P2 (indistinguishable from these data)" sat forty lines below the run's single cell, which is C2's. P2 and P21 are primitive: their lattice is a sub-lattice of that one, their cell has half its volume, and their indices are not the ones in the written files. A user acting on "P21" got a group name with no cell, beside a cell belonging to a different group, with nothing saying so. The report now names the alternatives whose centring differs from the chosen group's, gives the volume ratio of the cell each implies, and says that adopting one means reindexing. It stays byte-identical when every alternative shares the chosen centring, which is every other case that occurs - the enantiomorphic and origin-ambiguous pairs, where one cell really does describe them all. It does not print derived cell constants: the primitive cell of a C-centred monoclinic is not P2's conventional cell, and deriving that is a reduction plus a per-group setting choice, which is more than a report line should carry. A centring the data could not test now says so. Prediction runs in P so the search can confirm a centring from the reflections it extinguishes, but when the indexer returns the primitive sub-cell those reflections are never predicted, and an absent count of 0 reads as "predicts no absences" when it means "none was measured". The search abstains correctly and the centring is then taken from the metric, but none of that reached the user. The candidate table gains a three-valued centring column and the report raises a warning where the metric decides it. No decision logic changed - the flag is two counts the search had already made, and nothing reads it back. The twin-law H is now reported for every operator, and the adopted promotion's ratio against its bound. It was previously visible only inside a refusal message, which is why recalibrating that bound recently required rebuilding the statistic from reference data rather than reading it out of runs we already had. The ratio is reported at the Stage A promotion where it is computed, not in the Stage B table, whose rows all share one point group and would have printed the same number on every line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
a4228b9c1d |
tests: follow REPORT_VERSION to 2
|
||
|
leonarski_fandClaude Opus 5
|
491312f263 |
Fix the packaging split where it was only exercised on one configuration
Build Packages / build:rugnux:aarch64 (cross) (push) Failing after 2m8s
Build Packages / build:rugnux:windows (push) Successful in 14m6s
Build Packages / build:rugnux-tgz (x86_64) (push) Failing after 15m4s
Build Packages / build:windows:nocuda (push) Successful in 16m1s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 20m9s
Build Packages / build:viewer-tgz:cuda (push) Successful in 23m1s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 22m54s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 24m23s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 27m45s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 27m49s
Build Packages / build:rpm (rocky8) (push) Canceled after 7m42s
Build Packages / XDS test (durin plugin) (push) Canceled after 0s
Build Packages / XDS test (JFJoch plugin) (push) Canceled after 0s
Build Packages / XDS test (neggia plugin) (push) Canceled after 0s
Build Packages / Generate python client (push) Canceled after 0s
Build Packages / Build documentation (push) Canceled after 0s
Build Packages / Create release (push) Canceled after 0s
Build Packages / build:rpm (ubuntu2204) (push) Canceled after 4m8s
Build Packages / build:rpm (ubuntu2404) (push) Canceled after 2m52s
Build Packages / build:rpm (rocky8_sls9) (push) Canceled after 15m38s
Build Packages / build:rpm (rocky9) (push) Canceled after 7m40s
Build Packages / build:rpm (rocky9_sls9) (push) Canceled after 10m37s
Build Packages / DIALS test (push) Canceled after 2m45s
Build Packages / Unit tests (push) Canceled after 30m48s
Three failures, all from validating the rugnux path alone and assuming the rest followed. viewer/CMakeLists.txt had an unbalanced ENDIF: removing the cuFFT bundling block took its opening IF(JFJOCH_CUDA_AVAILABLE) with it and left the closer behind, so every configuration that adds viewer/ died on "flow control statements are not properly nested" -- both Windows jobs and the viewer tarball. The cross toolchain could not find Eigen. CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY confines find_package to the target roots, and the ubuntu2404 image installs Eigen from source in /opt/eigen-3.4 rather than as libeigen3-dev, so it sat outside them. It is header-only, so serving it from outside the target roots is safe; the root is now listed explicitly. Static cuFFT is not available everywhere. The Windows toolkit ships no cufft_static.lib, so CUDA::cufft_static does not exist there and linking it fails at generate time. Condition on the target rather than the platform, so a toolkit that gains or loses the static library needs no further change, and restore the Windows DLL bundling for both products -- now in the top-level CMakeLists, since rugnux needs it too and does not add viewer/. rugnux.exe therefore ships beside cufft64_*.dll rather than alone; the Linux and cross artifacts are still one file. That condition also has to keep the JFJOCH_PORTABLE_ONLY half. Testing only for the target quietly moved the SERVER build onto static cuFFT as well, where every executable -- broker, tests, tools -- would then need the CUDA device-link step and would fail on an undefined __cudaRegisterLinkedBinary_* at link. Configure still succeeds, so nothing catches that until the link; the server build keeps libcufft.so exactly as before. Checked by configuring all four flavours in the image CI actually uses -- rugnux, viewer and server on x86_64 plus the aarch64 cross -- and confirming each links the cuFFT it should. The rugnux archives now also carry their target architecture, since that build produces more than one: rugnux-<version>-linux-x86_64-cuda13.tar.gz beside the aarch64 one. CMAKE_SYSTEM_PROCESSOR is the target's, never the build host's. The viewer is x86_64-only and keeps its existing name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h |
||
|
leonarski_fandClaude Opus 5
|
fd9e7bce1f |
CI: drive the Windows rugnux build from cmd, as the viewer build already does
Build Packages / build:viewer-tgz:cpu (push) Failing after 2m31s
Build Packages / build:rugnux:aarch64 (cross) (push) Failing after 2m26s
Build Packages / build:windows:nocuda (push) Failing after 3m15s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Failing after 3m9s
Build Packages / build:viewer-tgz:cuda (push) Failing after 3m22s
Build Packages / build:rpm (rocky8_nocuda) (push) Failing after 3m27s
Build Packages / build:rpm (rocky9_nocuda) (push) Failing after 3m28s
Build Packages / build:windows:cuda (push) Failing after 4m15s
Build Packages / build:rugnux:windows (push) Failing after 4m31s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Failing after 4m6s
Build Packages / build:rpm (rocky8_sls9) (push) Failing after 4m14s
Build Packages / build:rpm (rocky8) (push) Failing after 4m2s
Build Packages / build:rpm (rocky9_sls9) (push) Failing after 4m6s
Build Packages / build:rpm (ubuntu2204) (push) Failing after 3m49s
Build Packages / build:rpm (rocky9) (push) Failing after 3m54s
Build Packages / Generate python client (push) Successful in 19s
Build Packages / Build documentation (push) Successful in 1m7s
Build Packages / build:rugnux-tgz (x86_64) (push) Failing after 8m55s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404) (push) Failing after 2m42s
Build Packages / XDS test (JFJoch plugin) (push) Canceled after 4m12s
Build Packages / Unit tests (push) Canceled after 11m42s
Build Packages / XDS test (neggia plugin) (push) Canceled after 4m10s
Build Packages / DIALS test (push) Canceled after 4m52s
Build Packages / XDS test (durin plugin) (push) Canceled after 4m14s
The job used shell: bash, and the Windows runner has no bash on PATH, so it failed before running cmake. It also skipped the MSVC environment, which every cmake invocation on that runner needs: build-windows calls vcvars64.bat, located through vswhere, at the start of each step. Follow it exactly - cmd, the same vcvars preamble, -G Ninja, and cpack from inside the build directory - and keep powershell only for the upload, which is what build-windows uses there too. CMAKE_PREFIX_PATH drops the Qt entry and keeps C:/deps: JFJOCH_RUGNUX_ONLY is the portable subtree minus viewer/, so it needs Eigen and ZLIB and no Qt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h |
||
|
leonarski_fandClaude Opus 5
|
6a7404f03f |
CI: upload the rugnux artifacts the way every other job does
Build Packages / build:rugnux:windows (push) Failing after 22s
Build Packages / build:rpm (rocky8_sls9) (push) Canceled after 0s
Build Packages / build:rpm (rocky8) (push) Canceled after 0s
Build Packages / build:rpm (rocky9_sls9) (push) Canceled after 0s
Build Packages / build:rpm (rocky9) (push) Canceled after 0s
Build Packages / build:rpm (ubuntu2204) (push) Canceled after 0s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Canceled after 0s
Build Packages / build:rpm (ubuntu2404) (push) Canceled after 0s
Build Packages / DIALS test (push) Canceled after 0s
Build Packages / XDS test (durin plugin) (push) Canceled after 0s
Build Packages / XDS test (JFJoch plugin) (push) Canceled after 0s
Build Packages / XDS test (neggia plugin) (push) Canceled after 0s
Build Packages / Generate python client (push) Canceled after 0s
Build Packages / Build documentation (push) Canceled after 0s
Build Packages / Create release (push) Canceled after 0s
Build Packages / build:viewer-tgz:cpu (push) Canceled after 1m34s
Build Packages / build:viewer-tgz:cuda (push) Canceled after 1m32s
Build Packages / build:rpm (rocky8_nocuda) (push) Canceled after 1m26s
Build Packages / build:rugnux-tgz (x86_64) (push) Canceled after 1m30s
Build Packages / Unit tests (push) Canceled after 1m36s
Build Packages / build:rugnux:aarch64 (cross) (push) Canceled after 1m28s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Canceled after 1m22s
Build Packages / build:rpm (rocky9_nocuda) (push) Canceled after 1m24s
Build Packages / build:windows:cuda (push) Canceled after 1m41s
Build Packages / build:windows:nocuda (push) Canceled after 1m52s
The three new rugnux jobs reached for akkuman/gitea-release-action@v1, which does not exist on this Gitea: actions resolve against gitea.psi.ch/<owner>/<repo>, so the runner tried to clone gitea.psi.ch/akkuman/gitea-release-action and failed. That happens while preparing the workflow, before any step runs, so all three jobs died immediately without building anything. Every other job uploads with gitea_upload_file.py and the release token, so use that: bash on the Linux jobs, powershell on the Windows one, matching the viewer tarball and installer steps they sit next to. The Linux archives are renamed .tar.gz -> .tgz first, as the viewer archive already is. actions/checkout is now the only external action the workflow references. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h |
||
|
leonarski_fandClaude Opus 5
|
8d32024dd1 |
Ship jfjoch_viewer and rugnux as two packages, and cross-build rugnux for arm64
Build Packages / build:rugnux:windows (push) Failing after 2s
Build Packages / build:rugnux-tgz (x86_64) (push) Failing after 14s
Build Packages / build:rugnux:aarch64 (cross) (push) Failing after 30s
Build Packages / Unit tests (push) Canceled after 2m33s
Build Packages / build:rpm (rocky8) (push) Canceled after 0s
Build Packages / build:rpm (rocky9_sls9) (push) Canceled after 0s
Build Packages / build:rpm (rocky9) (push) Canceled after 0s
Build Packages / build:rpm (ubuntu2204) (push) Canceled after 0s
Build Packages / build:rpm (ubuntu2404) (push) Canceled after 0s
Build Packages / DIALS test (push) Canceled after 0s
Build Packages / XDS test (durin plugin) (push) Canceled after 0s
Build Packages / XDS test (JFJoch plugin) (push) Canceled after 0s
Build Packages / XDS test (neggia plugin) (push) Canceled after 0s
Build Packages / Generate python client (push) Canceled after 0s
Build Packages / build:viewer-tgz:cuda (push) Canceled after 2m29s
Build Packages / Build documentation (push) Canceled after 0s
Build Packages / Create release (push) Canceled after 0s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Canceled after 2m19s
Build Packages / build:viewer-tgz:cpu (push) Canceled after 2m31s
Build Packages / build:rpm (rocky8_sls9) (push) Canceled after 1m54s
Build Packages / build:rpm (rocky9_nocuda) (push) Canceled after 2m21s
Build Packages / build:rpm (rocky8_nocuda) (push) Canceled after 2m23s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Canceled after 2m13s
Build Packages / build:windows:cuda (push) Canceled after 2m42s
Build Packages / build:windows:nocuda (push) Canceled after 2m45s
The viewer tarball also carried rugnux, jfjoch_extract_hkl and jfjoch_recompress. A GUI download that contains a batch processor is confusing on its own, and it forced anyone who only wanted rugnux to take Qt with it. JFJOCH_RUGNUX_ONLY is the viewer-only subtree minus viewer/ and tools/, so it needs no Qt at all, and each of the two options now yields one artifact holding one program: JFJOCH_VIEWER_ONLY jfjoch_viewer .tar.gz / installer / .dmg JFJOCH_RUGNUX_ONLY rugnux .tar.gz / .zip The two CLI tools are still built and no longer installed anywhere. rugnux gets a CPack component of its own, so its notices land in share/doc/jfjoch_rugnux. What makes the split possible is linking cuFFT statically in both products. It was the only CUDA component still dynamic - cudart and the fast-feedback indexer were already static - and shipping it as a .so meant each self-contained archive had to carry the library beside its executables and find it again through an $ORIGIN rpath. That rpath was set on jfjoch_viewer alone, while the same tarball installed three more executables, so those shipped with no runpath at all (cmake strips the build-tree one, which points at the build host's CUDA directory) and could not load the libcufft lying next to them. Static, an artifact is one file that runs: the x86_64 rugnux tarball needs only libc, libstdc++, libgcc_s, libm, libpthread, librt and libdl. The bundling code and the rpath both go away. libcufft_static.a carries a relocatable-device-code object, so a consuming executable needs a CUDA device link; CUDA 13 no longer ships the libcufft_static_nocallback.a that used to avoid this. CUDA_RESOLVE_DEVICE_SYMBOLS emits that step while leaving the host link driver alone, so the -march and -flto flags CI passes still apply. The server build keeps the shared library: its .deb/.rpm take CUDA from the distro, and every executable there would otherwise need the same device link. The guards that exclude server-side targets keyed on NOT JFJOCH_VIEWER_ONLY, which a rugnux-only build does not satisfy - it failed on jfjoch_writer pulling in JFJochImagePuller. They now key on JFJOCH_PORTABLE_ONLY, true for either product, rather than on one product's option in five places. docker/ubuntu2404 additionally cross-compiles rugnux for arm64 SBSA, which is both DGX Spark (GB10, sm_121) and Grace Hopper (GH200, sm_90); one binary with both fatbins runs on either. It needed three things beyond a cross gcc. arm64 is published on ports.ubuntu.com and Noble's deb822 sources carry no Architectures: field, so adding the architecture without splitting the entries 404s on every index. The CUDA cross packages are in neither the x86_64 repo nor the sbsa one but in cross-linux-sbsa, whose fatbinary_section.h ships only in the x86_64 target tree even though it is architecture-neutral. And HDF5 has two try_run probes, so qemu-user-static is needed as CMAKE_CROSSCOMPILING_EMULATOR; every other fetched dependency either guards its run-checks or has none. The cross-built tarball is one ELF aarch64 binary carrying all 17 GPU modules for both sm_90 and sm_121, needing no cuFFT and no CUDA runtime at run time. It is untested on hardware: XDS is x86-64 only, so CI can show it links but only a Spark or a GH200 can show it works. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h |
||
|
leonarski_fandClaude Opus 5
|
46ed703eca |
libjpeg-turbo: follow the parent build's toolchain when cross-compiling
libjpeg-turbo is brought in with ExternalProject_Add rather than FetchContent, because upstream does not support inclusion via add_subdirectory. An ExternalProject is a separate cmake invocation that inherits nothing from its parent, and the CMAKE_CACHE_ARGS here named only the feature options, so it always configured for the build host. That is invisible in a native build and fatal in a cross build: the library compiles for x86_64 inside an aarch64 build and the mistake only surfaces at the final link of whatever consumes it, as an architecture mismatch on libjpeg.a rather than anything pointing at this file. rugnux reaches it through JFJochWriter -> JFJochPreview, so it is on the path of every build, not just the viewer's. Forward the toolchain file and the compiler. CMAKE_TOOLCHAIN_FILE is empty in a native build and cmake ignores it there, so nothing changes off the cross path. libjpeg-turbo selects its own SIMD from the target architecture, so this also gets the NEON kernels for free on aarch64 and stops nasm from being consulted at all; a cross-compiled libjpeg.a comes out ELF aarch64 with 14 NEON objects. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h |
||
|
leonarski_fandClaude Opus 5
|
3ea1206775 |
Space-group search: raise the twin-law H bound from 1.70 to 1.85
Re-measured over ~780 reports - three batteries and four de-novo
integration-radius arms - H is the only Stage-A guard that ever fires: 14
refusals on three crystals, and not one chi-squared-ratio refusal or b veto
anywhere.
Two of the three are the genuine trigonal twins, reading 1.90 and 1.91 on
every arm and 3.61-4.80. The third is a weak tetragonal crystal reading
1.73-1.80 that is not twinned: its 422 is confirmed by every operator
correlation, and three independent tests agree - our own L test on the
P2(1)2(1)2(1) merge it was demoted into, xtriage on that same merge ("no
twinning is suspected... most likely due to an NCS axis parallel to the twin
axis"), and the arithmetic that a twin at the implied fraction caps twin-pair
correlation at 0.55 while the four added operators score 0.84-0.90.
Its excess is the coverage limit the comment above the bound already names:
the added 422 operators exchange h and k, so they carry a systematic floor
the parent's sign-flipping 2-folds do not. The two populations therefore
collide at 1.80 and 1.90, not at 1.70, and the bound now sits at the midpoint
of that window.
Degrading the data - 4x sigma inflation, or an eighth of the sweep - drives
the ratio towards 1 and never up, and the estimator is precise (bootstrap
standard error 0.2-3.7% of the ratio), so the collision is a genuine
population overlap rather than sampling noise. That also rules out scaling
the bound by its own error, which at any useful multiple refuses half the
battery.
Inert on the shipped defaults: no crystal in the battery reads between 1.70
and 1.85 there. It changes one assignment on an experimental integration
radius, from wrong to right. The committed twin test is unaffected - genuine
promotions in that harness read 0.959-1.016 against 4.23-5.36 for its twin,
so nothing lies in the interval that moved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P
|
||
|
leonarski_fandClaude Opus 5
|
80e10f51b3 |
rugnux: measure and report diffraction anisotropy
rugnux now says whether a dataset's fall-off is direction-dependent, and by how much. It corrects nothing and truncates nothing: no intensity is changed, no reflection is dropped on a directional criterion, and the written files do not depend on direction at all. Two quantities, because they are not the same thing. The anisotropic deltaB is the range of the principal components of the anisotropy tensor - a rate of fall-off. The diffraction limit along each principal direction is where <I/sigma(I)> in a 20 degree cone falls through 2 - where signal actually runs out. One battery case has only 0.28 A between its directional limits and a 58x ratio in cone <I/sigma>, so reporting either alone would miss it. The tensor is a Laue-constrained deviatoric ADP tensor fitted on INTENSITIES with no positivity cut, by weighted Gauss-Newton over 12 shells x 60 directions with a free constant per shell. Fitting amplitudes after a positivity cut, which is what xtriage and ctruncate do, destroys about 40% of the measured anisotropy - the cut keeps only the positive noise excursions in whichever direction has died, and that is the direction carrying the signal. Against the same 38 merged files rugnux reads 1.24x xtriage's eigenvalue spread and 1.61x ctruncate's; on strong near-isotropic data all three agree to a few percent, and they diverge exactly where a direction has died. The verdict is gated three ways - not detected, detected, or cannot determine - against the dataset's own systematic floor, measured in the tensor directions its Laue symmetry forbids. The floor cannot be measured on merged reflections, which have exact Laue symmetry by construction, so the floor is taken from the unmerged observations and the verdict is "cannot determine" without them. Triclinic has no forbidden subspace and always returns cannot determine. A cubic crystal returns exactly zero, because that is its symmetry and not a measurement. A second axis reports the resolution signature: a genuine Debye-Waller fall-off is linear through the origin in s^2, and a deficit that is flat is something else. Magnitude alone had promoted a crystal that is 68% not a Debye-Waller B into the top five of this battery; it now reads not detected with the caution attached. Following Sheriff & Hendrickson (1987) Acta Cryst. A43, 118-121 for the tensor and Popov & Bourenkov (2003) Acta Cryst. D59, 1145-1153 for the estimator. The directional limits are written as jfjoch_ local mmCIF items rather than _reflns.pdbx_aniso_diffraction_limit_*, whose dictionary definition is explicitly the ellipsoid fitted to a diffraction cut-off surface - a construction rugnux does not perform. The generic anisotropic B tensor items are written. Changes no existing number; only REPORT_VERSION moves, 1 to 2. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
0f5a6e41d6 |
rugnux: report the Wilson B in --mode scale, and correct what the docs claim about the fixed integration radius
--mode scale called AnalyzeTwinning but not CalcGlobalWilsonB, so WILSON_B= read nan there. Mirrors what --mode mx does; on one crystal scale mode now reports 72.03 against mx mode's 71.91 on the same data, and where the fitted B is not positive it still reads nan, which is the same verdict --mode mx reaches. The merged .mtz and .hkl are byte-identical before and after. docs/CPU_DATA_ANALYSIS.md and BraggStencil.h both said the flux the fixed r1 disk does not capture "is a function of resolution alone, which the per-shell scale absorbs". Neither half is true. The loss carries a directional component of several A^2 on top of the isotropic part. There is no per-shell scale, and there cannot usefully be one: every scale is fitted against a reference built from a reflection's own symmetry equivalents, equivalents share s^2 exactly, so any function of s^2 lies in the exact null space of the scaling model. What actually absorbs the isotropic part is the Wilson B, which is degenerate with it - so WILSON_B and _reflns.B_iso_Wilson_estimate carry an r1-dependent bias, measured at 18 A^2 of spread across four radius triplets on one crystal. The text also corrected a third error: in the default gaussian mode r1 is not the integration domain at all - the intensity is a profile-fit amplitude over a grid set by r2 - and a note now sits at the section a reader looking up the Wilson B actually lands on. Comments and documentation only for that part; verified by rebuilding and confirming byte-identical output with no flags. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
8db8267ff8 |
PCIe driver: build DKMS modules for the target kernel, not the running one
dkms.conf quoted the make binary: MAKE="'make' -C src/ all". dkms passes the
kernel it is building for by rewriting the leading "make" of that string into
"make -jN KERNELRELEASE=$kernelver" (dkms 3.2.1, line 1446) - an anchored
prefix substitution that the quotes defeat, so nothing was ever injected. The
Makefile then hardcoded /lib/modules/$(shell uname -r)/build and would have
ignored it in any case.
With AUTOINSTALL=yes the module was therefore always compiled against the
running kernel and installed into the tree of whichever kernel dkms was
building for. A dnf update that pulls in a new kernel builds against the old
one and drops the result in the new kernel's /extra, where it fails to load on
the next boot. Crossing RHEL 9.4 to 9.5 it would also compile the wrong side of
the vm_flags guard, which is how this was noticed.
The Makefile now takes KDIR, defaulting to the running kernel exactly as
before, and dkms.conf passes ${kernel_source_dir} - which dkms resolves for the
target kernel before sourcing the conf, and which honours --kernelsourcedir.
Dropping the quotes also lets dkms's -jN through, so the recursive invocations
become $(MAKE) to keep the jobserver.
Verified by building against a kernel other than the running one: the module
now comes out carrying the target kernel's vermagic.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PLMZgfMEdtPBZt1F2rNbRA
|
||
|
leonarski_fandClaude Opus 5
|
59ebe8d9ae |
PCIe driver: guard vm_flags_set for RHEL, which backported it into 5.14
RHEL and its derivatives report LINUX_VERSION_CODE 5.14.0 for the whole of el9 while backporting newer kernel APIs into it, so the version check alone picked the pre-6.3 branch on every el9 >= 9.5 and the module failed to build with "assignment of read-only member 'vm_flags'". HAVE_VM_FLAGS_SET could override it, but nothing sets it and dkms.conf does not pass it, so a build from the RPM had no way to reach it. vm_flags_set() and the const vm_flags member appear between kernel-5.14.0-440 and -450.el9, during the 9.5 development stream. RHEL_MINOR tracks the released minor exactly, so RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(9,5) is the test: 9.0-9.4 (GA -427) do not have them, 9.5 (-503), 9.6 (-570), 9.7 (-611) and 9.8 (-655) do. el10 is 6.12-based and LINUX_VERSION_CODE already covers it, as it does the removal of no_llseek that the other guard in this driver handles. Spelled through a JFJOCH_RHEL_GE() helper rather than inline, because the obvious form #if defined(RHEL_RELEASE_CODE) && RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(9,5) is a hard error on both gcc and clang for every non-RHEL kernel: && does not short-circuit past a parse error, and the undefined function-like macro leaves "0 (9,5)" behind. Also corrects > to >= on the upstream check. The wrappers landed in 6.3, so a vanilla 6.3.0 kernel was taking the pre-6.3 branch. Generated code on el8 is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PLMZgfMEdtPBZt1F2rNbRA |
||
|
leonarski_fandClaude Opus 5
|
ef07770c5f |
rugnux: name the unmerged export from the prefix, and drop centring absences
--export-unmerged and --export-unmerged-partials no longer take a filename;
they write <prefix>_unmerged.mtz and <prefix>_unmerged_partials.mtz, as
--write-process-h5 already does for <prefix>_process.h5.
Lattice-centring absences are no longer written. They are integrated on
purpose - prediction runs in P so the space-group search can confirm or
disprove the centring - but in an exported file they cost the reader the
answer: pointless reads the interstitial nodes, takes the lattice for
primitive, and demotes I23 to P2_1 3, while aimless roughly halves its
<I/sigma> merging them. They were also only present when the indexer had
settled on the conventional centred cell, so the file's content depended on
which of two equivalent cells it picked.
Screw and glide absences are still written. They are the evidence a space
group is chosen on, so a program reading this file can check that choice
independently; deleting them turns pointless's test into an assumption
("we cannot determine if the reflections are truly systematically absent").
Deleting them changes no merging statistic. XDS and DIALS draw the line in
the same place - XDS ships screw absences and never measures a centring one,
and DIALS's absence machinery has only screw-axis classes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P
|
||
|
leonarski_fandClaude Opus 5
|
bb5aaca68e |
rugnux: sum each rotation reflection's partials into one full on export
--export-unmerged now writes one row per reflection with its partials summed, which is how a rotation dataset is normally handed to a scaling program. The previous behaviour is still available as --export-unmerged-partials. Partials made the file's usefulness depend on the reading program's partial-assembly settings. On an orthorhombic case only 31% of partial sets passed aimless's default PARTIALS test, so it merged 2% of the file: multiplicity 4.7 against our own 12.4, CC1/2 0.788, and CC against our merged intensities 0.806. Summed, aimless uses 89% of the file and rejects nothing: multiplicity 12.5, CC1/2 0.931, completeness 96.1 to 98.0, pointless Laue confidence 0.669 to 0.792, CC 0.848. The other two test crystals are ties overall and win their outer shells, and the anomalous signal comes out stronger from the fulls. iotbx.merging_statistics, which never assembled partials at all, now reports real merging statistics rather than statistics of parts. Events are grouped by Combine()'s own rule so the file and the merge assemble exactly the same reflections; the sum is plain with variances in quadrature and no partiality division, which reproduces the merge's full times FRACTIONCALC to a few percent. Per-part metadata is combined on a partiality-weighted centroid rather than an intensity-weighted one: 8-41% of events have parts summing to zero or less, where intensity weights are undefined or sign-flipped, and 10-21% of intensity-weighted centroids fall outside the event's own frame range. Events below --min-partiality are not written. Declaring a reflection whose rocking curve caught a thousandth of itself as a complete measurement gives a scaling program noise with a small sigma and it weights it heavily; there is no partial test to catch it once it is a full. This is the same constant already applied to the same quantity in the merge, and it is what takes the orthorhombic case from CC 0.219 to 0.848. --min-partiality 0 writes every event. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_fandClaude Opus 5
|
86b7e07131 |
Drop Ceres' LAPACK, METIS and SuiteSparse back-ends: nothing calls them
Every Ceres solve in the project selects a dense linear solver explicitly -- DENSE_QR in PostRefine, RingOptimizer and StillsPartialityRefine, DENSE_NORMAL_CHOLESKY in XtalOptimizer, DENSE_SCHUR in GeometryRefiner -- and none of the seven Solver::Options sites sets dense_linear_algebra_library_type, which defaults to EIGEN. ceres::Covariance, the other SuiteSparse consumer, is never instantiated. So no sparse factorization and no LAPACK call is reachable. Measured rather than argued: an LD_PRELOAD shim exporting dgeqrf_, dpotrf_, dtrtrs_, spotrf_ and METIS_NodeND, each aborting on entry, let a full mx run (1800 frames, index -> refine -> integrate -> scale -> merge) finish with every counter at zero. The change is a no-op on results. Baseline and modified binaries produce byte-identical run.mtz, run.hkl and run.cif on a rotation dataset at -N 1, and two baseline runs give that same signature, so the comparison is meaningful. Only the DATE= stamp in the report differs. What it buys is six fewer shared libraries: libopenblas and libmetis, plus libgfortran, libquadmath, libgomp and libz behind them. rugnux links its own zlib statically -- zero undefined zlib symbols before and after -- so the dynamic libz only ever arrived through OpenBLAS. It also makes the build reproducible across hosts. SUITESPARSE defaults ON and self-disables only when its probe fails, so a machine where SuiteSparse_DIR is not found already compiled CERES_NO_SUITESPARSE while one carrying the SuiteSparse CMake config would not have. Pinning it removes that divergence, and shortens what a cross-compiled aarch64 build has to supply. EIGENSPARSE stays ON, so a sparse solver remains available if one is ever selected, without an external library. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h |
||
|
leonarski_fandClaude Opus 5
|
5248b1bc00 |
rugnux: export the integrated observations as an unmerged MTZ
Adds --export-unmerged <file.mtz>, in --mode mx and --mode scale alike. rugnux could only write merged reflections, which closed it off from the programs that consume unmerged observations - aimless, pointless, careless and iotbx.merging_statistics all read this file. The column layout follows what pointless itself writes (H K L M/ISYM BATCH I SIGI FRACTIONCALC XDET YDET ROT LP FLAG), with DELPHI, ZETA, BGMEAN and BGVAR added because we have them and a scale model can use them. Three details decide whether the file is usable: Lorentz-polarization is applied and recorded in LP. It is per-observation geometry spanning a factor of ~120 across a sweep and no reading program can recover it; merging with no scale model at all gives R_meas 0.372 / 0.346 / 0.324 in the low shells without it against 0.183 / 0.218 / 0.257 with it. The partiality is not divided out - it stays in FRACTIONCALC - and neither is the per-image scale, since these programs fit their own. The observations are partials and are flagged as such, packed as 256*M + ISYM with LDTYPE=1. Without the flag pointless reads every part as a whole observation and mis-assigns the symmetry; with it, a tetragonal case assembles 8.29M parts into 1.15M observations and comes back as its own space group at confidence 0.92. The scan axis is written as the negation of the stored goniometer axis, which is the Cambridge convention: against pointless's own orientation matrix that agrees to 0.6 degrees, where the axis as stored disagrees by 40. Merging the exported file with aimless reproduces rugnux's own merge to CC 0.9997 on a tetragonal case and 0.9978 on a cubic one, and the anomalous signal survives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P |
||
|
leonarski_f
|
61c603b274 |
v1.0.0-rc.163 (#73)
Build Packages / build:windows:cuda (push) Successful in 19m23s
Build Packages / build:windows:nocuda (push) Successful in 16m59s
Build Packages / build:viewer-tgz:cpu (push) Successful in 14m12s
Build Packages / build:viewer-tgz:cuda (push) Successful in 16m3s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m11s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 23m48s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m1s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 24m55s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m46s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 21m7s
Build Packages / build:rpm (rocky9) (push) Successful in 21m56s
Build Packages / XDS test (durin plugin) (push) Successful in 12m23s
Build Packages / build:rpm (rocky8) (push) Successful in 24m52s
Build Packages / Generate python client (push) Successful in 37s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 21m6s
Build Packages / Create release (push) Skipped
Build Packages / Build documentation (push) Successful in 55s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m0s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m2s
Build Packages / XDS test (neggia plugin) (push) Successful in 9m6s
Build Packages / DIALS test (push) Successful in 23m28s
Build Packages / Unit tests (push) Successful in 2h41m0s
* Packaging: each package installs its license notices under a directory of its own - `share/doc/jfjoch_broker`, `share/doc/jfjoch_writer`, `share/doc/jfjoch_viewer`, `share/doc/jfjoch_driver_dkms` - instead of all of them into the shared `share/doc/jfjoch`, so `jfjoch` and `jfjoch-writer` can be upgraded one at a time instead of only together. Reviewed-on: #73 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>1.0.0-rc.163 |
||
|
leonarski_fandjungfrau
|
4dc2534dbf |
v1.0.0.rc-162 (#72)
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>1.0.0-rc.162 |
||
|
leonarski_f
|
538f3504d3 |
v1.0.0.rc-161 (#71)
Build Packages / build:windows:nocuda (push) Successful in 20m4s
Build Packages / Unit tests (push) Skipped
Build Packages / build:viewer-tgz:cpu (push) Successful in 16m5s
Build Packages / build:viewer-tgz:cuda (push) Successful in 17m26s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 27m46s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m17s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 26m13s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 23m17s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 28m11s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 19m30s
Build Packages / build:rpm (rocky8) (push) Successful in 24m34s
Build Packages / build:rpm (rocky9) (push) Successful in 21m30s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 23m33s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 20m18s
Build Packages / DIALS test (push) Successful in 18m23s
Build Packages / XDS test (durin plugin) (push) Successful in 11m30s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 10m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m2s
Build Packages / Generate python client (push) Successful in 49s
Build Packages / Build documentation (push) Successful in 1m21s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 29m45s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>1.0.0-rc.161 |
||
|
leonarski_f
|
67dca388bd |
v1.0.0-rc.160 (#70)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 18m44s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m11s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m54s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m40s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m41s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m10s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m4s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m23s
Build Packages / build:rpm (rocky8) (push) Successful in 11m30s
Build Packages / build:rpm (rocky9) (push) Successful in 12m51s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m8s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m21s
Build Packages / DIALS test (push) Successful in 13m22s
Build Packages / XDS test (durin plugin) (push) Successful in 9m2s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m55s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m57s
Build Packages / Generate python client (push) Successful in 23s
Build Packages / Build documentation (push) Successful in 57s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 10m24s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>1.0.0-rc.160 |
||
|
leonarski_f
|
dd0bffb283 |
v1.0.0-rc.159 (#69)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 11m6s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m27s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m54s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m25s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m5s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m33s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m19s
Build Packages / build:rpm (rocky8) (push) Successful in 12m23s
Build Packages / build:rpm (rocky9) (push) Successful in 13m21s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m30s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m55s
Build Packages / DIALS test (push) Successful in 13m42s
Build Packages / XDS test (durin plugin) (push) Successful in 9m26s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 6m41s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m12s
Build Packages / Generate python client (push) Successful in 19s
Build Packages / Build documentation (push) Successful in 52s
Build Packages / Create release (push) Skipped
Build Packages / build:viewer-tgz:cpu (push) Successful in 5m29s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m12s
Build Packages / build:windows:cuda (push) Successful in 18m36s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>1.0.0-rc.159 |
||
|
leonarski_f
|
451310f43d |
v1.0.0-rc.158 (#68)
Build Packages / Unit tests (push) Successful in 1h32m35s
Build Packages / build:windows:cuda (push) Successful in 18m0s
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m37s
Build Packages / build:viewer-tgz:cuda (push) Successful in 8m55s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m13s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m11s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m35s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m57s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m23s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m45s
Build Packages / build:rpm (rocky8) (push) Successful in 11m39s
Build Packages / build:rpm (rocky9) (push) Successful in 14m0s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m42s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m38s
Build Packages / DIALS test (push) Successful in 14m55s
Build Packages / XDS test (durin plugin) (push) Successful in 7m11s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m7s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m34s
Build Packages / Generate python client (push) Successful in 28s
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 9m55s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>1.0.0-rc.158 |
||
|
leonarski_f
|
54c0100e8e |
v1.0.0-rc.157 (#67)
Build Packages / Unit tests (push) Successful in 1h28m28s
Build Packages / build:windows:nocuda (push) Successful in 14m45s
Build Packages / build:windows:cuda (push) Successful in 13m13s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m47s
Build Packages / build:viewer-tgz:cuda (push) Successful in 7m22s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m52s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m16s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m19s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 12m50s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m40s
Build Packages / build:rpm (rocky8) (push) Successful in 11m18s
Build Packages / build:rpm (rocky9) (push) Successful in 12m4s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m55s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m22s
Build Packages / DIALS test (push) Successful in 13m37s
Build Packages / XDS test (durin plugin) (push) Successful in 8m47s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m4s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m45s
Build Packages / Generate python client (push) Successful in 34s
Build Packages / Build documentation (push) Successful in 1m4s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 7m16s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>1.0.0-rc.157 |
||
|
leonarski_f
|
d6389e12da |
v1.0.0-rc.156 (#66)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 15m31s
Build Packages / build:viewer-tgz:cpu (push) Successful in 5m46s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m9s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m25s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m21s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m41s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m18s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m26s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m33s
Build Packages / build:rpm (rocky8) (push) Successful in 10m32s
Build Packages / build:rpm (rocky9) (push) Successful in 12m23s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m12s
Build Packages / DIALS test (push) Successful in 12m6s
Build Packages / XDS test (durin plugin) (push) Successful in 8m15s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m12s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m35s
Build Packages / Generate python client (push) Successful in 27s
Build Packages / Build documentation (push) Successful in 54s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 12m37s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>1.0.0-rc.156 |
||
|
leonarski_f
|
54c667190f |
v1.0.0-rc.155 (#65)
Build Packages / Unit tests (push) Successful in 1h26m8s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m38s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m45s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m39s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m55s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m51s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m35s
Build Packages / build:rpm (rocky8) (push) Successful in 12m28s
Build Packages / build:rpm (rocky9) (push) Successful in 13m20s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m15s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m43s
Build Packages / DIALS test (push) Successful in 14m21s
Build Packages / XDS test (durin plugin) (push) Successful in 7m48s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m52s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m31s
Build Packages / Generate python client (push) Successful in 15s
Build Packages / Build documentation (push) Successful in 53s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #651.0.0-rc.155 |
||
|
leonarski_f
|
6136f858af |
v1.0.0-rc.154 (#64)
Build Packages / Unit tests (push) Successful in 1h26m51s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m56s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m43s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m53s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m44s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m22s
Build Packages / build:rpm (rocky8) (push) Successful in 13m1s
Build Packages / build:rpm (rocky9) (push) Successful in 14m6s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m0s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m51s
Build Packages / DIALS test (push) Successful in 13m52s
Build Packages / XDS test (durin plugin) (push) Successful in 9m24s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m35s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m57s
Build Packages / Generate python client (push) Successful in 35s
Build Packages / Build documentation (push) Successful in 47s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability) Reviewed-on: #64 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>1.0.0-rc.154 |
||
|
leonarski_f
|
75e401f0e5 |
v1.0.0-rc.153 (#63)
Build Packages / Unit tests (push) Successful in 1h31m59s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 8m43s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m5s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m27s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m56s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 9m24s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 10m27s
Build Packages / build:rpm (rocky8) (push) Successful in 9m20s
Build Packages / build:rpm (rocky9) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m54s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m38s
Build Packages / DIALS test (push) Successful in 12m13s
Build Packages / XDS test (durin plugin) (push) Successful in 7m8s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m8s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m50s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / Build documentation (push) Successful in 50s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #631.0.0-rc.153 |
||
|
leonarski_f
|
c49bd2ac3b |
v1.0.0-rc.152 (#62)
Build Packages / XDS test (neggia plugin) (push) Successful in 6m2s
Build Packages / Unit tests (push) Successful in 1h37m1s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 12m4s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m30s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 12m52s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 11m53s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 12m38s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 13m30s
Build Packages / build:rpm (rocky8) (push) Successful in 10m47s
Build Packages / build:rpm (rocky9) (push) Successful in 11m48s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 10m40s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 9m44s
Build Packages / DIALS test (push) Successful in 12m59s
Build Packages / XDS test (durin plugin) (push) Successful in 8m33s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 6m24s
Build Packages / Build documentation (push) Successful in 57s
Build Packages / Create release (push) Skipped
* jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5) * jfjoch_viewer: Adjust Q bounds for azimuthal integration * jfjoch_azint: Add tool to do quick azimuthal integration Reviewed-on: #621.0.0-rc.152 |
||
|
leonarski_f
|
ef52dac2ee |
v1.0.0-rc.151 (#61)
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 11m34s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 12m52s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 12m54s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m48s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 12m50s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 13m54s
Build Packages / build:rpm (rocky8) (push) Successful in 12m46s
Build Packages / build:rpm (rocky9) (push) Successful in 11m56s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 10m34s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 9m54s
Build Packages / DIALS test (push) Successful in 13m1s
Build Packages / XDS test (durin plugin) (push) Successful in 8m32s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m44s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m3s
Build Packages / Generate python client (push) Successful in 13s
Build Packages / Build documentation (push) Successful in 47s
Build Packages / Create release (push) Skipped
Build Packages / Unit tests (push) Successful in 43m38s
* jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname Reviewed-on: #61 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>1.0.0-rc.151 |
||
|
leonarski_f
|
90e804acd7 |
v1.0.0-rc.150 (#60)
Build Packages / Unit tests (push) Successful in 42m49s
Build Packages / DIALS test (push) Successful in 29m45s
Build Packages / XDS test (durin plugin) (push) Successful in 19m27s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 18m52s
Build Packages / XDS test (neggia plugin) (push) Successful in 13m0s
Build Packages / Generate python client (push) Successful in 28s
Build Packages / Build documentation (push) Successful in 1m25s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m53s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 12m49s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m7s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m9s
Build Packages / build:rpm (rocky8) (push) Successful in 13m24s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m11s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m15s
Build Packages / build:rpm (rocky9) (push) Successful in 14m30s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 8m14s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 9m5s
* jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin. * jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins. Reviewed-on: #601.0.0-rc.150 |
||
|
leonarski_f
|
ea575f790a |
v1.0.0-rc.149 (#59)
Build Packages / Unit tests (push) Skipped
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 24m44s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 23m45s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 26m12s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 26m53s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 28m49s
Build Packages / build:rpm (rocky8) (push) Successful in 25m28s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 29m49s
Build Packages / XDS test (durin plugin) (push) Successful in 18m59s
Build Packages / Generate python client (push) Successful in 51s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m42s
Build Packages / Create release (push) Skipped
Build Packages / Build documentation (push) Successful in 1m51s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 26m30s
Build Packages / XDS test (neggia plugin) (push) Successful in 19m47s
Build Packages / build:rpm (rocky9) (push) Successful in 29m37s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 22m18s
Build Packages / DIALS test (push) Successful in 30m9s
* XDS plugin: Fix HDF5 mutex to run on multiple processors Reviewed-on: #591.0.0-rc.149 |
||
|
leonarski_f
|
cc3eb8352c |
v1.0.0-rc.148 (#58)
Build Packages / Unit tests (push) Skipped
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m28s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m9s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m47s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m58s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m39s
Build Packages / build:rpm (rocky8) (push) Successful in 11m43s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m59s
Build Packages / Generate python client (push) Successful in 35s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m48s
Build Packages / build:rpm (rocky9) (push) Successful in 12m32s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m24s
Build Packages / XDS test (durin plugin) (push) Successful in 7m35s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m50s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m40s
Build Packages / DIALS test (push) Successful in 11m19s
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #581.0.0-rc.148 |