185 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	e02dd2e924	rugnux: the two-pass guard weighs the reflections a space group is read from ... The guard that chooses between the file's geometry and the refined one judged the two passes on CC1/2 alone. CC1/2 is an average over tens of thousands of reflections and the principal axial rows hold a few dozen, so a pass can win it while measuring half the axial reflections the other one did - and the systematic absences that decide a screw axis are read from nothing else. An axial reflection crosses the Ewald sphere over a narrow range of the sweep, so a pass that loses frames loses whole rows rather than a scattering of reflections. Measured on a crystal that was losing a screw exactly this way: the refined pass held 7 of the 16 low-order axial reflections and won the gate by 0.0114 in CC1/2, which is inside its own noise. The screw evidence fell from 24.2 nats to 10.5, under the bar, and the run reported the symmorphic group. A third arm, beside the two the guard already had: refuse the refined pass when it holds under three quarters of pass 1's axial reflections AND its CC1/2 does not say it is better. It can only decide cases the gate was deciding on a difference inside its own noise, and it needs pass 1 to hold enough of the row for a screw to be claimable at all. Over the corpus it fires on three crystals: the one it was built for, which recovers its screw; a second that gains 0.057 in CC1/2 and was being handed the wrong geometry for reasons nobody was looking for; and a third that loses 0.009. The other 145 are untouched. This closes a regression the six-spot floor exposed. The floor did not cause it - at the old floor the refined pass was broken outright and the CC1/2 arm rejected it for unrelated reasons, which rescued this crystal by accident. What the floor removed was the accident. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 07:18:59 +02:00
leonarski_fandClaude Opus 5	2c4b804f70	docs: the scaling model carries all three deterministic corrections, as the code does ... The observation model said its L term was the prescaling correction, and that predictions carry its reciprocal as prescaling_corr. That was true when that one field held everything. It has since been narrowed to Lorentz x polarization, with the sensor's efficiency and the flight path carried beside it, and every site that corrects an intensity multiplies all three - the scaling fits, the merge ingest, the anisotropy analysis and the unmerged export alike. So the model term is the product, not the first factor of it, and a reader implementing against that page would have applied a third of the correction. The header comment on the efficiency factor had the same fossil: it said the total correction is prescaling_corr * qe_corr, written before the flight path existed. The comment two fields below it already said all three, so the file disagreed with itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 22:59:40 +02:00
leonarski_fandClaude Opus 5	fc41b1c34c	release: version 1.0.0-rc.167 ... VERSION bumped and update_version.sh run, which rewrites the version string everywhere it is recorded: the OpenAPI specification and the three clients generated from it, the frontend package and its version module, the Sphinx configuration, the Redoc page, and the FPGA HDL and PCIe driver strings. Every hunk is a version string and nothing else, checked rather than assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 22:45:37 +02:00
leonarski_fandClaude Opus 5	2538f5a956	docs: the manual describes the program this branch actually built ... A sweep of docs/ against the code, and a changelog a user can read. The report reference still described the section layout from before the report was rewritten, claimed anomalous keys were withheld from Friedel-merged runs when 144 of 148 stored reports carry them, and listed as default a set of keys that --developer now selects. The tutorial described an enantiomorph message that no longer exists; the option tables were missing --developer and --finalist-ledger and carried a short option the program does not have; the integration page's unmerged-MTZ column list predated the two new columns and asserted the absence of one of them. The analysis pages had the efficiency and flight-path corrections but none of the symmetry work: the pseudo-translation detector, the absence test that divides it out, the glide test, the evidence-keyed alternatives and the metric re-ask are now written up where the methods are described. The usage message omitted FLIGHT from the formula that turns a written intensity back into a raw count, while the writer has been emitting the column. The usage message is the authority on what the program does, so it says so now. The changelog had thirty-one entries filed under the previous release, most of them written from the inside: what a change did to the code rather than what it does for a reader. This release gets its own section with sixteen, each one capability. Every key, flag and column a user could grep for survives; what goes is the seam between one person's work and the next's, which is not something a user can act on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 22:44:09 +02:00
leonarski_fandClaude Opus 5	e8fbae879a	rugnux: the two offline instruments go, and one release states one report version ... The finalist-ledger and glide instruments were built to measure a question each, over stored merges, and both questions have been answered. They were never asked for and they are not part of the product: neither is installed, both duplicate what rugnux itself does with a flag, and each is one more executable to keep linking. The --finalist-ledger mode stays where it belongs, in rugnux. Their removal also retires the CUDA device-link property they would have needed in the portable build, which is the shape of problem an unused target creates: a configuration nobody builds, breaking for a target nobody runs. The changelog claimed four different REPORT_VERSION values inside one unreleased section, which is development history rather than something a user can act on. A release states its report version once; the entries keep what they changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 21:53:17 +02:00
leonarski_fandClaude Opus 5	ac1e8b8603	tests: the report version pin and the screw-gate section follow the code ... Two test expectations that the changes under them made stale, and one overstated sentence in the docs. REPORT_VERSION was bumped to 12 without touching the line that pins it - the very line whose comment says a key added to the report is a contract change and this is where it has to be acknowledged. Acknowledged, and the absence of SOHNCKE_SPACE_GROUP on a fixture that is GIVEN its group rather than searching for one is now asserted too, since that is what makes the key's contract honest. The section documenting that the E^2 gate is what saves a screw from fabricated violations no longer holds, because there are now two independent defences: with the gate off, the absent class still sits at 2% of its own row, so the zone is dead per reflection and licenses the absence evidence to override the count. The section is kept rather than deleted - it now pins the second defence, and a future P2 there would mean the deferral has stopped licensing a zone that is genuinely extinct. The docs said SOHNCKE_SPACE_GROUP is written on every run. It is written on every run whose group came from the search; a run given its group with -S has no Sohncke candidate to name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 21:12:42 +02:00
leonarski_fandClaude Opus 5	485bbeaf71	rugnux: a cell given with -C is matched in any equivalent description ... Every candidate the indexer forms is Niggli-reduced and primitive; the reference cell was compared against them exactly as typed. For a centred lattice those are different numbers entirely - a conventional F-cubic cell reduces to a rhombohedron with 60-degree angles, a centred monoclinic cell to axes it shares none of - so the true candidate failed the 5% test and the run reported that it had found no lattice. A user who supplied the cell as it is published was worse off than one who supplied nothing. The reference is now expanded into the primitive lattices its six numbers could stand for, one per centring, each reduced the way a candidate is; a candidate matching any of them is kept. The cell as typed stays in the set, so the indexer that returns the basis it was handed is unaffected. The cost is real and stated: a wrong cell now gets several chances to match rather than one. The controls that hold are on data - deliberately wrong cells are still refused, and every cell that worked before gives the same answer to the digit - but a synthetic lattice can be made to match through a centring hypothesis it has no right to, and no volume test tightens that without excluding a true candidate this corpus already contains. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 20:26:38 +02:00
leonarski_fandClaude Opus 5	78a941156b	symmetry: absence evidence overrides the violation count, one zone at a time ... A screw candidate was refused whenever more than a tenth of its predicted-absent reflections read as present, whatever the absences themselves said. On a strong axial row near the spindle those readings need not be structure factors at all: measured on one crystal, the same reflection reads +268 at one Ewald crossing and -44 at the other, a directional smear tail of the monsters beside it. Three such reflections refused a group whose absence evidence was the highest of every candidate scored, on exactly the axes the deposition names. So a class whose evidence clears the claim bar may now override the count. The likelihood has already priced those reflections in - they are inside sum_u - and still reads the class as extinct, so the count adds nothing it has not weighed, and adds it unweighted. The deferral is licensed ZONE BY ZONE, which is the part that makes it safe. The evidence is a group-level number while the count indicts particular zones, so an overwhelming genuine zone would otherwise pay another zone's debts: a four-fold screw row at 194 nats lifted the veto for two claimed two-fold rows reading 9 nats each with half their class individually present. Each zone carrying violations must now hold the deferral itself - dead at a nat per predicted absence rather than merely accumulating one over a long row, and not judged against a measured pseudo-translation, where the corrected evidence is a residue of the cone's own under-measure and the corrected count is the only instrument the modulation does not reach. Measured over the corpus: two crystals gain the screw their absences support, no crystal loses one, and every merged statistic is unchanged - a screw and its parent share a Laue class, so nothing about the merge moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 20:26:23 +02:00
leonarski_fandClaude Opus 5	118746d8da	symmetry: indistinguishable groups are keyed on the evidence, not on a count of absences ... The alternatives a run reports were the candidates whose absent AND violation counts matched the winner's exactly. Two settings of one point group routinely predict a different NUMBER of absences on a zone the sweep never measured - one extra reflection on a row with no control class of its own, contributing nothing either way - and that one reflection was enough to drop a candidate the data support exactly as well as the one adopted. Measured on an orthorhombic wedge: four candidates at the same 100.7 nats with no violations, of which the two predicting one more unmeasured absence were excluded, and one of those two was the right answer. The report then said the space group followed from the systematic absences, on an axis whose row was never collected. Keyed on the evidence, a super-centering that nets the same count but over-claims still scores differently and is still not offered. The adopted group does not change - it is the same candidate either way - so this only widens what the run admits it cannot separate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 20:25:51 +02:00
leonarski_fandClaude Opus 5	f98cbf5373	symmetry: a reference correlation with too little to correlate is refused, not scored zero ... PickBestReindex discards only non-finite scores, so returning 0.0 when fewer than the minimum number of reflections matched made that a SCORE rather than an abstention. An operator that mapped almost everything outside the reference's coverage - and therefore had nothing to be judged on - beat a measured identity that merely correlated a little negatively, and the whole run was reindexed on no data. NaN is what the per-image resolver's ReferenceCC already returns in the same situation, against the same minimum. This makes the two agree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 20:25:39 +02:00
leonarski_fandClaude Opus 5	de1b9299cb	rugnux: the report names the best Sohncke group on every run ... A crystal built from chiral molecules - any protein - has no glide plane and no inversion centre, so where the search names a group carrying a glide, the group that crystal could actually have is a different one. The run knows both and printed only the adopted one as a key. SOHNCKE_SPACE_GROUP= is written whenever a space group was determined, and repeats SPACE_GROUP_NAME= where no glide was found. That repetition is the point: a key that appears only sometimes has to be tested for before it can be read, and the reader this is for - someone who knows their sample is a protein and wants the answer their sample can have - should be able to grep one name without first deciding what the run concluded. REPORT_VERSION is 12. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 07:33:57 +02:00
leonarski_fandClaude Opus 5	6773e8516d	symmetry: a metric two-fold the lattice search refused is asked of the intensities ... The Niggli character walk takes the first character its tolerance matches, and on a lattice that is nearly but not exactly hexagonal it matches hexagonal. Under a hexagonal holohedry no candidate point group can carry the two strongest two-folds the data actually have, so the search lands on the weakest one and the crystal is processed in a group of order two where it should have been eight. The metric group knows better than the character walk: it holds every rotation the cell can host, including the ones the named lattice system has no room for. Each of those is now put to the intensities directly - one operator, scored the way Stage A scores its own, on the same reflection population, with the same strong-reflection gate and the same E^2 cap. That last part is what makes the answer usable: normalising over the full resolution range of a merge whose outer shells are correlated junk reads a genuine two-fold at CC 0.05, and the same operator over the population the pipeline itself pairs reads 0.88. The flattening and normalising the search does at its start is now one function, so an operator can be asked about without enumerating a point group around it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 07:21:41 +02:00
leonarski_fandClaude Opus 5	77390a847b	rugnux: a frame counts as indexed from six spots, the floor the setter already allowed ... The bar a frame had to clear to validate a lattice was 9, while the setter's own minimum was 6. On a crystal whose frames cannot carry 9 the difference is not a standard of evidence, it is arithmetic: the true lattice is found, indexes every frame that has the spots for it, and is then refused because too few frames could clear a count they never had spots to reach. A 7 A crystal with a median of eleven in-range spots per frame, a third of whose frames cannot reach 9 at all, validated on six frames of sixty and was thrown away - the lattice it named merges to completeness at its deposited resolution. The gate that carries the meaning is the fraction of frames that must validate, which is untouched: a wrong lattice does not index 20% of frames on six spots either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 07:10:07 +02:00
leonarski_fandClaude Opus 5	0d56a6219b	symmetry: a glide plane named from its own zone, and the Sohncke answer kept beside it ... A screw axis extinguishes a row; a glide plane extinguishes a zone. The search scored the first and had no candidate that could carry the second, so a small molecule whose cell and whose rotations were both found correctly stopped one glide short of its group. The zone is keyed by the rotation part of the improper operator, so a glide and its centring partner - the c and the n of the same plane - are one zone rather than two scored twice. A Sohncke group has no improper operator at all, so on chiral data the zone loop never runs and every Sohncke candidate scores what it scored before, bit for bit. That is why this is offered inline rather than behind a flag; it was still measured, on 140 protein datasets, and no protein answer moves and no non-Sohncke candidate reaches the top of the ranking. A glide is judged per reflection rather than on the zone's sum. The statistic is linear in the number of absences at fixed deadness, and a plane holds hundreds to thousands of reflections where a row holds tens, so a sum reaches hundreds of nats on a class that is merely a few times weak; per reflection the false and the genuine zones separate by a factor of nine. An unmeasurable zone refuses its candidate rather than abstaining, because a glide is an extra claim on top of a group that already fits without it. Both readings are reported on every run. A crystal of chiral molecules cannot have a glide, so a reader who knows the sample is a protein must be able to see the Sohncke group without processing the images again. A centre of symmetry is not offered at all: a candidate is enumerated only when its absences differ from a Sohncke candidate's, and an inversion centre predicts exactly what its Sohncke subgroup predicts. It cannot be claimed from intensities and this does not claim it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 07:09:26 +02:00
leonarski_fandClaude Opus 5	49938d2a24	symmetry: an axial absence is judged after the pseudo-translation modulating it is divided out ... A translational pseudo-symmetry splits every reflection into two classes by the parity of its index along the translation, one systematically strong and the other systematically weak. Where the translation is a half-integer along an axis, those two classes are exactly the absent class and the control class of a screw along that same axis - so the absence test divides one by the other and pays the suppression twice, reading a class that is present but suppressed as extinct, and buying a screw the crystal does not have. Measure the modulation along each axis from the intensities and divide it out before scoring. Per axis, not pooled: on the crystal this was found on it is 0.034 along the row carrying the false screw against 0.287 and 0.159 along the two rows whose screws are real, and a pooled estimate under-corrects the row that needs it while over-correcting the rows that do not. The violation counter is corrected with the evidence, because candidates rank on the sum of their zones and a correction that only ever lowers a score can never unseat a screw that has already been added. The false zone goes from 75.7 nats for its screw to 56.3 against it and from five violations in ninety to fourteen; the run adopts the deposited group. The two genuine screws on the same crystal pay 2.4 and 1.2 nats and gain no violations. On a crystal whose deposit puts a real screw on a row that also carries a half-integer translation, the correction fires, that screw pays 18.4 nats and still carries 59.5, and the verdict does not move. Across the corpus nine crystals engage it and one zone changes state. Only order-two screws are treated: a three-fold with a one-third translation is left unanswered rather than answered no. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 23:56:11 +02:00
leonarski_fandClaude Opus 5	8b9598c1c4	twinning: the L-test partners are chosen so a pseudo-translation cannot silence it ... The L-test compares a reflection with a partner a fixed step away, and the step it used preserves the parity class of a half-integer pseudo-translation - so a pseudo-body-centring was invisible to it by luck rather than by design. A translation of one third is not so lucky: it puts the two members of a pair in different modulation classes, inflates the statistic past the bound that is read as evidence AGAINST twinning, and the twin call is lost silently on a crystal that has one. Choose the partners so every pair stays inside one modulation class, which is what the half-integer case was already getting by accident. The obvious alternative - lengthening the step until it clears a third as well - was measured and rejected: it puts the statistic past that bound on more than two thirds of the corpus against a seventh today. Measured over a hundred and thirty-seven crystals: a hundred and twenty-nine unchanged, seven repaired, and exactly one verdict moves - a crystal whose twinning was being denied by its own pseudo-translation. On a synthetic perfect twin carrying a one-third translation, both indicators are destroyed before the change and both return after it. Nothing branches on this verdict, so no merged intensity moves; the whole effect is what the report says about the crystal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 20:09:00 +02:00
leonarski_fandClaude Opus 5	a4be5e6cd0	diagnostics: a translational pseudo-symmetry is detected and named ... A pseudo-translation makes one class of reflections systematically strong and the complementary class weak. It is the classic predictor of a failed molecular replacement, it raises the second moment where twinning lowers it so each masks the other's test, and rugnux did not look for it at all. Look for it: a native Patterson from the merged intensities, the largest off-origin peak taken as a fraction of the origin, and the vector refined against the data. Both halves are scored against a null computed for that crystal rather than a bound taken from elsewhere - the peak against intensities permuted within resolution shells, the modulation depth against the same search started from random vectors - because the noise floor of the first runs from 1.35 to 17.7 per cent across the corpus, so no fixed threshold could mean the same thing twice. Requiring both is what keeps the false-positive rate down: either alone calls one crystal in ten. A translation the data are exactly invariant under is a lattice vector, not a pseudo-symmetry. Recognising that removes eleven false calls from ninety-five merges folded in P1 - each reading 83 to 102 per cent of origin against a genuine maximum of 62 - and adds one true positive whose pseudo-translation lies underneath its own centring and is unreachable without it. Report-only: it gates nothing and changes no reflection, no scale and no group. It fires on eight of a hundred and thirty-seven crystals, and on those the verdict moves from OK to WARNINGS, which is the point of it. Median cost 55 ms, worst 1.9 s on the largest merge here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 20:08:40 +02:00
leonarski_fandClaude Opus 5	5b8ce26c83	integration: the flight path between the sample and the detector is corrected for, and named ... A reflection arriving at an angle to the detector normal crosses D/cos(alpha) of whatever lies between the sample and the sensor, not D, so it is attenuated more than one arriving head-on and reads low. That is the same geometry as the sensor crossing already corrected here and the opposite sign, and it was missing. The factor is exp(D/L*(1/cos(alpha)-1)) from the NIST attenuation coefficient of the medium, the stated distance and the stated wavelength. Nothing in it is fitted, and it is not justified by any measured amplitude: the flight path and the sensor crossing are collinear to better than 0.998 over the angular range any single experiment samples, so no fit of one can be evidence for the other. It is the tabulated absorption of a known thickness of a known material over a known path. The medium cannot be detected. No field of the NXmx application definition describes it, none of the masters this program reads carries one, and it cannot be inferred from the implied transmission either - in this corpus a station confirmed to use helium sits at 51% implied air transmission and one confirmed to use air at 63%, so any rule separating them is a threshold fitted between two points. It is therefore assumed, stated, and overridable: --flight-path air|helium|vacuum, defaulting to air. Helium is its own material rather than an alias for vacuum, attenuating about a six hundredth of air rather than nothing. On an untilted detector the correction is a function of resolution alone, so its entire effect on merged data is a shift in the Wilson B - which is what the report now prints beside the assumption, accurate to better than a tenth of an angstrom squared against measurement from 0.05 up to 28. Where that shift is large the report warns, because a wrong medium is then the largest number in the run: applied to data from the confirmed helium station it returns a B of 14 A^2 at 3.0 A resolution, which is not a value a crystal can have. The corpus contains its own control. One crystal, one station, three collections a quarter of an hour apart at falling energy through the same air: corrected, the Wilson B rises monotonically with the dose, as it must. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 17:55:43 +02:00
leonarski_fandClaude Opus 5	7c10d62dab	integration: the sensor efficiency is carried as its own quantity, not folded into the Lorentz-polarization factor ... It was multiplied into the per-reflection factor at prediction, so that factor held Lorentz, polarization and efficiency at once and the two spellings that reach a file - the wire key and the reflection dataset - meant something different from what they had meant the day before. The unmerged MTZ had to divide the two apart again at write time to fill its own columns, which is a good sign the wrong thing was being carried. Carry them separately. The prescaling factor is Lorentz and polarization again, what its name and both reference implementations mean by it, and the efficiency is its own field through prediction, integration, serialization and storage. Fifteen sites that want the total now multiply the two - once per reflection, not once per pixel. The efficiency is stored rather than recomputed on read, because the writer has no geometry to recompute it from, and because a file written before the correction existed would have had a radial trend invented for it. Sixty stored files were checked for the one combination that would be ambiguous - the old meaning of the factor beside a stored efficiency - and none carries it. Output does not move. Re-scaling a file written before the efficiency existed is byte-identical, which is a proof rather than a sample, since the stored factor is exactly one there. Where the efficiency is live, one product is reassociated - (L*Q)/P becomes (L/P)*Q - and about a third of the values differ in the last bit or two: every structural column is identical, so no reflection is gained, lost or reindexed, and no intensity in 1.4 million observations moves by as much as 1e-4 of its own sigma. The parity tests now compare the efficiency as well, and their non-vacuity guard watches it rather than the factor it left - which is the same guard that went blind when the efficiency was added to a field it was not watching. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 16:28:15 +02:00
leonarski_fandClaude Opus 5	e44ce069a2	docs: the polarization factor is credited, and the sign that sets its plane is documented ... The correction is Kahn's, and the paper was not cited anywhere. It is now, at the function and in the acknowledgements. The plane was already settable and only the API said so: a negative factor is a vertically polarized beam. Measured on the corpus, dividing by the factor as given collapses the azimuthal modulation of a real background from 27.6% to 0.77%, and asking for the wrong plane doubles it to 55.7% - so the convention is right and the sign is load-bearing. The usage text now says which sign means which plane. No dataset in the corpus declares a plane and every one that can be measured is horizontally polarized, so nothing here changes what any current run computes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 14:30:04 +02:00
leonarski_fandClaude Opus 5	6140de7249	rugnux: the unmerged MTZ carries the sensor efficiency in its own column, not inside LP ... LP is a column other programs read and un-apply. It is documented as the Lorentz-polarization factor, and until the sensor efficiency was folded into the same product that is what it held. Measured on our own unmerged file, LP spanned a factor of 1.1343 across the detector where pure L/P spans nothing of the sort - the excess is the efficiency, 13% end to end at 13 keV and 34% at 18 keV. Both reference implementations keep it out. Recomputing pure L/P from a stored XDS file's own geometry over its 124k reflections leaves RLP flat to 0.1% from 8.6 to 33.7 degrees, where a folded-in efficiency would have shown a 7% trend - and XDS has the sensor numbers in hand. DIALS fills LP from lorentz and polarization alone and writes QE as a separate column, even a column of ones. Split them: LP is L/P again, QE is the efficiency, and the intensity is unchanged, so a reader that un-applies LP recovers what it expects and one that wants raw counts divides by LP and multiplies by QE. Only the unmerged file moves - every other column is bit-identical and the file grows by exactly one float per reflection plus one header record. The merged files are byte for byte what they were. The process file gains an optional qe dataset beside the existing one rather than changing what that one means, so a file written before the efficiency existed still loads, and reads back as a correction of exactly one - which is what it was. It is stored rather than recomputed on read because the writer has no geometry to recompute it from, and because recomputing would have written a radial trend into every stored file that never had one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 14:29:45 +02:00
leonarski_fandClaude Opus 5	db4af06a2b	integration: the sensor efficiency reaches the stills GPU predictor too ... The efficiency correction was added to the rotation predictor on both the CPU and the GPU, and to the stills predictor on the CPU, but not to the stills predictor on the GPU - which the factory selects for stills whenever a card is present. So the correction was applied on a machine without a GPU and silently skipped on every machine with one: 6.6% apart on the geometry the tests use, 23.9% at 18 keV through a 450 micron sensor. The parity test that should have caught this is the reason it was not caught: it compares the predicted position and resolution of every reflection and nothing else, so a factor carried alongside them is invisible to it. It now compares the prescaling factor as well, with an assertion that the factor is not uniformly one so the comparison cannot pass vacuously, and the rotation path - which had no parity test at all - gets the same one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 14:29:22 +02:00
leonarski_fandClaude Opus 5	db9cc9106f	rugnux: the per-reflection correction factor is named for what it is, not for what it once held ... The factor multiplied into each integrated intensity was called rlp, for reciprocal Lorentz-polarization, and until this week that is all it held. It now also carries the sensor efficiency at the angle the beam arrives, and on the stills path it holds that efficiency and the polarization with no Lorentz term at all - correctly, since the Lorentz factor of a still is one. Three different products under one name that promises exactly one of them, in code where the neighbouring member is the total correction. Rename it prescaling_corr: multiplicative, applied before scaling, therefore not a scale, and silent about its contents - which is the point, since the contents have now grown twice. It is also what DIALS calls the same product. The stills refinement member spelled "1 / rlp" becomes inv_corr, and the comments and usage text that promised "the Lorentz-polarization factor and nothing else" now say what is actually there. The Lorentz term keeps its own name where it is computed, because that name is correct. The two external spellings are untouched: the CBOR key and the reflection dataset are a published format, and a reader that meets an unknown key would take the factor as zero, which both the merge key and the ingest treat as a reflection to drop - so every reflection would vanish and the run would still exit zero. No output changes: the merged and unmerged files of two full runs are byte for byte what the previous binary wrote, four stored files from before the efficiency correction still re-scale identically, and the reflection datasets of the process file are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 13:41:16 +02:00
leonarski_fandClaude Opus 5	d6ccdbdb61	rugnux: the report says what it found before it says how it found it ... The report was written in the order the pipeline runs, so a user reading one had to reach line 380 before meeting the first evaluative statement, and the warnings were last. Every run was 314 to 407 lines whether it succeeded or failed, about 200 of them fixed prose. The anisotropy section printed 21 keys and announced DETECTED (strong) on 54% of all runs - "strong" is the statistical confidence, which a reader takes as the severity - and reported a censored fit, which is survival-analysis vocabulary for good news. Assemble the report into a document and render from it, rather than streaming it out as the pipeline goes. That is what allows a verdict to sit above the evidence it was drawn from: SUMMARY carries VERDICT, one sentence of plain text, the warnings and ten facts, and it is composed after the sections that produce them. A clean run is 188 lines, a failed one 93, and the verdict is on line 25 in both. Warnings now carry a closed pathology vocabulary alongside their free text, so a consumer can switch on the code and a reader still gets the sentence. Everything removed from the default report is still written under --developer, which also carries the internals worth having when diagnosing the program rather than the crystal: the gate keys behind the anisotropy verdict, the operator and candidate tables, the sweep and spindle internals, and the essays. Nine statements the report made that were not true are fixed here as well. Among them: --mode scale printed a detector tilt of exactly zero on tilted data, which is worse than printing nothing because nothing about it looks wrong; every --mode scale run carried a "No image indexed" warning, because a key that is absent and a key measured to be zero were the same value; and a fitted resolution was asserted past the point where the run's own table shows CC1/2 at zero. REPORT_VERSION is 8. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 13:40:59 +02:00
leonarski_fandClaude Opus 5	9b96906150	integration: the parallax variance takes its attenuation length from the tables, not from lambda^3 ... The radial parallax broadening is the variance of the depth at which a photon converts, so it scales with the attenuation length. That length was approximated as photoelectric-dominated and scaled by lambda^3 from a single 13 keV reference per material. For silicon above 10 keV that is within 0.2%, but for CdTe it overstates the attenuation length by up to a factor of two, and by six above the cadmium K edge - which made this variance term 1.9x too large on 750 um CdTe data. Take the length from the same tabulated coefficients the efficiency correction uses. Silicon data is unaffected to within the approximation's own error; CdTe data gets a spot-width variance that matches the sensor it was recorded on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 11:07:17 +02:00
leonarski_fandClaude Opus 5	efa8ab1221	integration: each reflection is corrected for the sensor's efficiency at the angle it arrives ... A photon entering a flat sensor at an angle alpha to its normal crosses t/cos(alpha) of material instead of t, so the absorbed fraction rises toward the detector edge. The correction is QE(0)/QE(alpha) taken on the diffracted-beam direction against the detector normal, not on the scattering angle, so it follows a tilted or swung-out detector rather than assuming the two coincide. On an untilted detector this is a function of |s| alone: it is 99.7% a Wilson B offset and cancels exactly within a resolution shell, so merged protein data barely moves and no gain is claimed. It stops cancelling the moment the detector is tilted, because the incidence angle then acquires an azimuthal dependence: on a 30 degree swung-out geometry at 18 keV the within-shell spread reaches 21% median and 31% peak, and the anisotropy tensor moves with it. Attenuation lengths are the tabulated NIST coefficients rather than a wavelength-cubed approximation, which is within 0.2% for silicon above 10 keV but wrong for CdTe by a factor of two, and by six above the cadmium K edge. Photoelectric branching cancels in the ratio; K-fluorescence escape is not modelled, and the header says so. The correction self-disables where the physics makes it meaningless - an opaque sensor - so it needs no flag and is exactly neutral on all long-wavelength data and on thick CdTe. That also makes it a no-op on a file that stores its sensor thickness in the wrong unit, of which the corpus holds one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 11:07:01 +02:00
leonarski_fandClaude Opus 4.8	77a8a07c51	docs: 20 scouted external datasets recorded, credited, and reconciled ... Add the datasets pulled from the archive-wide PDB raw-data scout to the external-test-data page: new table rows (each with a DataCite-verified DOI, RCSB-deposited SG/cell/resolution, and the detector read from the image file), their multi-collection layouts, two more image-vs-PDB detector conflicts, and the two small-molecule reference cells. Multi-crystal wedge and non-native sets are excluded. Credit the two new repositories the data came from - MXRDR (ICM Warsaw) and the ESRF data portal - and bring the SBGrid and Zenodo counts in line with the table. All per-repository and per-section totals reconcile against the row count (95 PDB-coded + 7 without = 102). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DSMszqvyZUb6FHnSSS5nXY	2026-09-04 19:38:14 +02:00
leonarski_fandClaude Opus 5	2adf5d2e28	docs: the spindle severity documented as the worst-case trigger it is ... Section 5.5 of the indexing documentation describes the per-image score as shipped: the folded formula, the geometric cone width, the pair-normal recovery of an axis too long to see, and the three trigger states with the rule that CANNOT SAY must engage. It states plainly that the score is a worst-case bound under an assumption of no symmetry rather than an estimate - an axis of order three or higher perpendicular to the spindle in fact repairs the cone, which a still cannot know - and gives the engagement rates actually measured instead of the decoy-null figure, which showed only that the estimator does not hallucinate rows and was never a false-alarm rate over harmless mountings. The run-level orbit-based fraction is pointed to as the exact measure available once the group is known. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	307987c865	rugnux: the mounting's cost is computed exactly from the measured group, not guessed from an angle ... Offline, a merged rotation run has what a still lacks - a determined point group and an exact indexed orientation - so the run-level number no longer needs the pessimistic presumed-diad bound, and it no longer uses the nearest-axis angle either. That 15-deg warning heuristic was wrong in both directions: an aligned in-plane 2-fold of a dihedral group is repaired by the principal axis, a cubic group is never severe in any orientation, and a lone diad perpendicular to the spindle is severe with no axis anywhere near the spindle at all. The group's proper rotations are applied to the sweep's blind double cone in the crystal's actual orientation, and what no operator maps onto measured territory is counted, weighted by each shell's own cone width so the result is a fraction of unique reflections to this run's resolution limit. Friedel and the improper operators need no separate handling - the cone and the measured region are both inversion-symmetric. The number is machine-readable on purpose: SPINDLE_LOST_UNIQUE_FRACTION in the report (0-1, a bare number a pipeline can act on) and /entry/MX/spindleLostUniqueFraction in the master, with the warning prose only on top of it, fired when the group recovers less than half the cone's content. REPORT_VERSION stays 7: the format's own rule is that adding a key does not move it. This also settles what the nearest-axis keys hedged: with the measured group the mounting is cleared or convicted exactly, so their documentation now calls them descriptive and points at the new key for the verdict. Verified against Monte Carlo: P1 loses 2.0% of unique reflections at theta_max = 15 deg with nothing repaired; a lone diad on or perpendicular to the spindle repairs nothing; an axis of order >= 3 perpendicular to the spindle repairs everything; 622 with an in-plane diad on the spindle loses nothing; cubic loses nothing in any orientation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	26a82ddc72	spindle: the per-image severity travels the whole data path, and absence travels with it ... The score was computed and then thrown away - carried on the per-image message but transported nowhere - so the automation it exists for could not read it. It now flows like bkg_estimate at every layer: CBOR (per-image key, END-block run mean and per-image array), HDF5 (per-image /entry/MX/spindleBlindFraction in the data files, the array and spindleBlindFractionMean in the master), read-back into a re-opened dataset, the scan result, the receiver plots, the REST plot and scan_result schemas, and the viewer and frontend plot menus. Absence is load-bearing and every transport keeps it distinguishable from a measured zero: the CBOR key is simply missing, the HDF5 array holds NaN, and read-back turns NaN back into an absent optional rather than a value. A pipeline that read 0 where the truth is "no value" would take exactly the wrong action - a measured 0 says one sweep loses nothing, absence says nobody could look, and the second must engage the recovery protocol while the first must not. The round-trip tests pin all three states through CBOR and through a written-and-reopened file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	441b802e0c	docs: a large spindle-to-symmetry-axis angle does not clear the mounting ... The paragraph introduced with the two new report keys said an incomplete cusp could be "attributed to the mounting or cleared of it". The first half holds; the second does not. A symmetry axis within theta_max of PERPENDICULAR to the spindle carries the blind cone onto its opposite lobe, which a sweep leaves equally unmeasured, so a large angle is not on its own evidence that the mounting was harmless - and the key reports only the nearest axis. Also record the two keys in the changelog, and correct REPORT_VERSION there: the entry still said 6 after the bump to 7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	555c686205	merge: bridge a rocking event by an angle rather than by a frame count ... A rotation reflection arrives as a run of partials and is cut into rocking events wherever two successive frames are more than MAX_FRAME_GAP apart. That constant was 2.0 frames, in five places, and the quantity it stands for is not a frame count at all: it is an angle, a reflecting range of a tenth to half a degree. At a tenth of a degree per image two frames is a fifth of a degree of dead rotation and the bridge does what it was meant to; at a degree per image it is two degrees, as wide as a whole event, and two genuine crossings of the Ewald sphere are joined into one "full". Reconstructing the events on a paired control - the same photons sliced two ways - the fraction of events whose partialities sum past 1.5 is 14.3% at 1.0 degree against 6.0% at 0.1, the fused ones spanning a median of five frames. So the gap becomes half a degree of rotation, floored at one frame so an event is never cut at its own neighbours and capped at the two frames that were always allowed. For any oscillation of 0.25 degrees or less the quotient is at least two and the cap returns the literal 2.0f, so every finely sliced sweep is bridged exactly as before, on both the CPU and the GPU path. The value is taken once per run and carried to the device in CombineParams rather than recomputed in the kernel, so the two paths compare the same float by construction and the bit-parity contract the combine documents is preserved. The defect above is measured; the repair is not. No arm has been run with this change, which is why it is a commit of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:09:08 +02:00
leonarski_fandClaude Opus 5	91775cc4fb	geometry refinement: at coarse slicing, forgive the miss the exposure could have supplied ... The acceptance gate that decides which spots enter the geometry refinement compares a fractional-index miss against a fixed tolerance, and that comparison assumes the spot diffracted at the frame's midpoint. It did not. The rotation coordinate is bracketed, not observed - the spot diffracted somewhere inside the exposure - and profiling that unknown angle out of the least squares has a closed form. Writing u = m x q for the direction a rotation delta moves the observation along, the component of the miss along u is free up to the exposure's rms half-width and only the excess is charged; no other direction is touched, so |q|, and with it every d-spacing, is unaffected. Without it the gate is a resolution cut that tightens with the frame width, the miss growing as a/d. Where the benefit lives is the per-frame refinement, which holds one frame and therefore deliberately does not back-rotate - the frame's angle is a gauge its orientation block absorbs - and passes no goniometer at all. The spots on that frame still span the exposure, and that spread does not go away with the gauge, so the spindle and the wedge are handed over separately, for the gate alone. The spindle is also written into the rotation vector the frame constants carry, which was left at the lab x axis whenever no goniometer was passed; with a zero angle that vector is multiplied out and the block is held constant, so nothing changes by it today, but it was a trap for the next reader. The width is k = 1/sqrt(12), the rms of a rotation coordinate uniform over the frame, which is what a least-squares is calibrated on. Half the exposure - the worst case a spot could sit at - degrades accuracy against a ground truth, and so does forgiving the direction outright with no bound: the bound is load-bearing, and what it protects is fine slicing. Measured on 112 paired datasets, scored against deposited structure factors because neither ISa nor R_meas can arbitrate this - both move opposite to accuracy along the neighbouring partiality knob. At 0.5 degrees per image or coarser, n = 16: CC to the deposited data better on 9 of 12, median dISa +0.87%, and at the top of the a x oscillation lever better on 9 of 10 with the per-shell CC improving in all ten shells, most in the outer half. The win rises with that lever and the population below it is inert. It is applied only at 0.5 degrees per image and above. The band is principled - the dead zone matters once the exposure's rms rotation ambiguity, 0.29 of the wedge, is comparable to the crystal's own along-u rocking spread, which measures around 0.26 degrees, putting the boundary between 0.25 and 1.0 degrees - while the point inside that band is empirical and taken at the conservative end. It has to be there: the two datasets the corpus lost outright both sit at 0.2 degrees or finer, one of them finely sliced, and the response is not monotone in the width, so no bound on the width alone makes them safe. Below the trigger the block is skipped and the gate computes what it always computed; a finely sliced sweep reprocessed across this change is byte for byte identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:08:31 +02:00
leonarski_fandClaude Opus 5	8b88074001	rugnux: the joint post-refinement's gate is asked of the geometry it commits ... Five corrections to the joint fit, none of them meant to move a well-conditioned result. EffectiveCellFromParams used M_PI, which is not standard C++ and which MSVC does not define without _USE_MATH_DEFINES. image_analysis/ is on the viewer's dependency path, so it takes PI from JFJochMath.h like the rest of the tree. The bounds test ran on the fit-half solution, and what the run actually commits is the re-fit on all the reflections that follows it. That second solve moves the answer, and nothing re-asked the bounds of it, so the committed distance was held only by the solver's +-5 % box rather than by the 1 % the gate advertises. The test is now a function of a candidate geometry and is asked twice - once of the half that earns the right to fit, once of what will be written - and a re-fit that lands outside leaves the run at its header geometry, named in the log like the other refusals. It is inert on the corpus: over the 98 committed fits the largest committed move is 0.709 % of the distance, 0.885 % of a cell length and 1.07 px of the beam, so nothing that commits today stops committing. The bounds covered the distance, the beam and the cell lengths and left the cell ANGLES - beta for monoclinic, all three for triclinic - free to the solver's +-0.05 rad, which is +-2.86 deg. The uniform scale this replaced could not move an angle at all, so that was a new unguarded degree of freedom on the two systems whose conditioning is weakest. The bound is one degree, chosen from what the fit actually does: the largest angle it moves anywhere on the corpus is 0.35 deg, on a monoclinic beta, and the two triclinic cases with the largest moves are 0.09 and 0.07 deg. A degree is about three times the worst of those and still well inside the solver's box, so the box goes on reaching everywhere the gate accepts and no fit that commits today is refused. The gate pooled the two residual families into one held-out mean. With both caps saturated that is some 30000 positional values against 10000 excitation ones, so the excitation residual - the only thing in the problem that identifies the cell SCALE - is outvoted three to one in the decision that commits the cell. Neither family may now degrade. This one is not free. It refuses 1 of the 98 fits the corpus commits, a cubic case whose held-out excitation rose 15 % while its cell scale moved 0.75 %; refused, that dataset keeps its header geometry and lands where the two-step arm did, R_meas 0.2699 -> 0.2724 and ISa 3.15 -> 3.10 with the space group unchanged, and its cell deviation from the reference goes back from 0.05 % to 0.40 %. That is the shape of failure the split is meant to catch, and on the one dataset where the two disagree it costs a real if small improvement. The changelog claimed the refined distance "no longer depends on how wrong the file's distance was". It does. Swept over a millimetre of header distance the committed distance still moves monotonically and the merge degrades with it; the dependence is reduced about thirteenfold, not removed, and the line now says reduced. Finally the log printed the half-data fit's distance, beam and cell beside the word COMMIT while different numbers were committed - 107.908 mm printed against 107.913 mm applied two lines further down, on one sweep. It now prints whichever geometry the verdict is about, and prints the cell angles beside the lengths, since the angles are refined now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:00:24 +02:00
leonarski_fandClaude Opus 5	bb2a42774a	rugnux: the rotation post-refinement fits the crystal and the detector at once ... Post-refinement scaled the pass-1 lattice by one scalar against the observed rocking angles (step A), then read the detector distance off that scaled cell (step B). The split was there because the positional residual is degenerate with the cell scale - which is true of the positions ALONE, and is exactly what the excitation residual already computed in the same struct breaks. So the degeneracy that motivated the split was resolvable inside one problem all along, and splitting it manufactured error twice over: * Pass 1 frees the whole lattice against a frozen distance, so what it absorbs of a header distance error is ANISOTROPIC (0.16 % on the crystal measured), and no single scale can undo it. * Whatever bias is left in that scale goes straight into the distance, since the distance is only ever determined relative to the cell. Over six runs spanning three wavelengths of one crystal, distance error (%) = 1.33 x step-A bias (%/axis), r = 0.9947 - and the bias changed SIGN with the wavelength (-0.067 % at 1.9 A, +0.049 % at 2.7 A, +0.19 % at 3.3 A), reproducing the whole wavelength ordering of the distance error. The two steps' own printed numbers say it, on 112 of the corpus's datasets. Comparing each run's held-out POSITIONAL residual at the unrefined cell with the same residual at step A's scaled cell and at step B's committed geometry: step A leaves it WORSE than the unrefined cell on 110 of 112 (median 3.41x), and step B - which does fit that residual - cannot get back: its committed value is still worse than the unrefined cell on 110 of 112 (median 2.97x). Step B was improving on a baseline step A had corrupted. This part of the measurement is of the old code alone and owes nothing to the replacement. Now one Ceres problem: free crystal orientation, the cell (every parameter the crystal system leaves free), goniometer axis, detector distance and beam centre; residuals are the positional detector<->reciprocal one at each partial's observed spot and the distance-independent Ewald excitation one at each rocking centroid. Same deterministic hkl-hash split-half gate as before, with the move bounded: distance and every cell length within 1 %, beam within the existing bound. Detector tilt stays fixed (gauge-coupled to the orientation on one crystal). Committed, it leaves the held-out positional residual BELOW the unrefined cell's on the same 110 of 112 (median 0.90x) - though that residual is half its own objective, so read that line as a sanity check, not as an independent arbiter. The rotation-scale fit that followed step A now runs on the jointly committed lattice and axis; it is unchanged otherwise and reproduces its previous k to five digits. MEASURED. Three wavelengths of one rotation crystal, scored against an independent determination of the cell (unconstrained joint fit to XDS spot centroids) and against XDS's own refined cell: set cell error vs XDS committed distance R_meas ISa 1.9 A -0.048 % -> -0.041 % 60.274 -> 60.279 mm .0725/.0725 31.2/31.1 2.7 A +0.057 % -> -0.058 % 60.360 -> 60.266 mm .0681/.0676 25.8/26.3 3.3 A +0.230 % -> -0.033 % 60.491 -> 60.267 mm .1193/.0835 9.2/14.9 The three committed distances now agree with each other to 0.013 mm and with the external value (60.26 +- 0.04) instead of drifting 0.22 mm with the wavelength. Anomalous peak heights at the known sulphur sites rise from 7.82/8.18 to 10.09/9.04 - the best previously reachable state was 10.08/9.10 and required knowing the answer. The bistability documented on the long-wavelength set is gone. Sweeping the header distance over 1.0 mm, the two-step commits 60.49-60.54 however it starts (or refuses outright, losing the dataset: R_meas 0.276, ISa 2.9); the joint fit commits 60.256-60.313, tracking the truth, and lands in the correct cell basin every time. Same from two starting beam centres. CORPUS. 115 datasets, paired against the tip, 113 processed in both arms: * cell deviation from the reference: 58 better, 38 worse, 16 unchanged (sign test p = 0.052); median 0.171 % -> 0.128 %. The asymmetry is in the magnitudes: the largest improvement is 2.4 percentage points, the largest regression 0.125. The 24.4-point figure at the head of that list is not a cell change at all - it is a hexagonal lattice the base arm described in its C-centred orthorhombic setting, which a sorted-axis comparison reports as a large difference; the scorer's own cause for that dataset is an under-called point group, and the primitive volume ratio passed in both arms. * POINT GROUP == reference 105 -> 106. Exact space group 82 -> 81: one screw-axis call (P2 -> P2_1) on a crystal whose R_meas is above 139 %, where the cell itself improved. * V ratio in [0.97, 1.03]: 106 -> 106. Datasets processed: 113 -> 113. * R_meas and ISa are confounded here: of the nine material regressions, seven reached a FINER resolution limit, which raises R_meas and lowers ISa mechanically. The two that did not are small (+0.55 pp R_meas, -0.9 ISa). One unconfounded rescue is large - R_meas 105 % -> 27 % at 0.72 -> 0.51 A on a small-molecule set whose uniform scale step A had refused. Cost: 90-150 ms on runs of minutes. Bit-identical across repeat runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:00:24 +02:00
leonarski_fandClaude Opus 5	025ecf066e	report: the spindle-to-symmetry-axis angle reaches p_report.txt, not only stdout ... Every rotation run already computes the angle between the spindle and the nearest symmetry axis, and writes it into the merge statistics text - which is stdout only. Unless the angle crossed the 15 deg warning threshold it therefore did not survive the run, so the ordinary "the mounting was fine" case could not be greppped, and an incomplete cusp could not be attributed to the mounting or cleared of it after the fact. Section 4 now carries the angle and the axis order as keys on every run that determined a space group. REPORT_VERSION moves to 7. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-03 23:57:21 +02:00
leonarski_fandClaude Opus 5	491167c90d	rugnux: a Bravais class the reduction decided by rounding is re-asked on the metric's own cell ... The class is named by Niggli-reducing the indexed cell and looking the reduced cell up in the 44 lattice characters, and that lookup is a coin flip for any lattice whose Buerger cells straddle the Niggli type-I/type-II boundary. An F-centred cubic lattice does so by construction: it has reduced forms on both sides, the two sides carry different characters, and which side the reduction lands on is set by the last digits of whatever refinement produced the cell. Measured over 600 perturbations of one such lattice: 43% cubic F, 36% tetragonal I, 21% orthorhombic I, and the split is flat over a factor of ten in the noise. The class then caps the point-group search, so from the body-centred sub-cell the cubic three-fold is never enumerated and the run reports that nothing was refused - which is accurate, because nothing was asked. Le Page's two-fold search has no such key: it measures each rotation's obliquity on the lattice itself, in a primitive basis. LePageLattice turns the rotation group it finds into a conventional cell, a centring letter and an integral change of basis, and where that group is larger than the adopted class's holohedry the merge is reindexed into that cell and the space-group search is run again there, on both merges, with every gate live. Nothing here decides: the reindex is committed only where the search in the new setting confirms a strictly higher point group AND the centring the new cell describes, so a pseudo-symmetric metric leaves the answer already in hand standing. Measured as a paired battery over 113 rotation datasets: the re-ask fires on 7 and adopts on 1, and that one crystal - an F-centred cubic lattice the reduction had named body-centred tetragonal - moves to its deposited group, gaining 0.10 A of resolution and 2.8x the multiplicity at R_meas 0.117 -> 0.120. Nothing else moves, in space group, resolution, CC1/2, R_meas, multiplicity, I/sigma or completeness. A second such crystal, named body-centred orthorhombic, is offered the same cubic cell and confirms all 23 added operators at CC 0.96 with an H ratio of 1.00, and is still refused, on the merge chi^2 ratio at 2.50x a bound of 1.85. Letting H rescue that refusal is the one-line change an earlier round measured and rejected - it promotes the synthetic P 4_3 2_1 2 in the test suite to point group 432 - so it is not here, and that crystal is left where it was. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-03 13:53:14 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_f	61c603b274	v1.0.0-rc.163 (#73) ... * Packaging: each package installs its license notices under a directory of its own - `share/doc/jfjoch_broker`, `share/doc/jfjoch_writer`, `share/doc/jfjoch_viewer`, `share/doc/jfjoch_driver_dkms` - instead of all of them into the shared `share/doc/jfjoch`, so `jfjoch` and `jfjoch-writer` can be upgraded one at a time instead of only together. Reviewed-on: #73 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 13:38:29 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	6136f858af	v1.0.0-rc.154 (#64) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability) Reviewed-on: #64 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-25 18:12:00 +02:00