Commit Graph
11 Commits
Author SHA1 Message Date
leonarski_fandClaude Opus 5 cfcb84aece docs: bring the tool pages back in line with the programs
rugnux gained --model - R-free and 2Fo-Fc/Fo-Fc maps against an atomic model, and with it
the resolution of the enantiomorph and of a merohedral indexing ambiguity - without the
page ever mentioning it. It was the only option missing; the two lists now agree in both
directions, checked against the usage the binary prints.

The viewer page still said results are never saved and that no Windows package exists.
Both have been false for a while: the Processing panel runs full rugnux jobs on the open
dataset, writes _process.h5 and the merged reflections, registers each run as a
selectable view so runs can be compared, and can hand out the equivalent command line for
a cluster; and the installer is published with every release. The mask menu also loads
TIFFs now, and the View menu has layout presets.

The writer page documented -R for the root directory, which is the back-compatibility
alias for -d, and an HTTP status interface that no longer exists - status reaches the
broker over the writer notification socket, and a writer is stopped with a signal.

The test page pointed at .gitlab-ci.yml and at jfjoch_offline_process, which is not a
binary any more; the CrystFEL fixture pointed at HDF5DatasetWriteTest, which is not
either. The broker page linked ../broker/redoc-static.html, which MyST resolved by
copying the 700 kB file into _downloads/ rather than using the copy already in _static.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 20:53:37 +02:00
leonarski_fandClaude Opus 5 0ca159449f Bragg integration: integrate as far as the detector reaches, not to a fixed 1.0 A
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m28s
Build Packages / build:viewer-tgz:cuda (push) Successful in 7m50s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m44s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m30s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m9s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m5s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m1s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m39s
Build Packages / build:rpm (rocky8) (push) Successful in 10m52s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 10m50s
Build Packages / build:rpm (rocky9) (push) Successful in 11m45s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m11s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / Build documentation (push) Successful in 53s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (durin plugin) (push) Successful in 7m16s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m22s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m56s
Build Packages / DIALS test (push) Successful in 10m58s
Build Packages / Unit tests (push) Successful in 1h2m58s
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
BraggIntegrationSettings::DMinLimit_A had a setter that nothing anywhere called, so
it was always its 1.0 A default - in rugnux, the viewer and the broker alike, with
no option or API field to change it. It feeds the predictor as high_res_A, which
discards any reflection with |q| > 1/d_min, so integration simply stopped at 1.0 A
however far the detector reached.

Five of the 33 rotation test datasets have detectors reaching past it, down to
0.981 A. On one of them, run with no resolution limit, the shell table ended dead
at 1.00 A with that shell still at CC1/2 55.6% and <I/sig> 3.4 - cut mid-shell
rather than fading out. This branch had already made the sibling limits
detector-driven (spot finding, scaling), so the pipeline was finding spots the
detector could see and then refusing to integrate them.

Make it a std::optional: unset means as far as the detector reaches, a value limits.
The limit is only a bound on how far the lattice walk goes, never a second opinion
on what is measurable - both predictors independently drop reflections that miss the
detector (BraggPrediction.cpp, BraggPredictionRot.cpp) - which is what makes the
detector's own reach the right default. rugnux gains --integration-high-resolution
(0 = no limit, as for --spot-high-resolution); the derived per-axis prediction range
resolves against the same number, so the two cannot drift.

Full battery: 30/33 space groups, unchanged from before, 0 failures and the same
three known mismatches; 22 of 32 crystals bit-identical and nothing worse than 5
observations in ~500k. The datasets that gain do so because their detector reached
past 1.0 A - the effect is understated here because the harness caps each merge at
the XDS resolution anyway.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 14:03:34 +02:00
leonarski_fandClaude Opus 5 b0e315e73c Bragg prediction: derive the lattice walk from the cell, and expose it in the API
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m15s
Build Packages / build:viewer-tgz:cuda (push) Successful in 8m42s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m47s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m1s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m11s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m19s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m31s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m39s
Build Packages / build:rpm (rocky8) (push) Successful in 11m41s
Build Packages / XDS test (durin plugin) (push) Successful in 7m52s
Build Packages / Generate python client (push) Successful in 34s
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m10s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m0s
Build Packages / build:rpm (rocky9) (push) Successful in 13m39s
Build Packages / DIALS test (push) Successful in 14m29s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m33s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m25s
Build Packages / Unit tests (push) Successful in 1h36m21s
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
Follow-up to making max_hkl a setting: it is now an optional, and unset means "take
it from this crystal". The predictor keeps only |q| <= 1/d_min and h = a.q for the
real-space axis a, so |h| <= a/d_min exactly - and likewise |k| <= b/d_min and
|l| <= c/d_min. max(a,b,c)/d_min therefore bounds all three at once: nothing that
could be predicted lies outside it, and nothing inside it is reached by a shorter
axis. It applies to rotation and stills alike, both going through the one place the
prediction settings are built.

Offline (rugnux, viewer) the default is unset, so every crystal gets its own range;
--max-hkl overrides it. Online the broker holds a concrete number, because the cost
is the cube of it per image and a live acquisition should not have its frame rate
decided by whichever sample is mounted: max_hkl joins bragg_integration_settings in
the OpenAPI with a default of 100, so an omitted field arrives as that default (the
generated model carries it) rather than as "derive it", and the frontend exposes it
next to the integration model.

Measured against a fixed 100 on six rotation crystals: three are bit-identical, two
were being truncated and recover 419k and 5.8k observations with the high-shell
CC1/2 going 15.1 -> 25.8% and 52.1 -> 55.3%, and the space group is unchanged 6/6.
It reproduces a fixed 200 exactly, which is the bound being tight rather than merely
safe.

The sixth is worth recording: a 149/83/226 A cell derives 227, and because a single
scalar has to cover the longest axis the cube is ~16x what a per-axis box would be -
22% wall clock, for a net 22 observations out of 364k (the per-frame 65536-reflection
cap re-selects at the margin when more candidates are offered) and identical CC1/2,
ISa and space group. Per-axis limits would remove that; the predictors already map a
thread index to h, k and l separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 13:10:58 +02:00
leonarski_fandClaude Opus 5 a06c06931f Bragg prediction: how far to walk the lattice is a setting, not a literal
max_hkl was hardcoded to 100 at the one place production builds the prediction
settings, so the only way to change it was to edit and rebuild - and it is not a
constant of the method, it is a property of the cell. An axis is truncated once
a/d_min exceeds it: 100 covers a 150 A axis at 1.5 A, but the same axis at 1.0 A,
or a 250 A axis anywhere, loses its outermost reflections with nothing said.

Move it into BraggIntegrationSettings next to the other prediction/integration
parameters and add rugnux --max-hkl (1..511, default 100 - no behaviour change).
Like the integration radii and the background trim it stays out of the OpenAPI, so
the broker keeps the default it has today and live analysis cannot be handed a
range that would not finish; the offline front end, which knows its cell, can ask
for more. RugnuxCommandLine emits it when it is not the default.

Measured on five rotation crystals at --max-hkl 200: two are bit-identical at no
cost, and three were being truncated - one gains 419k observations (+17%) and
takes its high-shell CC1/2 from 15.1% to 25.8% for +14% wall clock, the other two
gain 12k and 5.8k observations with CC1/2 76.6->82.4% and 52.1->55.3% for +9% and
+1%. ISa is unchanged throughout, and no frame overflowed the prediction buffer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 12:00:45 +02:00
leonarski_fandClaude Opus 5 3b9e1424ea docs: RUGNUX.md on adaptive detection and --search-min-zeta
Adaptive spot detection became the default for rotation data as well in
6f4917dce; RUGNUX.md still said rotation kept the fixed-threshold finder, which
is also the opposite of what the usage message and CPU_DATA_ANALYSIS say.
--search-min-zeta had no entry in the option tables at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 11:23:11 +02:00
leonarski_fandClaude Opus 5 0f1851cf82 rugnux: drop the 1.5 A spot-finding limit on rotation data
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
Build Packages / build:viewer-tgz:cpu (push) Successful in 8m16s
Build Packages / build:viewer-tgz:cuda (push) Successful in 9m21s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m55s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m9s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m19s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m26s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m31s
Build Packages / build:rpm (rocky8) (push) Successful in 11m51s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 13m17s
Build Packages / XDS test (durin plugin) (push) Successful in 7m37s
Build Packages / Generate python client (push) Successful in 29s
Build Packages / Build documentation (push) Successful in 1m9s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m33s
Build Packages / build:rpm (rocky9) (push) Successful in 13m37s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m40s
Build Packages / DIALS test (push) Successful in 14m11s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m59s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m39s
Build Packages / Unit tests (push) Successful in 1h1m40s
Rotation kept a 1.5 A high-resolution limit for spot finding on the
strength of one indexing-rate measurement (100.0 -> 96.8% on a strong
set). Measured properly, over the whole 33-crystal rotation battery, it
does not earn its place:

  * no space-group decision changes - the same 30/33 agree with XDS, and
    the three that differ are the known pre-existing cases, unchanged;
  * 29 of 33 crystals are identical to the digit - same indexing rate,
    R_meas, CC1/2, ISa. The limit was doing nothing on the large
    majority;
  * where it does bite, the limit is the worse setting. The one crystal
    that loses appreciable indexing rate without it (99.50 -> 94.22%)
    comes back with lower R_meas (29.4 -> 28.1), higher high-resolution
    CC1/2 (27.9 -> 29.1) and higher ISa (5.77 -> 6.17). Another loses
    0.4% of frames and gains 2.8 points of CC1/2_hi. Fewer frames
    indexed, better data from them;
  * runtime is unchanged (16m46s vs 17m32s over the battery).

So the indexing-rate cost is real but does not carry through to the
merged data, which is what the limit was protecting. Unset now means "as
far as the detector reaches" for rotation as well as stills;
--spot-high-resolution still sets a limit for weak, high-background data
where the extra high-resolution spots are genuinely noise.

This also removes the flag that distinguished "the user asked for no
limit" from "the user said nothing" - with no rotation default left,
both mean the same thing. While rewriting the comment block, corrects
its neighbouring claim that rotation keeps the fixed-threshold finder;
adaptive detection has been the default for both workflows since
6f4917dce.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 10:39:23 +02:00
leonarski_fandClaude Opus 5 16bf3408f0 Address code-review findings; make detection limits detector-driven
One changeset, developed together in response to a review of this branch, so the
files carry several of the changes at once. Full test suite passes (733 cases).

Spot finding
- Split ImageSpotFinder into Detect() (flag strong pixels - the expensive
  per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with
  Run() = both. The per-image min-pix escalation now detects ONCE and repeats
  only the cheap extraction, instead of re-running the whole finder four times
  per frame as it did on the default path. It also keeps the winning attempt's
  spot list rather than re-extracting it, so the frame that is integrated is
  exactly the frame that was scored - which a GPU re-extract could not guarantee
  (float atomic ordering).
- spot_finding_time_s no longer swallows indexing time, and indexing_time_s now
  sums every escalation call instead of reporting only the last.

Detection limits follow the detector
- The azimuthal-integration upper q and the spot-finding high-resolution limit
  are now std::optional, in the C++ structs AND in the OpenAPI schema, and
  resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_
  recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a
  pixel outside that q range could never be strong - the integration range
  silently bounded what detection could see, regardless of the requested
  resolution limit. Regenerated the C++ and TypeScript clients; the viewer and
  the web frontend each gained a "to detector edge" switch.

Detection defaults are now per workflow (measured, not assumed)
- Stills: adaptive detection, min-pix chosen per image, no resolution clipping.
- Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit.
  On a 33-crystal rotation battery, adaptive detection helped four hard crystals
  but deterministically broke three (a lost space group, a halved indexing rate,
  a collapsed merge), and the detector-edge limit cost indexing on a strong
  rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and
  --no-adaptive-spots is new.

Indexer seed escalation
- Stop escalating once a seed's lattice explains >= 90% of the seed spots.
  Previously any frame with >= 80 spots always paid three indexer calls, online
  broker included.

Merge-consistency filter
- --min-image-cc gated on a per-image CC computed BEFORE the stills partiality
  post-refinement and never refreshed; the refiner now recomputes it, so the
  reported CC describes the data that are actually merged.
- Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the
  merge, the error model and MergeStats can no longer disagree about which
  images are in (the --scale path merged unfiltered while its statistics were
  filtered).

Per-image B-factor refinement (-B) removed
- Measured on four serial-stills datasets: it is a no-op where the per-image fit
  is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas
  +23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on
  14-25% of images). It had also been silently DISCARDED since the partiality
  post-refinement landed - reported but not applied. Rather than fix and keep a
  knob with no demonstrated benefit, the flag and the whole image_scale_b_factor
  chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and
  read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs.
  ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS.
  (The Wilson per-image b_factor is a different quantity and stays.)

Stills partiality width now fits both of its components
- sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d*
  with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2
  on d*^2. The angular-only width is fitted over a d*^2-dense population, so it
  was pinned by the high-resolution edge and collapsed at low d*: median
  partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55%
  of them under the merge's partiality floor, and the survivors divided by those
  values - which inflated the merged low-resolution intensity scale 3.6x
  (~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x,
  no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and
  R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining
  R-free are all blind to that ramp, which is why it survived earlier validation;
  the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged).

Removed dead code from add-then-remove churn
- Prediction-time "still partiality" (unreachable: no setter), the phantom
  IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the
  constant it always was), StillsPartialityRefine's caller-less Settings
  constructor and its reference to a long-gone env var, ProcessImage's unread
  bool return, an unused include, and a dead viewer overlay hook.

Also
- Viewer: the magnifier compared a QImage with itself, so its scene rect was set
  once ever and it could not pan into a larger dataset; the hover tail timer
  could fire after leaveEvent and resurrect the resolution readout outside the
  image.
- update_version.sh regenerated the frontend lock file BEFORE bumping the
  version (every release shipped an off-by-one lock), and did git rm/git add on
  a path that has not existed since the client moved to src/client - with no
  set -e, both failed silently.
- fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a
  redirect, not a test, so dkms add never ran.
- Unit tests for the adaptive-threshold host functions, which had none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 09:07:00 +02:00
leonarski_fandClaude Opus 4.8 a8c1006c49 Choose min-pix-per-spot adaptively per image for serial-stills indexing
For stills indexing the minimum-pixels-per-spot filter is now chosen per image
instead of being fixed: the frame is indexed at min-pix 3/2/1 and the setting that
maximises indexed-spot count weighted by indexed fraction (n_indexed^2 / n_total)
is kept, then integrated once at that min-pix. The fraction factor keeps a smaller
min-pix's extra spots only when the lattice actually explains them, so strong frames
retain their real weak spots (extending resolution) while noise-flooded frames stay
strict.

The mode is selected by the presence of --min-pix-per-spot, now optional
(SpotFindingSettings::min_pix_per_spot is std::optional<int64_t>): omit it for the
adaptive per-image path, give a value to force a fixed min-pix. It applies only to
the stills indexing path -- rotation indexing builds one global lattice and keeps a
fixed min-pix, and the online receiver and the FPGA host path always carry a concrete
value, so neither changes. IndexAndRefine::ProcessImage now returns whether the frame
indexed, to drive the per-image selection.

Exposed in the jfjoch_viewer spot-finding settings (adaptive-threshold and
adaptive-min-pix checkboxes, each greying out the control it overrides); the broker
uses neither.

Validated on the full rotation regression battery (no regression) and the whole
serial-stills target battery at full image count.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 18:31:35 +02:00
leonarski_f 67dca388bd v1.0.0-rc.160 (#70)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 18m44s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m11s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m54s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m40s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m41s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m10s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m4s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m23s
Build Packages / build:rpm (rocky8) (push) Successful in 11m30s
Build Packages / build:rpm (rocky9) (push) Successful in 12m51s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m8s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m21s
Build Packages / DIALS test (push) Successful in 13m22s
Build Packages / XDS test (durin plugin) (push) Successful in 9m2s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m55s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m57s
Build Packages / Generate python client (push) Successful in 23s
Build Packages / Build documentation (push) Successful in 57s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 10m24s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-07-19 09:39:28 +02:00
leonarski_f 451310f43d v1.0.0-rc.158 (#68)
Build Packages / Unit tests (push) Successful in 1h32m35s
Build Packages / build:windows:cuda (push) Successful in 18m0s
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m37s
Build Packages / build:viewer-tgz:cuda (push) Successful in 8m55s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m13s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m11s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m35s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m57s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m23s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m45s
Build Packages / build:rpm (rocky8) (push) Successful in 11m39s
Build Packages / build:rpm (rocky9) (push) Successful in 14m0s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m42s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m38s
Build Packages / DIALS test (push) Successful in 14m55s
Build Packages / XDS test (durin plugin) (push) Successful in 7m11s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m7s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m34s
Build Packages / Generate python client (push) Successful in 28s
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 9m55s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation.
* jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window.
* rugnux: Accept an absolute `-o` output prefix in offline processing.
* Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed).
* Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-07-12 19:42:29 +02:00
leonarski_f 54c0100e8e v1.0.0-rc.157 (#67)
Build Packages / Unit tests (push) Successful in 1h28m28s
Build Packages / build:windows:nocuda (push) Successful in 14m45s
Build Packages / build:windows:cuda (push) Successful in 13m13s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m47s
Build Packages / build:viewer-tgz:cuda (push) Successful in 7m22s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m52s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m16s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m19s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 12m50s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m40s
Build Packages / build:rpm (rocky8) (push) Successful in 11m18s
Build Packages / build:rpm (rocky9) (push) Successful in 12m4s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m55s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m22s
Build Packages / DIALS test (push) Successful in 13m37s
Build Packages / XDS test (durin plugin) (push) Successful in 8m47s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m4s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m45s
Build Packages / Generate python client (push) Successful in 34s
Build Packages / Build documentation (push) Successful in 1m4s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 7m16s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`.
* rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling.
* rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics.
* rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1.
* rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group.
* rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix.
* jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation.
* jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning.
* jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint.
* jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed).
* jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types.
* jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails.
* jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak.
* Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation.
* CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-07-11 07:19:11 +02:00