The openapi-generator invocation still passed --git-host=git.psi.ch and a user id of
jungfraujoch, from before the move to gitea.psi.ch/mx/jungfraujoch. Those properties are
not cosmetic: they become the source URL in the generated README and pyproject, so the
published client documentation - docs/python_client/README.md, which is copied out of the
generated tree - told readers to pip install from a host that no longer answers.
Regenerating with the corrected flags changes those two lines and nothing else, verified
against the committed tree.
update_version.sh, make_doc.sh and gen_python_client.sh were all mode 644, so the
"run update_version.sh" the documentation asks for fails on the shebang. CMake and the CI
both work around it by invoking them through bash.
make_doc.sh builds a throw-away venv in the working tree and deletes it on the last line,
which set -e skips whenever pip or sphinx fails - so a failed docs build left tmp_venv/
behind. Delete it from a trap instead, and ignore it along with the default output
directory and the sdist directory gen_python_client.sh creates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script now removes the generated C++ model, the frontend client and the
published python docs before regenerating, but not python-client/ itself - and
docs/python_client/docs is filled by copying that directory. So a schema dropped
from the API kept its generated model in the PyPI package and its .md page in
the published docs, linked from no index. Eight such pages are in the tree
today, JfjochSettingsSsl among them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One changeset, developed together in response to a review of this branch, so the
files carry several of the changes at once. Full test suite passes (733 cases).
Spot finding
- Split ImageSpotFinder into Detect() (flag strong pixels - the expensive
per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with
Run() = both. The per-image min-pix escalation now detects ONCE and repeats
only the cheap extraction, instead of re-running the whole finder four times
per frame as it did on the default path. It also keeps the winning attempt's
spot list rather than re-extracting it, so the frame that is integrated is
exactly the frame that was scored - which a GPU re-extract could not guarantee
(float atomic ordering).
- spot_finding_time_s no longer swallows indexing time, and indexing_time_s now
sums every escalation call instead of reporting only the last.
Detection limits follow the detector
- The azimuthal-integration upper q and the spot-finding high-resolution limit
are now std::optional, in the C++ structs AND in the OpenAPI schema, and
resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_
recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a
pixel outside that q range could never be strong - the integration range
silently bounded what detection could see, regardless of the requested
resolution limit. Regenerated the C++ and TypeScript clients; the viewer and
the web frontend each gained a "to detector edge" switch.
Detection defaults are now per workflow (measured, not assumed)
- Stills: adaptive detection, min-pix chosen per image, no resolution clipping.
- Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit.
On a 33-crystal rotation battery, adaptive detection helped four hard crystals
but deterministically broke three (a lost space group, a halved indexing rate,
a collapsed merge), and the detector-edge limit cost indexing on a strong
rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and
--no-adaptive-spots is new.
Indexer seed escalation
- Stop escalating once a seed's lattice explains >= 90% of the seed spots.
Previously any frame with >= 80 spots always paid three indexer calls, online
broker included.
Merge-consistency filter
- --min-image-cc gated on a per-image CC computed BEFORE the stills partiality
post-refinement and never refreshed; the refiner now recomputes it, so the
reported CC describes the data that are actually merged.
- Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the
merge, the error model and MergeStats can no longer disagree about which
images are in (the --scale path merged unfiltered while its statistics were
filtered).
Per-image B-factor refinement (-B) removed
- Measured on four serial-stills datasets: it is a no-op where the per-image fit
is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas
+23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on
14-25% of images). It had also been silently DISCARDED since the partiality
post-refinement landed - reported but not applied. Rather than fix and keep a
knob with no demonstrated benefit, the flag and the whole image_scale_b_factor
chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and
read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs.
ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS.
(The Wilson per-image b_factor is a different quantity and stays.)
Stills partiality width now fits both of its components
- sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d*
with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2
on d*^2. The angular-only width is fitted over a d*^2-dense population, so it
was pinned by the high-resolution edge and collapsed at low d*: median
partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55%
of them under the merge's partiality floor, and the survivors divided by those
values - which inflated the merged low-resolution intensity scale 3.6x
(~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x,
no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and
R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining
R-free are all blind to that ramp, which is why it survived earlier validation;
the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged).
Removed dead code from add-then-remove churn
- Prediction-time "still partiality" (unreachable: no setter), the phantom
IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the
constant it always was), StillsPartialityRefine's caller-less Settings
constructor and its reference to a long-gone env var, ProcessImage's unread
bool return, an unused include, and a dead viewer overlay hook.
Also
- Viewer: the magnifier compared a QImage with itself, so its scene rect was set
once ever and it could not pan into a larger dataset; the hover tail timer
could fire after leaveEvent and resurrect the resolution readout outside the
image.
- update_version.sh regenerated the frontend lock file BEFORE bumping the
version (every release shipped an off-by-one lock), and did git rm/git add on
a path that has not existed since the client moved to src/client - with no
set -e, both failed silently.
- fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a
redirect, not a test, so dkms add never ran.
- Unit tests for the adaptive-threshold host functions, which had none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis.
If things go wrong with analysis, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Improve logic on switching on/off spot finding
* jfjoch_broker: Increase maximum spot count for FFBIDX to 65536
* jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD)
* jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis
* jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection
* jfjoch_writer: Save more information about each reflection
Reviewed-on: #32
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.111 instead).
* jfjoch_viewer: Add ROI results to the dataset info plots
* jfjoch_writer: Remove HTTP interface, as it is not needed/used at the moment
Reviewed-on: #24
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 5