v1.0.0-rc.146 #56

Merged

leonarski_f merged 133 commits from 2605-scaling into main 2026-05-28 18:48:35 +02:00

Conversation 0 Commits 133 Files Changed 273

+16303 -2642

leonarski_f commented 2026-05-28 13:26:05 +02:00

Owner

Copy Link

Copy Source

This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.

jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.

WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.

This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.

leonarski_f added 130 commits 2026-05-28 13:26:06 +02:00

HDF5DataFile: Saving images is a configurable option 930cfb0b35

HDF5DataFile: Include File name explicitly + FileWriter: Handle NXmxIntegrated in a smarter way 173198be40

jfjoch_process: Cleanup help + by default save HDF5, but without analysis results 6c8c953c92

FileWriter: Fix cd5d97aa55

DiffractionExperiment: Max image number is 2 million + 1; it is unsafe to collect more f5176b56a9

ImagePusher: Serializer defined inside ImagePusher class, not in subclasses 27bcb19328

FileWriter: Master HDF5 contains scalar per-image processing results, reader is also made to prioritize reading from master file 025c9b3aee

Fix 91c714f1a4

jfjoch_test: Fix tests da15714080

HDF5NXmx: Don't write nPeaks in HDF5NXmx - this needs to be consistent with spot finding results, so should come from spot finder 8b5eb5a208

HDF5NXmx: Clean-up 1b8c528ed2

jfjoch_process: Adapt behavior to write file with processing results and link to original images (to be tested) 0503ef0271

jfjoch_process: Fix typo 93914e1fb9

FileWriter: Allow to save master files with custom suffix (like _process.h5) d718999335

JFJochHDF5Reader: Add ReadReflections function 38c2826c09

jfjoch_test: Improve a bit XtalOptimizer_hexagonal test hoping for more consistent result cf2ca90fb7

jfjoch_process: Fix bounds for VDS mapping cc42ae6bf6

ScaleAndMerge: Simplify merge and go back to "standard" weights e392a3ae1b

ScaleAndMerge: Add "just" merge function for now 3e4e6019c4

ScaleAndMerge: Minor clean-up 81af116b4d

Split scaling and merge into separate files (WIP) a90ec13418

HKLKey: Fix a441e96b50

Refactor splitting scale and merge b9af590ff5

jfjoch_test: Fix TCPImagePusherTest to ensure it is more stable 4b6a042e4a

jfjoch_process: Implement split scale & merge. eea438185c

ScalingResult: Dedicated data structure 440e54d588

Move to ScaleOnTheFly model b854c0f356

HKLKey: Test properly, fix bug with merging Friedel pairs when no SG given e70b21f6de

jfjoch_process: Optimize design choices for scaling integration 486310fd87

jfjoch_process: Print statistics moved to Merge.h/Merge.cpp and ScalingResult.cpp/ScalingResult.h b53f0d6474

jfjoch_process: Add space group number handling 9ec8ebcde8

jfjoch_process: Fixes to parameter logic 4e6a6e2aa2

Merge: Remove unused statistics values (for now) aa347096d2

ScaleOnTheFly: Calculate rotation partiality correctly c92d01222c

ScaleOnTheFly: Fix B term cecb3024e0

Merge: Remove old statistics 0cf801c2d2

Clean-up b1fe201047

Merge: Remove scaling correction for now (need to think about it) a468e46222

jfjoch_process: Multiple scaling iterations b7627b5a11

jfjoch_process: removed geometry restriction option e45d1ff4d3

HKLKeyGenerator: Make HKLKeyGenerator having already prebuilt gemmi data structures 2fa9293fda

jfjoch_process: Fix reading missing metadata 078c871ec1

Merge: Simplify (but not yet gaining performance) c55e91ccfa

Report merge time f0601289d9

Merge branch '2605-scaling-bugfix' into 2605-scaling 8a9d1d39ec

BraggIntegrate2D: For very large sigma's, use 2% of I as sigma (if larger than sqrt(I)) 02327bf610

Merge: Simplify code for merging, make it more efficient d5b928fa73

Gemmi: Include through FetchContent full gemmi library (not limited cpp/hpp files) b06dfc8357

Revert "Gemmi: Include through FetchContent full gemmi library (not limited cpp/hpp files)" ... e1c661e48d

This reverts commit b06dfc8357.

JFJochHDF5Reader: read pixel_mask defined under /detectorSpecific b27b140bf0

Gemmi: Add more functionality from gemmi 0.7.5 08bf186766

Enable reading MTZ files with reference Fcalc 6aa6890a8b

Merge: Add partiality limit e70d71912b

ScaleOnTheFly: Add CC image/ref 2397a108dd

CrystalLattice: Need to include gemmi via exact path 1afbaf6f11

JFJochMessages: Need to include gemmi via exact path 1bccbda370

Save/transfer/read/display image scale results 0e9f7cc956

IndexAndRefine: Save scaled reflections ffc5c1f561

jfjoch_process: Remove French & Wilson - let's leave it to Phenix to figure out a39a751670

jfjoch_process: Minor refactor 1b781660c9

jfjoch_process: Display scaling time in per image metrics cd492db2eb

jfjoch_process: Display metrics in millisecond and cut one digit of precision - down to 10 us ac769ea930

jfjoch_process: Add CC-based merging limit de36a8c421

jfjoch_scale: Add tool, work in progress 65ff3a2b68

jfjoch_scale: Add scaling iteration count 7212f5a628

ScaleOnTheFly: DiffractionExperiment is first argument to constructor, as this is convention used elsewhere b0a64fb254

IndexAndRefine: Use pragma once 6852908dff

ScaleOnTheFly: mosaicity input is float, for consistence with other code 83ae1f0ced

jfjoch_scale: Minor improvements 6dd9997610

Merge: Add CC1/2 - I don't like it though, as there is big overlap between merge and merge statistics (better to calculate 1/2 datasets in merge separately?) 43d8c93bd1

IndexAndRefine: Fix pragma once 7e656ae1f9

Merge: For anomalous signal, treat Friedel pairs separate, but add half-datasets 674ac4afa2

Merge: Use random number generator with fixed seed to assign half-datasets. Also make it per image, not per reflection 3e9c9c3f9b

jfjoch_test: Fix scaling tests b8e770e424

Merge: Make table a bit prettier 68e3cbf1b4

HDF5NXmx: Save unit cell after rotation indexing as main one + lattice (in UB-matrix format) + input unit cell as extra information 0ceff6e91b

Merge: Mark TODO bed03840ea

IndexAndRefine: Keep unit cell dimensions constant when using rotation indexing 19f8d5e776

HDF5NXmx: Fix 25d3f660be

Merge: Some clean-up of CC merge mask + add unit cell handling in IndexAndRefine 1bf70dcae7

Improve handling/calculating merge mask bb40c9518b

jfjoch_scale: Clean-up 079d85628b

Merge.h: Clean-up 6185b870cf

Integrate consensus unit cell handling 023e08e80d

jfjoch_process: remove printing final rotation lattice 8b7bd4c8b6

Merge: Fix ambiguity in names 4ca6684252

Update resolution of reflections based on consensus unit cell f5447b3478

UpdateReflectionResolution: Separate file + clear bug 39fef1bcef

Add possible reflection and completeness to MergeStatistics 1d3ccdaa00

Merge: Apply systematic absences to merging/merging statistics be341e4cce

UnitCell: Use direct path to gemmi unitcell 980074baf9

jfjoch_test: Add test for systematic absences 1bc96d0f17

ScaleOnTheFly: Make scale factor always positive 68047ef787

jfjoch_viewer: Fix displaying per image CC ac907e8aef

jfjoch_viewer: Add Image B-factor 4e2043ec4b

Merge: Add CCref 8371743cbb

HDF5: Save per reflection scaling factor 7c0cc22e47

jfjoch_scale: Seems to work (though unstable) d555042dc6

Merge: Fix CC calculation 371a69c5b8

CorrelationCoefficient: Separate class c90e4facce

Merge: Merging happens in MergeOnTheFly class, not in function a05ec050ed

jfjoch_scale: A bit nicer handling of reflection statistics and preparation 1a70c1987a

ScalingSettings: file format and intensity settings e4ff4dca9e

Merge: Generate R-free flags 0eb0c50c66

General function to write reflections in multiple formats (need to better integrate into jfjoch_scale and jfjoch_process) 39823ddb67

jfjoch_process: Introduce long options + image stride 270e906fe7

Merge: More printing is done directly, 18c0d0cf6f

jfjoch_process/jfjoch_scale: Improve parameter handling 52281bfd58

jfjoch_scale: Improve process of reading reflections, particularly with multiple files 8b307a54ee

jfjoch_process: Simplify writing result of space group search, without Logger e59f11cb44

jfjoch_process: Select output format for scaling results 805dc7d164

jfjoch_tests: Fix JFJochReader_ReadReflections_VDS 8d80c64bed

jfjoch_scale: Fix typo 65e4b2fdb9

jfjoch_scale: Align help messages 8d8e9717d0

jfjoch_viewer: Fix shift of labels and pixels 6ad580d8c1

VERSION: 1.0.0-rc.146 2e18291256

CHANGELOG 41896a7ef2

XtalOptimizer: Zero initialize latt_vec0, latt_vec1, latt_vec2 65a28b2b34

XtalOptimizer: Freeze latt_vec2 for highly regular systems 952559d3a7

XtalOptimizer: ChatGPT fix to monoclinic extraction logic 752d3e95f1

BraggIntegrate2D: Keeps observed x and y positions for reflections 6621ddb2ef

XtalOptimizer: Store 1/lambda (not lambda) + add check for very small lambda + set beam center as one parameter block + fix poni rotation convention (rot2 was likely wrong) dce6ee3d6c

XtalOptimizer: Fix bugs in XtalOptimizerRotationOnly: mismatch between rotated and omega=0 coordinate system; add regularization of rotation to zero; use reciprocal cell in refinement (no need to do inverse) 815080940a

ScaleOnTheFly: Add (optional) regularization 6f3bcdd76f

ScaleOnTheFly: WIP: ScalingPostRefResidual fe19ff8ec6

Scaling/Merging: Work in progress to improve the information that is fed to scaling. This is a mess at the moment - with redundant data structures, etc. - will clean this up, later. a85eb4b137

Scaling/Merging: Work in progress to clean-up data strctures 61776a446e

Scaling/Merging: Clean-up data structures ea36b26a4f

CalcISigma: Fix running with msg.reflections not yet set 2a2df9ad1a

ScaleOnTheFly: Enable postrefinement for still images c22b566503

leonarski_f added 3 commits 2026-05-28 17:15:22 +02:00

jfjoch_process: Add orientation-only refinement 058468a0b3

OpenAPI: Add orientation only refinement af847239fc

CHANGELOG 6c107d5f85

leonarski_f merged commit fc68a9baed into main 2026-05-28 18:48:35 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

appleb_m (appleb_m) buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) perl_d (perl_d) takaba_k (takaba_k)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#56