1142 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	8a9d80eb71	PixelRefine: profile multiplier as --profile-multiplier flag; outlier rejection default off ... (1) Promote the Term-2 profile-width multiplier from the PR_RMULT env knob to a real BraggIntegrationSettings::profile_multiplier (default 6) + jfjoch_process --profile-multiplier flag. Removes the last PR_* env knob; IndexAndRefine reads the setting. (2) Flip the merge outlier rejection default to OFF (ScalingSettings outlier_reject_nsigma 6 -> 0); it stays available via --reject-outliers <nsigma>, since it helps the jet R-free but is dataset-dependent (drops crystal-2 CCref). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:48:11 +02:00
leonarski_fandClaude Opus 4.8	102a2a7c81	PixelRefine: strip experimental env knobs (orientation, sweep, Lorentz, ML census) ... Remove the env-gated experiments that A/B'd to dead-ends or are no longer needed, returning PixelRefine to the clean factored Terms 1+2 plus the one validated keeper (r1_multiplier, default 6): - PR_ORIENT (per-image orientation refinement): R-free no-op (0.2618 vs 0.2625) - XtalOptimizer's orientation is already optimal. Removes ShoeboxResidual, OrientationRegularizer, PixelObs::weight, the refinement block and its fields. - PR_SWEEP (orientation + cell-scale sweep): R-free no-op, degraded high-res CC1/2 (per-image overfit). Removes SweepOrientationCell and its fields. - PR_LORENTZ (rotation Lorentz/zeta): hurt both directions (the factored partiality already subsumes it); was already reverted. - PR_MLCENSUS (multi-lattice census in AnalyzeIndexing): served its purpose (~3-5% of jet frames are multi-lattice; shelved). PR_RMULT (the validated Term-2 multiplier knob) is kept. Defaults unchanged: crystal 2 / jet / hybrid -R -r pixelrefine all reproduce. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:34:54 +02:00
leonarski_fandClaude Opus 4.8	6f733d74c2	Merge: make outlier rejection a ScalingSettings parameter + CLI flag, default 6 ... Promote the per-observation merge outlier rejection from the temporary PR_REJECT env knob to a real setting: ScalingSettings::OutlierRejectNsigma (default 6, <=0 disables), driven into MergeOnTheFly via the constructor, with a --reject-outliers CLI flag in jfjoch_process. Default-on at 6 sigma matches XDS (MISFITS/REJECT) and DIALS (normalised-deviation test); it validated on the full-jet R-free (0.2625 -> 0.2585). Applies to both the PixelRefine and classical merge paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 12:42:25 +02:00
leonarski_fandClaude Opus 4.8	a47b376dc3	Merge: per-observation outlier rejection (env-gated PR_REJECT) ... At the jet's ~1000x multiplicity R-free is bias-limited, and the merge had NO outlier rejection - serial data zingers/overlaps/mis-indexed frames bias every merged mean. Add a robust per-observation cut: drop observations whose corrected intensity lies > reject_nsigma error-model sigmas from the reflection's MEDIAN. The error-model sigma already captures the genuine (partiality) scatter, and the median is a robust centre, so only the tail beyond the real spread is removed - not good partials. The median is computed in RefineErrorModel (which already pools the observations per reflection); AddImage applies the cut. Env-gated via PR_REJECT=<nsigma> (off by default); logs the count removed. On the jet (CC proxy) it lifts CCref +8 (nsigma 6, 0.6% cut) to +11 (nsigma 3, 7.4% cut) - the cut is vs the data's own median, not the reference, so the gain is real cleaner means. R-free validation + the nsigma sweet spot (over-rejection risk at low nsigma) are for Filip's full-jet R-free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 20:56:53 +02:00
leonarski_fandClaude Opus 4.8	45ee8c2b40	PixelRefine: env-gated orientation + cell-scale sweep (PR_SWEEP) for R-free A/B ... R-free validation (full jet, 1.5 A) confirmed the r1_multiplier fix: PR x6 beats traditional (R-free 0.2625 vs 0.2802), the multiplier optimum is ~6 (x6==x9 on R-free; x9 only buys CC1/2 internal consistency), and per-image orientation *refinement* is a no-op (0.2618 vs 0.2625). Re-adds the reference-driven orientation + uniform cell-scale SWEEP behind PR_SWEEP (off by default) to test whether the cell-scale degree of freedom - which the gradient orientation refinement lacks, and which moves the high-res shells radially - helps R-free at 1.5 A on serial data. CCref is a near-no-op on the jet (as for the gradient path), but that does not certify R-free, so it is left for validation. NOTE: PR_RMULT / PR_ORIENT / PR_SWEEP remain temporary diagnostic env knobs; once the sweep R-free is in, bake r1_multiplier=6, drop the no-op paths, strip the knobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 19:59:50 +02:00
leonarski_fandClaude Opus 4.8	cfcd4c9e56	PixelRefine: R1 multiplier (fix tight-Term2 regression) + env-gated orientation refine ... Full-jet R-free showed the factored model regressed vs the traditional pipeline (R-free 0.34 vs 0.26, CCref 58 vs 76), traced to Term 2 using the raw measured profile width: it is too tight (~0.002 A^-1), so the template sits off the ~0.4 px centroid-floor scatter and underestimates the intensity (box-sum is immune). Fix, XDS-style (integrate over ~6 sigma): multiply the measured R1 by r1_multiplier (default 6) before use. On the jet at 1.3 A this recovers CCref 55.2 -> 59.3 (~ traditional's 60.5); crystal 2 -0.7. Tunable via env PR_RMULT for R-free. Also reinstates the pre-factored per-image orientation refinement (per-pixel ShoeboxResidual against the shoebox, regularised to the spot-centroid orientation, re-predict), behind env PR_ORIENT (off by default), to A/B its effect on serial-data R-free. On the jet CCref it is a no-op (XtalOptimizer orientation already at the optimum), but CCref is a weak R-free proxy here, so it is left for R-free validation. NOTE: PR_RMULT / PR_ORIENT are temporary diagnostic env knobs; the final multiplier value and the orientation-refine decision are to be fixed against R-free, then baked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 09:18:09 +02:00
leonarski_fandClaude Opus 4.8	bf6efc7fe9	Integration radius: default r1=4, CLI flag, PixelRefine shares the knob ... Bumped the default signal-box radius to r1=4 (r2=6, r3=8): on the lysozyme jet (1% DMM) it lifts CCref 50.1->52.2% and CC1/2 90.8->92.5% (its broadened spots spill past a radius-3 box), is neutral on the mono crystal and on the classical integrator. Added 'jfjoch_process --integration-radius <r1|r1,r2,r3>' (a single value derives r2=r1+2, r3=r1+4) and wired PixelRefine's shoebox radius to BraggIntegrationSettings::GetR1() so it shares the classical integrator's knob. (Explored but rejected: an elliptical/anisotropic Term-2 profile. The jet's tangential spots are mildly anisotropic - axis ratio 1.15->1.44 low->high res, azimuthal mosaic, separate from the radial DMM bandwidth - but using the measured 2x2 covariance as the profile DEGRADED the jet, CC1/2 92.5->83.5: the tight measured width loses to a generous aperture, same lesson as the radius bump.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 23:27:36 +02:00
leonarski_fandClaude Opus 4.8	100fe7b7e7	PixelRefine: make factored Terms 1+2 the model, remove old wiring ... PixelRefine is now an intensity-only operation: geometry is fixed (refined upstream by XtalOptimizer) and the only objective is the factored per-reflection likelihood (FACTORED_MODEL.md Terms 1+2) - measured per-resolution profile width R1 plus one Fisher-weighted intensity/scaling residual per reflection, fitting the per-image scale G and B. Validated on crystal 2 (fixed_master.h5 as stills, 1.7 A): CC1/2 84-92%, CCref 77-92%, flat - reproduces the env-flag prototype and matches the rotation path from the stills path. Removed: - the per-pixel ShoeboxResidual loss and PixelResidual cost functor; - all in-PixelRefine geometry refinement (orientation/cell/beam/distance/R), the regularised-orientation LSQ, signal-weighting, and the global sweep; - Term 3 (per-spot recentring) - a confirmed no-op on both crystals; - the diagnostic scaffolding (covariance, centroid, adaptive_R1) and the PR_* env knobs + stderr dumps in IndexAndRefine; - the PredictImage/ChiSquaredImage renderers and the entire viewer PixelRefine window/table/params + worker bindings + shoebox overlay. The sweep box-integrator background median became mean (consistency) by virtue of removing the sweep. METHODS.md rewritten for the current model; findings recorded in FINDINGS-2026-06.md. Net -2200 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 22:02:18 +02:00
leonarski_fandClaude Opus 4.8	6f2033db00	PixelRefine: checkpoint before cleanup (factored model + all diagnostic levers) ... Snapshot of the messy state: factored likelihood Terms 1+2+3 behind PR_* env flags (PR_INTENSITY/PR_SHAPE/PR_RECENTER) alongside the old per-pixel ShoeboxResidual, plus diagnostic scaffolding (PR_R0/R1/COV/FIX_R0/FIX_R/ ADAPT_R1/CENTROID/RECENTER) and the FACTORED_MODEL.md spec. Next commit makes Terms 1+2 the model and strips all of this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 21:34:33 +02:00
leonarski_fandClaude Opus 4.8	c93d381dc8	Error model: harden the fit against pathological inputs (code review) ... Addresses code-review findings on RefineErrorModel: - Floor the 1/dev^2 bin weight relative to the data scale (1e-3 of the median bin dev^2), not an absolute 1e-30: a near-zero-scatter bin could otherwise acquire a runaway weight and hijack the global (a,b) fit. - Reject a near-collinear normal-equation system relatively (det > 1e-10*Ass*AII) instead of with an absolute threshold that an ill-conditioned fit can pass. - Reset the model to identity at entry so any early return leaves it inactive rather than keeping a stale a/b alongside a freshly-cleared mean map (which would make CorrectedSigma fall back to the per-observation I). - PixelRefine: correct the orient_prior comment - with the sweep on, the LSQ anchor is the swept orientation (intended), not the spot-centroid one. Verified unchanged on the lyso test set (ISa 1.1, CC1/2 90.3%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 18:53:02 +02:00
leonarski_fandClaude Opus 4.8	e6a50b45c7	Integration: mean background + global error model (trustworthy sigmas) ... Background estimate: use the mean of the local ring, not the median. For a right-skewed (Poisson) background the median sits below the mean, so subtracting it under-subtracts and biases every weak intensity positive; over multiplicity this becomes fake <I/sig> of a few in no-signal high-resolution shells. Fixed in both PixelRefine and BraggIntegrate2D (the classical route had the same bug). <I/sig> now tracks CC honestly and the true resolution limit is visible. Error model: fit a global a, b (XDS form sigma'^2 = a*sigma^2 + (b*I)^2) from the scatter of symmetry equivalents at the merge level (so both integrators benefit), and print it with ISa = 1/b in jfjoch_process. The (b*I)^2 term uses the reflection mean (not the per-observation I_i, which biases the weights and collapses CC); a,b come from a relative-weighted bin regression. Replaces the earlier per-resolution-shell variant, which was partly masking the background bias. METHODS.md: document both (Sections 6-7), integrator-agnostic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 18:39:32 +02:00
leonarski_f	3cdf26232c	PixelRefine: Document on local changes	2026-06-12 17:29:28 +02:00
leonarski_f	db68c8dc38	PixelRefine: Results seem to be much better	2026-06-12 17:28:18 +02:00
leonarski_f	47dc19dd03	PixelRefine: Improvements to accept more reasonable count of reflections	2026-06-12 10:12:23 +02:00
leonarski_f	c8db50ab41	jfjoch_viewer: missing pixel refine table	2026-06-12 07:47:28 +02:00
leonarski_f	d31063ca3f	PixelRefine: Some improvements	2026-06-11 21:25:05 +02:00
leonarski_f	48d4fb0d0f	XDS plugin: Fix mutex	2026-06-11 18:35:02 +02:00
leonarski_f	bd5fef7f61	PixelRefine: Simplify (remove Lorentz correction, remove background from azimuthal integration)	2026-06-10 18:37:46 +02:00
leonarski_f	7478c0390f	jfjoch_scale: Remove postrefinement model option	2026-06-10 16:37:15 +02:00
leonarski_f	59281f6330	Revert "Bragg integration: option to use azimuthal integration profile" ... This reverts commit e4230bc14e.	2026-06-10 16:36:37 +02:00
leonarski_f	579d36fe71	Revert "jfjoch_process: Add option to use azimuthal integration as background for Bragg integration" ... This reverts commit b22d5929a1.	2026-06-10 16:36:37 +02:00
leonarski_f	b22d5929a1	jfjoch_process: Add option to use azimuthal integration as background for Bragg integration	2026-06-10 14:52:58 +02:00
leonarski_f	e4230bc14e	Bragg integration: option to use azimuthal integration profile	2026-06-10 14:41:36 +02:00
leonarski_f	e051eed033	jfjoch_process: Remove postrefinement	2026-06-09 19:46:39 +02:00
leonarski_f	efe882f4b6	jfjoch_viewer: Better display (to be tested) of pixel refine	2026-06-09 16:28:17 +02:00
leonarski_f	6c85aaba2b	BraggPrediction: Include X-ray bandwidth	2026-06-09 15:01:44 +02:00
leonarski_f	6af22b6a0c	jfjoch_viewer: show image CC based on standard pipeline	2026-06-09 13:28:56 +02:00
leonarski_f	003fea1b1e	jfjoch_viewer: fix sorting by indexing status	2026-06-09 13:28:35 +02:00
leonarski_f	2d202f1d44	jfjoch_viewer: fixes to pixel refine window	2026-06-09 13:20:46 +02:00
leonarski_f	feca63f4b9	jfjoch_viewer: fixes to pixel refine	2026-06-09 12:35:06 +02:00
leonarski_f	8a582b8a90	JFJochMagnifierWindow: Fixed	2026-06-09 12:09:40 +02:00
leonarski_f	30dcc98f89	JFJochMagnifierWindow: Zoom is saved ... it is not optimal (when image is first loaded, than it starts with weird zoom), but can be fixed later	2026-06-09 12:03:45 +02:00
leonarski_f	5735302691	Merge: CC1/2 limit adjustment	2026-06-09 11:06:16 +02:00
leonarski_f	e4b66f9cd9	PIxelRefine: Another iteration	2026-06-09 08:04:27 +02:00
leonarski_f	254462b9f2	jfjoch_process: Clarify what happens in scaling when reference data provided	2026-06-09 07:28:13 +02:00
leonarski_f	e8a9b1840d	PixelRefine: Make it faster by doing one cell calculation per shoe-box	2026-06-08 22:45:22 +02:00
leonarski_f	05711a1077	jfjoch_viewer: Add pixel refinw and magnifier windows (to be tested)	2026-06-08 21:22:19 +02:00
leonarski_f	698a98be35	PixelRefine: Claude fixed my bugs	2026-06-08 20:06:53 +02:00
leonarski_f	32e91f7287	Minor fixes	2026-06-08 15:56:20 +02:00
leonarski_f	96edee9b2d	jfjoch_writer: Add incident_wavelength_spread	2026-06-08 15:47:12 +02:00
leonarski_f	155c53acd8	jfjoch_process: First pixel refine integration	2026-06-08 15:37:27 +02:00
leonarski_f	6f31159607	PixelRefine: Add bandwidth contribution	2026-06-08 13:59:01 +02:00
leonarski_f	6f6098d508	PixelRefine: Work in progress (Claude)	2026-06-08 13:44:39 +02:00
leonarski_f	66a48c4266	PixelRefine: Work in progress (Claude)	2026-06-08 12:43:10 +02:00
leonarski_f	9a991b6614	PixelRefine: Work in progress	2026-06-08 11:59:48 +02:00
leonarski_f	0d6b278767	LatticeReduction: Move outside of XtalOptimizer	2026-06-08 11:11:35 +02:00
leonarski_f	50f953bcc5	PixelRefine: Work in progress	2026-06-08 09:41:58 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58 1.0.0-rc.148	2026-06-08 08:30:35 +02:00
leonarski_f	75de40f52b	v1.0.0-rc.147 (#57) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_viewer: Add reciprocal space viewer * jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset * jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress) Reviewed-on: #57 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-02 11:49:24 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56 1.0.0-rc.146	2026-05-28 18:48:35 +02:00