diff --git a/common/JFJochMessages.h b/common/JFJochMessages.h index 9498895dc..cc3154d7f 100644 --- a/common/JFJochMessages.h +++ b/common/JFJochMessages.h @@ -499,6 +499,9 @@ struct EndMessage { std::vector spot_count_low_res; std::vector spot_count_indexed; std::vector image_indexed; + // 1 where the image's observations reached the merged data, 0 where none did (MergeStatistics::image_merged); + // empty where no merge ran. Offline (rugnux) only. + std::vector image_merged; std::vector indexed_lattice_count; std::vector v_bkg_estimate; // Per-image spindle_blind_fraction; NaN where the frame had no value. NaN is the CANNOT-SAY diff --git a/docs/CBOR.md b/docs/CBOR.md index b6abf765d..4c81ef0ca 100644 --- a/docs/CBOR.md +++ b/docs/CBOR.md @@ -305,6 +305,7 @@ Every scalar of the image message is optional: a quantity that was not computed | spot_count_low_res | Array(int32) | Per-image number of low-resolution spots | | | spot_count_indexed | Array(int32) | Per-image number of spots fitting indexing solution | | | image_indexed | Array(uint8) | Per-image indexing result; 0 = not indexed, nonzero = indexed | | +| image_merged | Array(uint8) | Per-image merge result; 1 = the image's observations reached the merged data, 0 = none did. Empty where no merge ran (rugnux) | | | v_bkg_estimate | Array(float) | Per-image background estimate | | | v_spindle_blind_fraction | Array(float) | Per-image spindle_blind_fraction; NaN where the frame had no value (which is "cannot say", not zero) | | | ice_ring_score | Array(float) | Per-image strongest ice-ring intensity over the smooth radial background (1 = no ice) | | diff --git a/docs/HDF5.md b/docs/HDF5.md index 577c11a1b..fa6640edc 100644 --- a/docs/HDF5.md +++ b/docs/HDF5.md @@ -391,6 +391,7 @@ in the data files only). | `imageScaleMosaicity` | deg | scaling-model mosaicity | | `sweepQuality` | | why this image's stretch of the sweep was flagged — see below | | `frameDisposition` | | what became of this image's observations in the merged data — see below | +| `imageMerged` | | image's observations reached the merged data (0/1); written by rugnux beside the merge (stills: the crystal took part; rotation: the frame is not `rejected` in `frameDisposition`), absent where no merge ran (`--no-merge`) and in files written before it existed | **Per-image lattices:** `latticeIndexed` `[n_images, 9]` (Å) — the real-space lattice (flattened 3×3); `latticeIndexedExtra` `[n_images, max_extra_lattices, 9]` (Å) — additional orientation diff --git a/docs/JFJOCH_VIEWER.md b/docs/JFJOCH_VIEWER.md index 7103a42e2..832f98dfc 100644 --- a/docs/JFJOCH_VIEWER.md +++ b/docs/JFJOCH_VIEWER.md @@ -234,6 +234,8 @@ keyboard without them `Home` / `End` are `Fn`+`←` / `Fn`+`→` and `Page Up` / | Dataset-info plot: `Shift` + hover | Load the hovered image | | Spot / reflection list: double click | Zoom the diffraction image on that spot or prediction | | Image list: double click | Load that image | +| Image list: click a column header | Sort by that column (Index and Merged sort Yes, No, N/A) | +| Image list: right click the Index or Merged header | Filter the rows by that column - tick Yes, No or N/A; a filtered header is marked `*`, and the two filters combine, so Index Yes with Merged No lists the images indexed but not merged. Merged is read from `/entry/MX/imageMerged` of a rugnux `_process.h5` and is N/A for a file without it | ## Opening data diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 53ec4405c..bfb6e9b18 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -289,7 +289,9 @@ different space group, resolution limit, anomalous setting or outlier rejection. `-o/-N/-s/-e/-S/-A/-z/--scaling-*` options as the full run, and (unlike the full pipeline) does not run a space-group search: it merges in the space group and unit cell the file records, and `-S` / `-C` override them. A `_process.h5` written before the group was stored carries none, and merges -in P1 unless `-S` says otherwise. +in P1 unless `-S` says otherwise. Beside the reflection files and the report it writes +`_plot.txt` for its own merge - per image, the scale and whether the image's observations are +in the merged data (the report's `IMAGES_MERGED` is the sum of that column). A reference MTZ (`-z`) is accepted here on **stills** data, where it fixes the space group and cell, reports CCref and hands over its R-free flags; on rotation data the rotation scaler diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index e98c3e4ff..35243f711 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -243,11 +243,13 @@ The keys most scripts read, by section; the rest of this page explains the ones | `OSCILLATION_RANGE`, `STARTING_ANGLE`, `ROTATION_AXIS` | Rotation only: the per-image rotation the run integrated at (deg: the file's, scaled where the run adopted a goniometer rotation scale), the first image's angle and the axis | | `DETECTOR_DISTANCE`, `BEAM_CENTRE`, `DETECTOR_TILT` | The geometry the run integrated at (mm; the point of normal incidence in pixels; the three PONI rotations in degrees) - on a rotation run the post-refined one, not the header's | | `DIRECT_BEAM` | Where the direct beam meets the detector under that tilt (pixels) | +| `IMAGES_INDEXED`, `LATTICES_INDEXED`, `IMAGES_INTEGRATED` | Section 2: the images the per-image indexer indexed (the count behind `INDEXING_RATE`), the lattices it indexed on them - an image of two crystals counts two - and the images integrated. An image is integrated on its main lattice alone: an extra lattice is indexed and its spots assigned, but it is not integrated, so `IMAGES_INTEGRATED` is also the number of integrated lattices. As `INDEXING_RATE`, the first two are a stills measure and are written on rotation only with `--developer`; `--mode scale`, which indexes nothing, writes `IMAGES_INTEGRATED` alone (the images with integrated reflections in its input). `IMAGES_PROCESSED` ≥ `IMAGES_INDEXED`, `LATTICES_INDEXED` ≥ `IMAGES_INDEXED` | | `LATTICE_FOUND`, `UNIT_CELL_CONSTANTS`, `CELL_ESD` | Section 2: whether a lattice was found, the cell (a b c α β γ, Å and degrees) and its e.s.d.s ([below](#cell-esds)) | | `CELL_FORMS`, `OTHER_CELL_FORM_*` | Stills: whether the frames on the run's cell were one crystal form or two, and the other one, left out of the merge ([below](#two-crystal-forms-stills)) | | `SPACE_GROUP_NAME`, `SPACE_GROUP_NUMBER` | The space group the files are written in ([below](#the-space-group-and-the-sohncke-answer-beside-it)) | | `MOSAICITY_DEG` | Rotation only ([below](#mosaicity)) | | `MERGE` | Section 3: `PERFORMED` or `NOT_PERFORMED` (`--no-merge`) | +| `IMAGES_MERGED` | Section 3: the images whose observations are in the merged data - on stills the crystals (patterns) that went into the merge, past the cell form, the scale and `--min-image-cc`; on rotation the frames not rejected (`FRAMES_MERGED` + `FRAMES_DOWNGRADED` where the sweep-quality diagnostic ran). The sum of the `merged` column of `_plot.txt` and of `/entry/MX/imageMerged` in the `_process.h5`; written by `--mode scale` too | | `INCLUDE_RESOLUTION_RANGE` | `dmax dmin` of the written reflections (Å), as XDS's keyword | | `FITTED_RESOLUTION` | Where the fall-off of the standard CC1/2 (the `CC_HALF` of the shell table) crosses the `--resolution-cc-target` - the number to quote; absent where the curve is not a fall-off (a `RESOLUTION_FIT` warning says why) | | `FITTED_RESOLUTION_INTERNAL` | Rotation only, internal: the crossing of the automatic cut's own fit, which weights each reflection by the scale of the frames it was measured on so that a weak stretch of the sweep does not drag it in. The written range (`INCLUDE_RESOLUTION_RANGE`) is cut from this one; on a sweep without a weak stretch it equals `FITTED_RESOLUTION` | @@ -297,7 +299,9 @@ hundred lines. deliberately low: a warning is a prompt to check something, not a verdict, and some of them fire on data that turn out fine — the closed type for machinery, the open sentence for a person. -- Below them, one line each, the facts a reader needs before reading further: space group, cell, +- Below them, one line each, the facts a reader needs before reading further: the images (`Images`: + processed, indexed with the lattices on them - stills only -, integrated and merged, the counts of the + keys above), space group, cell, mosaicity, the beam centre, the powder and ice rings, overloads, overlapping spots, multiple lattices, resolution, completeness, signal, anomalous signal, anisotropy, pseudo-symmetry, twinning, the indexing ambiguity of serial stills (`Indexing ambiguity`: how the data resolved it, in the terms of the `AMBIGUITY_*` keys below, or `UNRESOLVED` where the hands are diff --git a/docs/RUGNUX_TUTORIAL.md b/docs/RUGNUX_TUTORIAL.md index 65b98bce0..2bc51a2f1 100644 --- a/docs/RUGNUX_TUTORIAL.md +++ b/docs/RUGNUX_TUTORIAL.md @@ -379,12 +379,15 @@ Absorption is corrected only by the empirical surface a rotation run fits and ke | `sigma_M_deg` | Mosaicity as the rocking-curve **standard deviation** σM in degrees — not a FWHM, and not XDS's `REFLECTING_RANGE`. See the note below. | | `cc_to_merge` | Correlation of the image's own observations with the merged data. | | `cc_n` | Observations that correlation was computed over; 0 where none was. | - | `merged` | 1 where the image's observations are in the merged data, 0 where they are not — an image that never indexed, one an earlier guard dropped, and one ΔCC1/2 convicted all read 0. `nan` on a run that merged nothing, `--no-merge` included, where the question has no answer. | - | `disposition` | The same answer graded, to plot directly: 1 merged, 0.5 downgraded (merged, but over a degraded stretch of the sweep), 0 rejected. 0.5 is a marker, not a weight — the merge applies no factor of its own to a downgraded frame, which is carried at the weight its own scale and sigmas give it. `nan` for an image that never got a scale and so never reached the merge, and on a run that merged nothing. | + | `merged` | 1 where the image's observations are in the merged data, 0 where they are not — an image that never indexed, one an earlier guard dropped (on stills: another crystal form, a collapsed scale, `--min-image-cc`), and one ΔCC1/2 convicted all read 0. An image of several lattices is integrated, and so merged, on its main lattice alone, so the column is 0 or 1 and its sum is the report's `IMAGES_MERGED`; the same vector is `/entry/MX/imageMerged` of the `_process.h5`. `nan` on a run that merged nothing, `--no-merge` included, where the question has no answer. | + | `disposition` | The same answer graded, to plot directly: 1 merged, 0.5 downgraded (merged, but over a degraded stretch of the sweep), 0 rejected. 0.5 is a marker, not a weight — the merge applies no factor of its own to a downgraded frame, which is carried at the weight its own scale and sigmas give it. `nan` for an image that never got a scale and so never reached the merge, and on a run that merged nothing. On stills, with no sweep to grade, 1 merged and 0 scaled but left out. | + | `indexed` | 1 where the per-image indexer indexed the image, 0 where it did not (the `imageIndexed` of the HDF5 file; its sum is the report's `IMAGES_INDEXED`). `nan` from `--mode scale`, which indexes nothing. | A quantity nothing measured for an image is written `nan`, which gnuplot and numpy both skip; the columns never shift and a row is never left out. A stills run writes the table too, with `nan` in - the columns a sweep would fill; `--no-merge` writes it with the scaling columns `nan`. A joint merge of + the columns a sweep would fill; `--no-merge` writes it with the scaling columns `nan`; `--mode scale` + writes it for its own merge, with the per-image analysis columns (`bkg`, `resolution_A`, `spots`, + `indexed`) `nan`. A joint merge of sweeps writes one row per frame of the sweeps it merged, sweep after sweep in input order: each row's per-image analysis is its own sweep's, the scaling columns the joint merge's; a sweep left out of the merge has no rows. diff --git a/frame_serialize/CBORStream2Deserializer.cpp b/frame_serialize/CBORStream2Deserializer.cpp index 1468e3ba6..3fef5cc90 100644 --- a/frame_serialize/CBORStream2Deserializer.cpp +++ b/frame_serialize/CBORStream2Deserializer.cpp @@ -1791,6 +1791,8 @@ namespace { GetCBORInt32Array(value, message.spot_count_indexed); else if (key == "image_indexed") GetCBORUInt8Array(value, message.image_indexed); + else if (key == "image_merged") + GetCBORUInt8Array(value, message.image_merged); else if (key == "v_bkg_estimate") GetCBORFloatArray(value, message.v_bkg_estimate); else if (key == "v_spindle_blind_fraction") diff --git a/frame_serialize/CBORStream2Serializer.cpp b/frame_serialize/CBORStream2Serializer.cpp index 345d30858..c9fe48f42 100644 --- a/frame_serialize/CBORStream2Serializer.cpp +++ b/frame_serialize/CBORStream2Serializer.cpp @@ -987,6 +987,7 @@ void CBORStream2Serializer::SerializeSequenceEnd(const EndMessage& message) { CBOR_ENC(mapEncoder, "spot_count_low_res", message.spot_count_low_res); CBOR_ENC(mapEncoder, "spot_count_indexed", message.spot_count_indexed); CBOR_ENC(mapEncoder, "image_indexed", message.image_indexed); + CBOR_ENC(mapEncoder, "image_merged", message.image_merged); CBOR_ENC(mapEncoder, "v_bkg_estimate", message.v_bkg_estimate); CBOR_ENC(mapEncoder, "v_spindle_blind_fraction", message.v_spindle_blind_fraction); CBOR_ENC(mapEncoder, "ice_ring_score", message.ice_ring_score); diff --git a/image_analysis/scale_merge/Merge.cpp b/image_analysis/scale_merge/Merge.cpp index 10797019a..6ee398eb2 100644 --- a/image_analysis/scale_merge/Merge.cpp +++ b/image_analysis/scale_merge/Merge.cpp @@ -232,6 +232,7 @@ void MergeOnTheFly::AddImages(const std::vector &outcomes) { std::unique_lock ul(merged_mutex); std::vector rejected(outcomes.size(), 0); + images_merged.assign(outcomes.size(), 0); ReduceImagesInOrder( outcomes.size(), nthreads, [&](size_t i, std::vector &out) { @@ -240,6 +241,7 @@ void MergeOnTheFly::AddImages(const std::vector &outcomes) { }, [&](size_t i, const std::vector &out) { reject_count += rejected[i]; + images_merged[i] = !out.empty(); Accumulate(out, static_cast(i)); }); } diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index 1625414c7..d81ad5e2d 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -246,6 +246,10 @@ struct MergeStatistics { // (MeasureSweepQuality), and what became of every frame of the sweep (MeasureBatchDeltaCCHalf). SweepQuality sweep_quality; + // Per processed image: 1 where its observations reached the merge, 0 where none did - on stills the + // crystal took part, on rotation the frame is not rejected (FrameDisposition). Empty where no merge ran. + std::vector image_merged; + // Diffraction anisotropy (AnalyzeAnisotropy): the anisotropy tensor, the diffraction limit along // each of its principal directions, and whether either is established above this dataset's own // systematic error. Report-only - nothing is corrected and no reflection is removed. Empty @@ -423,6 +427,7 @@ class MergeOnTheFly { // ln(I / median); the stills merge sets both to n b, the linear band. double reject_up = 0.0, reject_down = 0.0; size_t reject_count = 0; + std::vector images_merged; // per outcome of AddImages: 1 where an observation of it reached the merge // One accepted observation, reduced to what the accumulator adds for it. struct Contribution { @@ -483,6 +488,8 @@ public: // AddImage for every outcome, the image id being its index: the same merge, bit for bit, with the // per-observation work spread over the workers and the accumulation kept in image order. void AddImages(const std::vector &outcomes); + // Per outcome of the last AddImages: 1 where at least one of its observations went into the merge. + [[nodiscard]] const std::vector &ImagesMerged() const { return images_merged; } // Every test AddImages applies, each evaluated on its own, for every reflection of every outcome (in // their order); call after AddImages. A diagnostic: the merge itself does not use it. diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index a4557d8a3..86a4bfaf3 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -8194,6 +8194,14 @@ RotationScaleMerge::Result RotationScaleMerge::RunArm(bool scale_partials, bool TimingMark merge_mark(*logger, "rsm merge and statistics"); auto r = MergeAndStats(n_groups, for_search, combined_on_gpu && scaled_fulls_on_gpu, full_stats); merge_mark.Stop(); + // A frame is in the merge as the disposition ledger counts it (CountSweepDisposition); where the ledger + // did not run, as the per-frame guards left it. + const auto &disposition = r.statistics.sweep_quality.frame_disposition; + r.statistics.image_merged.assign(n_frames, 0); + for (int f = 0; f < n_frames; ++f) + r.statistics.image_merged[f] = disposition.empty() + ? frame_in_merge[f] + : disposition[f] != static_cast(FrameDisposition::Rejected); r.scaling_iterations_partials = partial_loop.iterations; r.scaling_iterations_fulls = fulls_loop.iterations; r.scaling_converged = partial_loop.converged && (fulls_loop.iterations == 0 || fulls_loop.converged); diff --git a/reader/HDF5MetadataSource.cpp b/reader/HDF5MetadataSource.cpp index 19c278dfe..03dfb2211 100644 --- a/reader/HDF5MetadataSource.cpp +++ b/reader/HDF5MetadataSource.cpp @@ -599,6 +599,7 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen dataset->spot_count_ice_control = master_file->ReadOptVector("/entry/MX/peakCountIceRingControl"); dataset->indexing_result = master_file->ReadOptVector("/entry/MX/imageIndexed"); + dataset->merge_result = master_file->ReadOptVector("/entry/MX/imageMerged"); dataset->bkg_estimate = master_file->ReadOptVector("/entry/MX/bkgEstimate"); dataset->spindle_blind_fraction = master_file->ReadOptVector("/entry/MX/spindleBlindFraction"); dataset->ice_ring_score = master_file->ReadOptVector("/entry/MX/iceRingScore"); diff --git a/reader/JFJochMultiReader.cpp b/reader/JFJochMultiReader.cpp index 903246a79..f1daa490c 100644 --- a/reader/JFJochMultiReader.cpp +++ b/reader/JFJochMultiReader.cpp @@ -98,6 +98,7 @@ JFJochMultiReader::JFJochMultiReader(std::vector> dataset->spot_count_ice_rings.clear(); dataset->spot_count_ice_control.clear(); dataset->indexing_result.clear(); + dataset->merge_result.clear(); dataset->indexing_lattice_count.clear(); dataset->bkg_estimate.clear(); dataset->pixel_sum.clear(); diff --git a/reader/JFJochReaderDataset.h b/reader/JFJochReaderDataset.h index 7dd906bea..cf8b0319b 100644 --- a/reader/JFJochReaderDataset.h +++ b/reader/JFJochReaderDataset.h @@ -62,6 +62,7 @@ struct JFJochReaderDataset { std::vector spot_count_ice_control; std::vector indexing_result; + std::vector merge_result; // /entry/MX/imageMerged: 1 merged, 0 not; empty in a file without it std::vector indexing_lattice_count; std::vector bkg_estimate; // The XFEL pulse ID and event code of each image (/entry/xfel), where the file records them. diff --git a/rugnux/DiagnosticOutput.cpp b/rugnux/DiagnosticOutput.cpp index 25e104ae7..d47b12fa0 100644 --- a/rugnux/DiagnosticOutput.cpp +++ b/rugnux/DiagnosticOutput.cpp @@ -110,7 +110,7 @@ std::string RenderDetectorDiagnostic(const DiffractionExperiment &experiment, } void FillJointPlotRows(PerImagePlot &joint, size_t first_row, int dataset, const PerImagePlot &own) { - for (auto *v : {&joint.angle_deg, &joint.bkg_estimate, &joint.resolution_A, &joint.spot_count}) + for (auto *v : {&joint.angle_deg, &joint.bkg_estimate, &joint.resolution_A, &joint.spot_count, &joint.indexed}) v->resize(joint.image_count, NAN); for (size_t f = 0; f < own.image_count; f++) { const size_t row = first_row + f; @@ -120,6 +120,7 @@ void FillJointPlotRows(PerImagePlot &joint, size_t first_row, int dataset, const joint.bkg_estimate[row] = Element(own.bkg_estimate, f); joint.resolution_A[row] = Element(own.resolution_A, f); joint.spot_count[row] = Element(own.spot_count, f); + joint.indexed[row] = Element(own.indexed, f); } } @@ -141,30 +142,23 @@ std::string PerImagePlotText(const PerImagePlot &plot) { "# cc_to_merge: correlation of the image's observations with the merged data\n" "# cc_n: observations that correlation was taken over\n" "# merged: 1 where the image's observations are in the merged data, 0 where not\n" - "# disposition: 1 merged, 0.5 downgraded, 0 rejected\n"; + "# disposition: 1 merged, 0.5 downgraded, 0 rejected\n" + "# indexed: 1 where the per-image indexer indexed the image, 0 where not\n"; for (size_t k = 0; k < plot.dataset_files.size(); k++) file += fmt::format("# dataset {}: {}\n", k + 1, plot.dataset_files[k]); file += "#" + fmt::format("{:>7} {:>7} {:>7} {:>10} {:>12} {:>12} {:>8} {:>12} {:>13} {:>12} {:>8}" - " {:>7} {:>11}\n", + " {:>7} {:>11} {:>7}\n", "ordinal", "dataset", "image", "angle_deg", "bkg", "resolution_A", "spots", - "scale_G", "sigma_M_deg", "cc_to_merge", "cc_n", "merged", "disposition"); + "scale_G", "sigma_M_deg", "cc_to_merge", "cc_n", "merged", "disposition", "indexed"); - // Whether "merged" has anything to answer at all. With no merge behind the run - --no-merge, or - // a sweep nothing indexed on - a column of zeros would read as a sweep every frame of which was - // thrown out, which is a different statement from "no merge was made". - const bool merge_ran = !plot.frame_disposition.empty() - || std::any_of(plot.scale_g.begin(), plot.scale_g.end(), - [](float g) { return std::isfinite(g); }); + // With no merge behind the run - --no-merge, or a sweep nothing indexed on - "merged" has nothing to + // answer: a column of zeros would read as a run every image of which was thrown out, which is a + // different statement from "no merge was made". + const bool merge_ran = !plot.merged.empty(); for (size_t i = 0; i < plot.image_count; i++) { const float scale_g = Element(plot.scale_g, i); - // Did this image's observations reach the merged data? The sweep-quality diagnostic says so - // outright; where it did not run, an image with no fitted scale contributed nothing, which - // is the same statement by a weaker route. - const int merged = i < plot.frame_disposition.size() - ? plot.frame_disposition[i] - != static_cast(FrameDisposition::Rejected) - : std::isfinite(scale_g); + const bool merged = merge_ran && i < plot.merged.size() && plot.merged[i]; // The same answer graded, to plot directly: 1 merged, 0.5 downgraded, 0 rejected. 0.5 marks // "downgraded" and is not a weight - the merge applies no factor of its own to those frames, // they are carried at whatever weight their scale and sigmas give them. A frame that never got @@ -176,15 +170,17 @@ std::string PerImagePlotText(const PerImagePlot &plot) { case FrameDisposition::Downgraded: disposition = 0.5f; break; case FrameDisposition::Rejected: disposition = std::isfinite(scale_g) ? 0.0f : NAN; break; } - } else if (merge_ran && std::isfinite(scale_g)) - disposition = 1.0f; + } else if (merge_ran) + disposition = merged ? 1.0f : std::isfinite(scale_g) ? 0.0f : NAN; + const float indexed = Element(plot.indexed, i); file += fmt::format("{:>8} {:>7} {:>7} {:>10.4f} {:>12.4g} {:>12.4g} {:>8.0f} {:>12.5g} {:>13.4f}" - " {:>12.4f} {:>8} {:>7} {:>11}\n", + " {:>12.4f} {:>8} {:>7} {:>11} {:>7}\n", i, plot.dataset[i], plot.image[i], Element(plot.angle_deg, i), Element(plot.bkg_estimate, i), Element(plot.resolution_A, i), Element(plot.spot_count, i), scale_g, Element(plot.mosaicity_deg, i), Element(plot.cc_to_merge, i), Element(plot.cc_n, i), - merge_ran ? std::to_string(merged) : "nan", disposition); + merge_ran ? std::to_string(static_cast(merged)) : "nan", disposition, + std::isfinite(indexed) ? std::to_string(static_cast(indexed > 0.0f)) : "nan"); } return file; } diff --git a/rugnux/DiagnosticOutput.h b/rugnux/DiagnosticOutput.h index 4da07f2dd..42cbbc939 100644 --- a/rugnux/DiagnosticOutput.h +++ b/rugnux/DiagnosticOutput.h @@ -32,7 +32,8 @@ std::string RenderDetectorDiagnostic(const DiffractionExperiment &experiment, // dataset and image say which input (from 1, in the order given) and which of its own images a row is, // so a run made of several files - stills masters read as one, a joint merge of sweeps - plots per file. // frame_disposition holds FrameDisposition (Merge.h) and is empty when the sweep-quality diagnostic -// did not run. +// did not run; merged (MergeStatistics::image_merged) is empty when no merge ran, indexed when the run +// indexed no image itself (--mode scale). struct PerImagePlot { std::vector dataset_files; // the file of dataset k + 1, for the header std::vector dataset; @@ -46,6 +47,8 @@ struct PerImagePlot { std::vector cc_to_merge; std::vector cc_n; std::vector frame_disposition; + std::vector merged; + std::vector indexed; size_t image_count = 0; // the rows }; diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 9b751a52a..6cb49ca89 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -1330,6 +1330,14 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, // 0 while the whole sweep merges. if (result.indexing_rate.has_value()) Add(s, KeyReal("INDEXING_RATE", *result.indexing_rate, "{:.4f}", rotation)); + // The same as counts: the images the per-image indexer indexed and the lattices it indexed on them + // (a stills measure, as the rate), and the images integrated - each on its main lattice alone. + if (result.images_indexed) + Add(s, KeyInt("IMAGES_INDEXED", *result.images_indexed, rotation)); + if (result.lattices_indexed) + Add(s, KeyInt("LATTICES_INDEXED", *result.lattices_indexed, rotation)); + if (result.images_integrated) + Add(s, KeyInt("IMAGES_INTEGRATED", *result.images_integrated)); Add(s, KeyBool("LATTICE_FOUND", result.consensus_cell.has_value())); if (!result.indexing_ambiguity_operators.empty()) Add(s, KeyText("INDEXING_AMBIGUITY_OPERATORS", result.indexing_ambiguity_operators)); @@ -1737,6 +1745,9 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, const double cc_target = experiment.GetScalingSettings().GetResolutionCCTarget(); Add(s, KeyEnum("MERGE", "PERFORMED", {"PERFORMED", "NOT_PERFORMED"})); + // The images whose observations are in the merged data - the merged column of _plot.txt. + if (!ms.image_merged.empty()) + Add(s, KeyInt("IMAGES_MERGED", std::ranges::count(ms.image_merged, uint8_t{1}))); Add(s, KeyText("INCLUDE_RESOLUTION_RANGE", fmt::format("{:.3f} {:.3f}", o.d_max, o.d_min))); // FITTED_RESOLUTION is presented by this report as "the number to quote", so it must not @@ -3819,6 +3830,19 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, const auto untested = [&](const char *label, std::vector codes) { doc.checks.push_back({label, std::move(codes), ReportCheck::Status::Undetermined, "not measured", {}}); }; + { + // How many images made it through each stage, as counted in sections 1-3. + std::string images = fmt::format("{} processed", result.images_processed); + if (result.images_indexed && !rotation) + images += fmt::format(", {} indexed ({} lattices)", *result.images_indexed, + result.lattices_indexed.value_or(0)); + if (result.images_integrated) + images += fmt::format(", {} integrated", *result.images_integrated); + if (result.has_merge_statistics && !result.merge_statistics.image_merged.empty()) + images += fmt::format(", {} merged", + std::ranges::count(result.merge_statistics.image_merged, uint8_t{1})); + row("Images", images); + } if (result.space_group) row("Space group", group); if (result.consensus_cell) diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index 9a20944a5..50eb0e275 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -1472,8 +1472,12 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b plot.cc_to_merge = scaling.image_cc; plot.cc_n = scaling.image_cc_n; } - if (result.has_merge_statistics) + if (result.has_merge_statistics) { plot.frame_disposition = result.merge_statistics.sweep_quality.frame_disposition; + plot.merged = result.merge_statistics.image_merged; + } + if (!joint) + plot.indexed = result.image_indexed; // One input as far as this run knows: the caller renumbers a run of several (rugnux_cli.cpp). // The goniometer is already shifted to the processed range, so ordinal i is at its angle i. const auto goniometer = experiment_.GetGoniometer(); @@ -1497,6 +1501,11 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b end_msg.image_scale_factor = scaling.image_scale_g; end_msg.image_scale_cc = scaling.image_cc; end_msg.image_scale_mosaicity = scaling.mosaicity_deg; + if (result.has_merge_statistics) { + const auto &merged = result.merge_statistics.image_merged; + end_msg.image_merged.assign(merged.begin(), merged.begin() + std::min(end_msg.max_image_number, + merged.size())); + } } if (result.cell_esd) { diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 28d0d9e27..0cbb3312f 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -592,7 +592,7 @@ struct RugnuxOutputs { }; // One of several _process.h5 files --mode scale merged as one: its frames, those of them indexed, and those of -// them that went into the merge (MergedCrystals). +// them that went into the merge (MergeStatistics::image_merged). struct ScaleInputCount { std::string file; int64_t images = 0, indexed = 0, merged = 0; @@ -637,6 +637,10 @@ struct ProcessResult { std::optional indexing_rate; // Per image of this pass, by ordinal (image start + ordinal * stride): whether it indexed (> 0). std::vector image_indexed; + // This pass's images counted (absent where the pass did not count them): those the per-image indexer + // indexed, the lattices it indexed on them (an image of two crystals counts two), and those integrated - + // on the main lattice alone: an extra lattice of an image is indexed, not integrated. + std::optional images_indexed, lattices_indexed, images_integrated; // How far the merged data are expected to reach, predicted from spot finding alone (the median of // the per-image estimates, see SpotUtils.h GetResolution). Independent of the merge, so it can be // read against INCLUDE_RESOLUTION_RANGE - or on a run that never merges. diff --git a/rugnux/RugnuxImagePass.cpp b/rugnux/RugnuxImagePass.cpp index cdb00ddb6..cfd61bc5c 100644 --- a/rugnux/RugnuxImagePass.cpp +++ b/rugnux/RugnuxImagePass.cpp @@ -772,6 +772,18 @@ void Rugnux::ImagePass(PipelineLocals &p) { result.mean_processing_time = plots.GetMeanProcessingTime(); result.indexing_rate = plots.GetIndexingRate(); plots.GetPlotRaw(result.image_indexed, PlotType::IndexingRate, ""); + { + std::vector lattices; + plots.GetPlotRaw(lattices, PlotType::IndexingLatticeCount, ""); + result.images_indexed = std::ranges::count_if(result.image_indexed, [](float v) { return v > 0.0f; }); + result.lattices_indexed = 0; + for (size_t i = 0; i < lattices.size() && i < result.image_indexed.size(); ++i) + if (result.image_indexed[i] > 0.0f) + *result.lattices_indexed += std::lround(lattices[i]); + if (indexer) + result.images_integrated = std::ranges::count_if(indexer->GetIntegrationOutcome(), + [](const IntegrationOutcome &o) { return !o.reflections.empty(); }); + } // Powder calibration: the whole run has now been accumulated - the summed (q x azimuth) profile in // plots, or the pooled spot list - so fit the detector geometry to it. The caller reports it and diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index da77c5896..fb78fc3d0 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -852,6 +852,7 @@ void StillsMergeScaled(const DiffractionExperiment &experiment, const std::vecto phase("Computing statistics"); TimingMark stats_mark(logger, "stills statistics"); out.statistics = merge_engine.MergeStats(out.merged, outcomes, reference_data, out.d_min); + out.statistics.image_merged = merge_engine.ImagesMerged(); // The same kept reflections binned over the reference range (--report-resolution); the // stills error model is fitted over the whole range already, so no ISa of its own. if (const auto range = cut_ss.GetReportResolutionRange_A(); range && !for_search) diff --git a/rugnux/StillsLiveMerge.cpp b/rugnux/StillsLiveMerge.cpp index 8dd019516..a8ea43901 100644 --- a/rugnux/StillsLiveMerge.cpp +++ b/rugnux/StillsLiveMerge.cpp @@ -282,6 +282,7 @@ ProcessResult StillsLiveResult(const DiffractionExperiment &experiment, const St result.images_processed = static_cast(images_analysed); if (images_analysed > 0) result.indexing_rate = static_cast(images_indexed) / static_cast(images_analysed); + result.images_indexed = images_indexed; result.consensus_cell = experiment.GetUnitCell(); result.space_group = experiment.GetGemmiSpaceGroup(); result.powder_measured = false; @@ -369,16 +370,3 @@ void AppendScaleInput(std::vector &outcomes, std::vector &outcomes, - size_t first, size_t last) { - MergeOnTheFly merge(experiment); - merge.ReferenceCell(experiment.GetUnitCell()); - merge.FilterByImageCC(experiment.GetScalingSettings().GetMinCCForImage() > 0.0); - size_t n = 0; - for (size_t i = first; i < last; ++i) - n += !merge.Mask(outcomes[i]) - && std::any_of(outcomes[i].reflections.begin(), outcomes[i].reflections.end(), [](const Reflection &r) { - return r.image_scale_corr > 0.0f && std::isfinite(r.image_scale_corr); - }); - return n; -} diff --git a/rugnux/StillsLiveMerge.h b/rugnux/StillsLiveMerge.h index dcb569aea..8f1082442 100644 --- a/rugnux/StillsLiveMerge.h +++ b/rugnux/StillsLiveMerge.h @@ -146,7 +146,3 @@ std::string ScaleInputsRefusal(const std::vector &inputs, bool cell_ // frames already there, so that frames of different files stay distinct. void AppendScaleInput(std::vector &outcomes, std::vector &&more); -// Of the frames [first, last) as MergeStills left them, the crystals that went into the merge: on the cell -// merged in, scaled, and not left out by --min-image-cc. -size_t MergedCrystals(const DiffractionExperiment &experiment, const std::vector &outcomes, - size_t first, size_t last); diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index f8f180ba4..b804c0616 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -2796,8 +2796,9 @@ int RugnuxInputRun::PrepareRugnuxInput() { experiment.ImagesPerTrigger(static_cast(so.images)); for (size_t k = 0; k < input_counts.size(); ++k) { auto &c = input_counts[k]; - c.merged = static_cast(MergedCrystals(experiment, reflections, input_first_frame[k], - input_first_frame[k + 1])); + const auto &merged = so.merge.statistics.image_merged; + c.merged = std::count(merged.begin() + input_first_frame[k], merged.begin() + input_first_frame[k + 1], + uint8_t{1}); logger.Info("Input {}: {} of {} indexed crystals merged, {} left out, from {}", k + 1, c.merged, c.indexed, c.indexed - c.merged, c.file); } @@ -3010,11 +3011,42 @@ int RugnuxInputRun::PrepareRugnuxInput() { &scaled_fulls); } + // The per-image table of this merge: which input and image each row is, the scaling columns and whether + // the image's observations are in the merged data. The per-image analysis of the run that wrote the + // file is not re-read, so its columns (and indexed) are nan. + if (!output_prefix.empty()) { + PerImagePlot plot; + plot.image_count = reflections.size(); + plot.dataset.resize(plot.image_count); + plot.image.resize(plot.image_count); + plot.angle_deg.assign(plot.image_count, NAN); + const auto goniometer = experiment.GetGoniometer(); + for (size_t k = 0; k + 1 < input_first_frame.size(); ++k) { + plot.dataset_files.push_back(std::filesystem::absolute(input_files[k]).lexically_normal().string()); + for (size_t i = input_first_frame[k]; i < input_first_frame[k + 1]; ++i) { + plot.dataset[i] = static_cast(k) + 1; + plot.image[i] = static_cast(i - input_first_frame[k] + (k == 0 ? start_image : 0)); + if (goniometer) + plot.angle_deg[i] = goniometer->GetAngle_deg(static_cast(plot.image[i])) + + goniometer->GetWedge_deg() / 2.0f; + } + } + const ScalingResult scaling(reflections); + plot.scale_g = scaling.image_scale_g; + plot.mosaicity_deg = scaling.mosaicity_deg; + plot.cc_to_merge = scaling.image_cc; + plot.cc_n = scaling.image_cc_n; + plot.frame_disposition = merged_statistics.sweep_quality.frame_disposition; + plot.merged = merged_statistics.image_merged; + WritePerImagePlot(output_prefix, plot); + } + // --mode scale re-merges stored reflections, so it determines a merging result and gets the // same report. It has no ProcessResult of its own; fill the parts that exist here. ProcessResult scale_result; scale_result.images_processed = input_files.size() > 1 ? reflections.size() : reader.GetNumberOfImages(); scale_result.scale_inputs = input_counts; + scale_result.images_integrated = static_cast(n_images); scale_result.consensus_cell = experiment.GetUnitCell(); scale_result.space_group = experiment.GetGemmiSpaceGroup(); scale_result.used_beam_x_pxl = experiment.GetBeamX_pxl(); @@ -4048,6 +4080,12 @@ int RugnuxInputRun::RunPreparedInput() { logger.Info("Total throughput:{:.2f} MB/s", result.throughput_MBs); if (result.indexing_rate.has_value()) logger.Info("Indexing rate: {:.2f}%", result.indexing_rate.value() * 100.0); + if (result.images_indexed) + logger.Info("Images: {} processed, {} indexed ({} lattices), {} integrated{}", result.images_processed, + *result.images_indexed, result.lattices_indexed.value_or(0), result.images_integrated.value_or(0), + result.has_merge_statistics + ? fmt::format(", {} merged", std::ranges::count(result.merge_statistics.image_merged, uint8_t{1})) + : std::string()); for (size_t k = 0; k < result.dataset_inputs.size(); k++) { const DatasetInputCount &in = result.dataset_inputs[k]; logger.Info(" input {}: {:.2f}% of {} images ({})", k + 1, diff --git a/tests/CBORTest.cpp b/tests/CBORTest.cpp index 648f9e800..8def570ef 100644 --- a/tests/CBORTest.cpp +++ b/tests/CBORTest.cpp @@ -593,6 +593,23 @@ TEST_CASE("CBORSerialize_End_SpaceGroup", "[CBOR]") { CHECK(*out.space_group_number == static_cast(sg->number)); } +TEST_CASE("CBORSerialize_End_ImageMerged", "[CBOR]") { + std::vector buffer(1024 * 1024); + CBORStream2Serializer serializer(buffer.data(), buffer.size()); + + EndMessage message{}; + message.max_image_number = 4; + message.image_indexed = {1, 1, 0, 1}; + message.image_merged = {1, 0, 0, 1}; + REQUIRE_NOTHROW(serializer.SerializeSequenceEnd(message)); + + auto deserialized = CBORStream2Deserialize(buffer.data(), serializer.GetBufferSize()); + REQUIRE(deserialized); + REQUIRE(deserialized->end_message); + CHECK(deserialized->end_message->image_indexed == message.image_indexed); + CHECK(deserialized->end_message->image_merged == message.image_merged); +} + TEST_CASE("CBORSerialize_End_RefinedGeometry", "[CBOR]") { std::vector buffer(1024 * 1024); CBORStream2Serializer serializer(buffer.data(), buffer.size()); diff --git a/tests/DiagnosticOutputTest.cpp b/tests/DiagnosticOutputTest.cpp index 885c8671c..92e207549 100644 --- a/tests/DiagnosticOutputTest.cpp +++ b/tests/DiagnosticOutputTest.cpp @@ -126,6 +126,8 @@ TEST_CASE("DiagnosticPlot_Format") { static_cast(FrameDisposition::Downgraded), static_cast(FrameDisposition::Rejected), static_cast(FrameDisposition::Rejected)}; + plot.merged = {1, 1, 0, 0}; + plot.indexed = {1.0f, 1.0f, 0.0f, 1.0f}; WritePerImagePlot(prefix, plot); const auto all = ReadLines(prefix + "_plot.txt"); @@ -141,7 +143,7 @@ TEST_CASE("DiagnosticPlot_Format") { // Every column is described in the header. const auto legend = Columns(lines[0]); - REQUIRE(legend.size() == 13); + REQUIRE(legend.size() == 14); for (size_t c = 1; c < legend.size(); c++) { bool described = false; for (size_t l = 0; l + 1 < header; l++) @@ -156,6 +158,7 @@ TEST_CASE("DiagnosticPlot_Format") { CHECK(legend[10] == "cc_n"); CHECK(legend[11] == "merged"); CHECK(legend[12] == "disposition"); + CHECK(legend[13] == "indexed"); for (size_t i = 1; i < lines.size(); i++) { const auto row = Columns(lines[i]); @@ -181,6 +184,10 @@ TEST_CASE("DiagnosticPlot_Format") { CHECK(Columns(lines[3])[12] == "nan"); CHECK(Columns(lines[4])[12] == "0"); + // Indexed by the per-image indexer, whatever became of the image in the merge. + CHECK(Columns(lines[3])[13] == "0"); + CHECK(Columns(lines[4])[13] == "1"); + // The count the correlation beside it was taken over. CHECK(Columns(lines[1])[10] == "500"); @@ -191,24 +198,28 @@ TEST_CASE("DiagnosticPlot_Format") { TEST_CASE("DiagnosticPlot_NoDisposition") { const std::string prefix = "diagnostic_plot_nodisp_test"; - // Without the sweep-quality diagnostic, a frame counts as merged when it was scaled at all. + // Stills, with no sweep-quality diagnostic: merged is the merge's own answer, and the disposition + // follows it - a crystal scaled but left out (here the third) is rejected, one never scaled is nan. PerImagePlot plot; OneDataset(plot, 3); plot.scale_g = {1.0f, NAN, 0.8f}; + plot.merged = {1, 0, 0}; WritePerImagePlot(prefix, plot); const auto lines = Rows(ReadLines(prefix + "_plot.txt")); REQUIRE(lines.size() == 3); CHECK(Columns(lines[0])[11] == "1"); CHECK(Columns(lines[1])[11] == "0"); - CHECK(Columns(lines[2])[11] == "1"); + CHECK(Columns(lines[2])[11] == "0"); CHECK(Columns(lines[0])[12] == "1"); CHECK(Columns(lines[1])[12] == "nan"); - CHECK(Columns(lines[2])[12] == "1"); + CHECK(Columns(lines[2])[12] == "0"); - // No goniometer: the angle column is present and unplottable, so the columns never shift. - CHECK(Columns(lines[0]).size() == 13); + // No goniometer: the angle column is present and unplottable, so the columns never shift; no + // per-image indexing (--mode scale) leaves indexed nan. + CHECK(Columns(lines[0]).size() == 14); CHECK(Columns(lines[0])[3] == "nan"); + CHECK(Columns(lines[0])[13] == "nan"); } TEST_CASE("DiagnosticPlot_NoMerge") { @@ -242,6 +253,7 @@ TEST_CASE("DiagnosticPlot_JointSweeps") { OneDataset(b, 3); b.angle_deg = {90.05f, 90.15f, 90.25f}; b.spot_count = {7.0f, 8.0f, 9.0f}; + b.indexed = {1.0f, 0.0f, 1.0f}; // The joint merge's table: its rows are the sweeps' frames end to end, its scale the joint merge's. PerImagePlot joint; @@ -273,4 +285,8 @@ TEST_CASE("DiagnosticPlot_JointSweeps") { CHECK(Columns(lines[2])[4] == "nan"); CHECK(std::stof(Columns(lines[3])[6]) == 8.0f); CHECK(std::stof(Columns(lines[2])[7]) == 2.0f); + // Each sweep's own per-image indexing. + CHECK(Columns(lines[0])[13] == "nan"); + CHECK(Columns(lines[3])[13] == "0"); + CHECK(Columns(lines[4])[13] == "1"); } diff --git a/tests/JFJochReaderTest.cpp b/tests/JFJochReaderTest.cpp index 926aa853a..70406c4b5 100644 --- a/tests/JFJochReaderTest.cpp +++ b/tests/JFJochReaderTest.cpp @@ -1509,6 +1509,8 @@ TEST_CASE("JFJochReader_NiggliClass_VDS", "[HDF5][Full]") { REQUIRE_NOTHROW(reader.ReadFile("test95_master.h5")); auto dataset = reader.GetDataset(); CHECK(dataset->experiment.GetImageNum() == 2); + // Written without a merge, as every file before imageMerged existed: no merge status, no error. + CHECK(dataset->merge_result.empty()); std::shared_ptr reader_image, reader_image_2; diff --git a/tests/MergeScaleTest.cpp b/tests/MergeScaleTest.cpp index b41e7176a..77a05dd2d 100644 --- a/tests/MergeScaleTest.cpp +++ b/tests/MergeScaleTest.cpp @@ -788,6 +788,45 @@ TEST_CASE("MergeOnTheFly_AddImagesIsTheSerialMerge") { CHECK(std::isnan(serial_stats.overall.r_pim)); } +// Which images reached the merge: one with an observation in it does; one with no reflections, one never +// scaled and one off the cell merged in do not. +TEST_CASE("MergeOnTheFly_ImagesMerged") { + DiffractionExperiment x; + x.SetSpaceGroup(TestSpaceGroup()); + const UnitCell cell{50.0f, 60.0f, 70.0f, 90.0f, 90.0f, 90.0f}; + const UnitCell other_cell{55.0f, 60.0f, 70.0f, 90.0f, 90.0f, 90.0f}; + + const auto image = [](const UnitCell &c, float image_scale_corr) { + IntegrationOutcome o; + o.latt = CrystalLattice(c); + for (int h = 1; h <= 5; ++h) { + Reflection r{}; + r.h = h; + r.k = 1; + r.l = 2; + r.d = 10.0f / static_cast(h); + r.I = 100.0f; + r.sigma = 10.0f; + r.prescaling_corr = 1.0f; + r.partiality = 1.0f; + r.image_scale_corr = image_scale_corr; + o.reflections.push_back(r); + } + return o; + }; + std::vector outcomes; + outcomes.push_back(image(cell, 1.0f)); + outcomes.emplace_back(); // nothing integrated + outcomes.push_back(image(cell, 0.0f)); // never scaled + outcomes.push_back(image(other_cell, 1.0f)); // off the cell merged in + outcomes.push_back(image(cell, 1.0f)); + + MergeOnTheFly merge(x); + merge.ReferenceCell(cell); + merge.AddImages(outcomes); + CHECK(merge.ImagesMerged() == std::vector{1, 0, 0, 0, 1}); +} + // With the error model fitted, the merge takes every observation, however small its partiality, but reports a // reflection only if one of its observations reaches the existence partiality (0.2). TEST_CASE("MergeOnTheFly_EveryObservationMergedReflectionReportedOnlyIfOneReachesTheExistencePartiality") { diff --git a/tests/ResultReportTest.cpp b/tests/ResultReportTest.cpp index f81cc7b18..ecc8062bb 100644 --- a/tests/ResultReportTest.cpp +++ b/tests/ResultReportTest.cpp @@ -329,6 +329,7 @@ TEST_CASE("SweepQuality_HDF5RoundTrip", "[HDF5][Full][Diagnostics]") { end_message.sweep_quality_reasons = ReasonVocabulary(); end_message.frame_disposition = expected_disposition; end_message.frame_disposition_codes = DispositionVocabulary(); + end_message.image_merged = {1, 1, 0, 0, 1, 1}; FileWriter writer(start_message); std::vector image(x.GetPixelsNum(), 42); @@ -351,6 +352,8 @@ TEST_CASE("SweepQuality_HDF5RoundTrip", "[HDF5][Full][Diagnostics]") { CHECK(dataset->sweep_quality_reasons == ReasonVocabulary()); CHECK(dataset->frame_disposition == expected_disposition); CHECK(dataset->frame_disposition_codes == DispositionVocabulary()); + // Merged or not, per image: the viewer's image list sorts on it. + CHECK(dataset->merge_result == std::vector{1, 1, 0, 0, 1, 1}); } REQUIRE(H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_ALL) == 0); @@ -1155,6 +1158,47 @@ TEST_CASE("ResultReport_RotationScale", "[Diagnostics]") { CHECK(kept.find("\nGONIOMETER_ROTATION_SCALE_SUSPECT= FALSE\n") != std::string::npos); } +// How many images made it through each stage, as counts beside the rate: indexed (with the lattices on them), +// integrated and merged - in the keys and in the summary line. +TEST_CASE("ResultReport_ImageCounts", "[Diagnostics]") { + DiffractionExperiment x(DetJF(1)); + ProcessResult result; + result.images_processed = 10; + result.indexing_rate = 0.6f; + result.images_indexed = 6; + result.lattices_indexed = 8; + result.images_integrated = 6; + result.consensus_cell = UnitCell{.a = 50.0f, .b = 60.0f, .c = 70.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + result.has_merge_statistics = true; + result.merge_statistics.stills = true; + result.merge_statistics.image_merged = {1, 1, 0, 1, 0, 0, 0, 0, 0, 0}; + + const auto text = RenderResultReport("prefix", "in.h5", x, result); + CHECK(text.find("\nIMAGES_PROCESSED= 10\n") != std::string::npos); + CHECK(text.find("\nIMAGES_INDEXED= 6\n") != std::string::npos); + CHECK(text.find("\nLATTICES_INDEXED= 8\n") != std::string::npos); + CHECK(text.find("\nIMAGES_INTEGRATED= 6\n") != std::string::npos); + CHECK(text.find("\nIMAGES_MERGED= 3\n") != std::string::npos); + CHECK(text.find("10 processed, 6 indexed (8 lattices), 6 integrated, 3 merged") != std::string::npos); + + // Rotation indexes the sweep as a whole: the per-image indexer's counts are for developers, as its rate. + x.Goniometer(GoniometerAxis("omega", 0.0f, 0.1f, Coord(1, 0, 0), {})); + x.ImportIndexingSettings(IndexingSettings().RotationIndexing(true)); + const auto rotation = RenderResultReport("prefix", "in.h5", x, result); + CHECK(rotation.find("IMAGES_INDEXED=") == std::string::npos); + CHECK(rotation.find("LATTICES_INDEXED=") == std::string::npos); + CHECK(rotation.find("\nIMAGES_INTEGRATED= 6\n") != std::string::npos); + CHECK(rotation.find("\nIMAGES_MERGED= 3\n") != std::string::npos); + CHECK(rotation.find("10 processed, 6 integrated, 3 merged") != std::string::npos); + RunProvenance dev; + dev.developer = true; + CHECK(RenderResultReport("prefix", "in.h5", x, result, dev).find("\nIMAGES_INDEXED= 6\n") != std::string::npos); + + // No merge: no count of merged images, rather than a zero. + result.has_merge_statistics = false; + CHECK(RenderResultReport("prefix", "in.h5", x, result).find("IMAGES_MERGED=") == std::string::npos); +} + TEST_CASE("LeftoverLattice_Misorientation", "[Diagnostics]") { const CrystalLattice triclinic(52.0f, 61.0f, 73.0f, 81.0f, 76.0f, 69.0f); const auto rotated = [](const CrystalLattice &l, float deg, const Coord &axis) { diff --git a/viewer/windows/JFJochViewerImageListWindow.cpp b/viewer/windows/JFJochViewerImageListWindow.cpp index 8fab1b7b9..0e5a781ca 100644 --- a/viewer/windows/JFJochViewerImageListWindow.cpp +++ b/viewer/windows/JFJochViewerImageListWindow.cpp @@ -6,10 +6,36 @@ #include #include #include +#include namespace { constexpr int ScaleSortRole = Qt::UserRole + 1; constexpr double ScaleNotAvailableSortValue = -1.0e100; + // The Yes/No columns, filtered from the header's context menu. + constexpr int IndexColumn = 2; + constexpr int MergedColumn = 3; + const QStringList YesNoValues = {"Yes", "No", "N/A"}; +} + +void ImageListFilterModel::SetFilter(int column, const QSet &values) { + allowed[column] = values; + invalidateFilter(); +} + +void ImageListFilterModel::ClearFilter(int column) { + allowed.remove(column); + invalidateFilter(); +} + +bool ImageListFilterModel::Allows(int column, const QString &value) const { + return !allowed.contains(column) || allowed[column].contains(value); +} + +bool ImageListFilterModel::filterAcceptsRow(int source_row, const QModelIndex &source_parent) const { + for (auto it = allowed.cbegin(); it != allowed.cend(); ++it) + if (!it.value().contains(sourceModel()->index(source_row, it.key(), source_parent).data().toString())) + return false; + return true; } JFJochViewerImageListWindow::JFJochViewerImageListWindow(QWidget *parent) : JFJochHelperWindow(parent) { @@ -24,7 +50,7 @@ JFJochViewerImageListWindow::JFJochViewerImageListWindow(QWidget *parent) : JFJo tableModel = new QStandardItemModel(this); setupTableModel(); - proxyModel = new QSortFilterProxyModel(this); + proxyModel = new ImageListFilterModel(this); proxyModel->setSourceModel(tableModel); proxyModel->setSortRole(ScaleSortRole); @@ -47,23 +73,61 @@ JFJochViewerImageListWindow::JFJochViewerImageListWindow(QWidget *parent) : JFJo resize(800, 600); connect(tableView, &QTableView::doubleClicked, this, &JFJochViewerImageListWindow::onTableRowDoubleClicked); + + // A click on a header sorts; its context menu filters the Yes/No columns. + tableView->horizontalHeader()->setContextMenuPolicy(Qt::CustomContextMenu); + connect(tableView->horizontalHeader(), &QHeaderView::customContextMenuRequested, + this, &JFJochViewerImageListWindow::onHeaderMenu); +} + +void JFJochViewerImageListWindow::onHeaderMenu(const QPoint &pos) { + const int column = tableView->horizontalHeader()->logicalIndexAt(pos); + if (column != IndexColumn && column != MergedColumn) + return; + + QMenu menu(this); + for (const QString &value : YesNoValues) { + QAction *action = menu.addAction(value); + action->setCheckable(true); + action->setChecked(proxyModel->Allows(column, value)); + } + menu.addSeparator(); + QAction *all = menu.addAction("Show all"); + QAction *chosen = menu.exec(tableView->horizontalHeader()->mapToGlobal(pos)); + if (!chosen) + return; + + QSet values; + if (chosen != all) + for (QAction *action : menu.actions()) + if (action->isCheckable() && action->isChecked()) + values.insert(action->text()); + if (chosen == all || values.size() == YesNoValues.size()) + proxyModel->ClearFilter(column); + else + proxyModel->SetFilter(column, values); + + const QString name = column == IndexColumn ? "Index" : "Merged"; + tableModel->setHeaderData(column, Qt::Horizontal, proxyModel->IsFiltered(column) ? name + " *" : name); } void JFJochViewerImageListWindow::setupTableModel() { - tableModel->setColumnCount(8); + tableModel->setColumnCount(9); tableModel->setHeaderData(0, Qt::Horizontal, "#"); tableModel->setHeaderData(1, Qt::Horizontal, "Bkg"); tableModel->setHeaderData(2, Qt::Horizontal, "Index"); - tableModel->setHeaderData(3, Qt::Horizontal, "Spots"); - tableModel->setHeaderData(4, Qt::Horizontal, "Res. [A]"); - tableModel->setHeaderData(5, Qt::Horizontal, "Max val"); - tableModel->setHeaderData(6, Qt::Horizontal, "Scale factor"); - tableModel->setHeaderData(7, Qt::Horizontal, "Scale CC"); + tableModel->setHeaderData(3, Qt::Horizontal, "Merged"); + tableModel->setHeaderData(4, Qt::Horizontal, "Spots"); + tableModel->setHeaderData(5, Qt::Horizontal, "Res. [A]"); + tableModel->setHeaderData(6, Qt::Horizontal, "Max val"); + tableModel->setHeaderData(7, Qt::Horizontal, "Scale factor"); + tableModel->setHeaderData(8, Qt::Horizontal, "Scale CC"); } void JFJochViewerImageListWindow::addDataRow(int imageNumber, double backgroundEstimate, const QString& indexingResult, + const QString& mergeResult, int spotCount, double resolutionEstimate, int64_t max_value, @@ -81,14 +145,16 @@ void JFJochViewerImageListWindow::addDataRow(int imageNumber, double backgroundE bgItem->setData(backgroundEstimate, ScaleSortRole); rowItems.append(bgItem); - QStandardItem *indexingItem = new QStandardItem(indexingResult); - if (indexingResult == "Yes") - indexingItem->setData(1, ScaleSortRole); - else if (indexingResult == "No") - indexingItem->setData(0, ScaleSortRole); - else - indexingItem->setData(-1, ScaleSortRole); - rowItems.append(indexingItem); + for (const QString &result : {indexingResult, mergeResult}) { + QStandardItem *item = new QStandardItem(result); + if (result == "Yes") + item->setData(1, ScaleSortRole); + else if (result == "No") + item->setData(0, ScaleSortRole); + else + item->setData(-1, ScaleSortRole); + rowItems.append(item); + } QStandardItem *spotItem = new QStandardItem(); spotItem->setData(spotCount, Qt::DisplayRole); @@ -136,7 +202,7 @@ void JFJochViewerImageListWindow::clearAllData() { void JFJochViewerImageListWindow::datasetLoaded(std::shared_ptr in_dataset) { dataset = std::move(in_dataset); - // One row of eight QStandardItems per image, arriving on every dataset tick of a live + // One row of nine QStandardItems per image, arriving on every dataset tick of a live // acquisition. Not worth doing for a window nobody has open. if (!isVisible()) { pending_fill = true; @@ -181,6 +247,10 @@ void JFJochViewerImageListWindow::fillTable() { indexing_result = "Yes"; } + QString merge_result = "N/A"; + if (dataset->merge_result.size() > i) + merge_result = dataset->merge_result[i] == 0 ? "No" : "Yes"; + int64_t max_value = 0; if (dataset->max_value.size() > i) max_value = dataset->max_value[i]; @@ -196,6 +266,7 @@ void JFJochViewerImageListWindow::fillTable() { addDataRow(i + 1, bkg_estimate, indexing_result, + merge_result, spot_count, res_estimation, max_value, diff --git a/viewer/windows/JFJochViewerImageListWindow.h b/viewer/windows/JFJochViewerImageListWindow.h index 9752ff3ce..162b8aae0 100644 --- a/viewer/windows/JFJochViewerImageListWindow.h +++ b/viewer/windows/JFJochViewerImageListWindow.h @@ -7,21 +7,37 @@ #include #include #include +#include +#include #include "JFJochHelperWindow.h" #include "../../reader/JFJochReader.h" +// The rows whose text in each filtered column is one of the values let through there, as a spreadsheet's +// column filter; a column with no entry is not filtered. +class ImageListFilterModel : public QSortFilterProxyModel { + QMap> allowed; +public: + using QSortFilterProxyModel::QSortFilterProxyModel; + void SetFilter(int column, const QSet &values); + void ClearFilter(int column); + bool IsFiltered(int column) const { return allowed.contains(column); } + bool Allows(int column, const QString &value) const; +protected: + bool filterAcceptsRow(int source_row, const QModelIndex &source_parent) const override; +}; + class JFJochViewerImageListWindow : public JFJochHelperWindow { Q_OBJECT QTableView *tableView; QStandardItemModel *tableModel; - QSortFilterProxyModel *proxyModel; + ImageListFilterModel *proxyModel; void setupTableModel(); void addDataRow(int imageNumber, double backgroundEstimate, - const QString &indexingResult, int spotCount, + const QString &indexingResult, const QString &mergeResult, int spotCount, double resolutionEstimate, int64_t max_value, double image_scale_factor, double image_scale_cc_percent); @@ -51,6 +67,7 @@ public slots: void imageLoaded(std::shared_ptr image) override; private slots: void onTableRowDoubleClicked(const QModelIndex &index); + void onHeaderMenu(const QPoint &pos); }; diff --git a/writer/HDF5NXmx.cpp b/writer/HDF5NXmx.cpp index e8af8eb5e..92b95ff78 100644 --- a/writer/HDF5NXmx.cpp +++ b/writer/HDF5NXmx.cpp @@ -1306,6 +1306,7 @@ void NXmx::EndResultVectors(const EndMessage &end) { SaveVectorIfMissing(*hdf5_file, "/entry/MX/peakCountLowRes", end.spot_count_low_res); SaveVectorIfMissing(*hdf5_file, "/entry/MX/peakCountIndexed", end.spot_count_indexed); SaveVectorIfMissing(*hdf5_file, "/entry/MX/imageIndexed", end.image_indexed); + SaveVectorIfMissing(*hdf5_file, "/entry/MX/imageMerged", end.image_merged); SaveVectorIfMissing(*hdf5_file, "/entry/MX/indexedLatticeCount", end.indexed_lattice_count); SaveVectorIfMissing(*hdf5_file, "/entry/MX/bkgEstimate", end.v_bkg_estimate); SaveVectorIfMissing(*hdf5_file, "/entry/MX/spindleBlindFraction", end.v_spindle_blind_fraction);