reader: a link to a file that is not there is not a dataset that exists

Every DECTRIS Eiger master links saturation_value, pixel_mask,
bit_depth_readout and serial_number into a companion <prefix>_meta.h5, and
that file is routinely not kept when a dataset is archived or deposited. The
existence test asked only whether the LINK was written, which it is, so every
optional-field guard in the reader answered yes and the read that followed
threw. A deposited Eiger 16M set could not be opened at all, over values the
reader was perfectly prepared to do without.

Exists() now asks the second question too - whether the object the link names
can be reached - so an orphaned link reads as absent and the fallbacks behind
it do their job.

The saturation value is then allowed to be missing outright, because on such a
file it is: neither the NXmx name nor the DECTRIS one is readable, and there is
no third place to look. Left unset, GetSaturationLimit() falls back to the
container's own overflow. That is the safe direction - it can only fail to call
a pixel saturated, where too LOW a value drops the whole reflection and
silently removes the strongest data - and the run says out loud that nothing
will be called saturated.

The set that could not be opened now processes to 2.17 A against a deposited
2.40 A, in the deposited space group, with a cell agreeing to 0.08%. Output is
byte-identical on datasets that already opened.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
(cherry picked from commit 0637979f6d95b446406ab70d1f3982841d195b36)

docs/CHANGELOG.md

@@ -2,6 +2,7 @@
 ## 1.0.0
 ### 1.0.0-rc.166
 * `rugnux` reads a PILATUS miniCBF rotation sweep natively, with no conversion: naming any frame - or the directory holding it - processes the whole sweep that frame's template belongs to. `--mode scale` still needs a `_process.h5`.
+* A master whose companion `_meta.h5` was not kept still opens: the fields an Eiger master links into that file are treated as absent rather than as a read error, and a run that finds no saturation value anywhere says so and carries on instead of refusing the dataset.
 * `jfjoch_viewer` opens PILATUS miniCBF sweeps - naming any frame opens the whole sweep - and can run a processing job on one.
 * A miniCBF sweep takes its rotation axis from the goniometer angles the header states, instead of assuming the axis every such file was previously assumed to have.
 * A miniCBF sweep takes the mounting from the imgCIF axis table its header carries - which laboratory direction the image's columns and rows run along, and which the spindle turns about - and, where there is no table, from a `+SLOW` on the `Oscillation_axis` line, which says the spindle runs along the image's slow direction.

reader/HDF5MetadataSource.cpp

@@ -986,10 +986,26 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen
                 ReadLength_m(*master_file, "/entry/instrument/detector/sensor_thickness") * 1e6);
         if (master_file->Exists("/entry/instrument/detector/sensor_material"))
             detector.SensorMaterial(master_file->GetString("/entry/instrument/detector/sensor_material"));
-        detector.SaturationLimit(SaturationLimitFromValue(
-                ReadIntWithLegacyFallback(*master_file,
-                        "/entry/instrument/detector/saturation_value",
-                        "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff")));
+        // Optional, because a file that states no saturation value anywhere is a real and common
+        // thing: an Eiger master links saturation_value into a companion _meta.h5, and a deposited
+        // dataset frequently does not include that file, leaving neither the NXmx name nor the
+        // DECTRIS one readable. Left unset, DiffractionExperiment::GetSaturationLimit() falls back
+        // to the container's own overflow, which is the safe direction - it can only fail to call a
+        // pixel saturated, where too LOW a value drops the whole reflection and silently removes the
+        // strongest data (see BitDepthImage below). Refusing the file outright is the one option that
+        // helps nobody.
+        if (master_file->Exists("/entry/instrument/detector/saturation_value")
+            || master_file->Exists("/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff"))
+            detector.SaturationLimit(SaturationLimitFromValue(
+                    ReadIntWithLegacyFallback(*master_file,
+                            "/entry/instrument/detector/saturation_value",
+                            "/entry/instrument/detector/detectorSpecific/countrate_correction_count_cutoff")));
+        else
+            Logger("HDF5Reader").Warning("The file states no saturation value - neither NXmx saturation_value nor "
+                           "the DECTRIS countrate_correction_count_cutoff is readable, which is what "
+                           "an Eiger master looks like when its companion _meta.h5 was not kept. No "
+                           "pixel will be called saturated; if this detector overloads, its strongest "
+                           "reflections will be integrated as if they were valid.");
         // The reader hands every image out as signed int32 whatever the file stored (see PixelSigned
         // below), so that is the container depth the rest of the code has to see. DetectorSetup defaults
         // DECTRIS to 16 bits and GetByteDepthImage() prefers the detector's value over the image

tests/HDF5WritingTest.cpp

@@ -1451,3 +1451,34 @@ TEST_CASE("HDF5FilePusher_finalize_failure_recovers", "[HDF5FilePusher][Repro]")
             std::filesystem::remove(e.path());
     REQUIRE(H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_ALL) == 0);
 }
+
+// A link that is written in the file and points at something not there is NOT an existing dataset.
+// Every DECTRIS Eiger master links saturation_value, pixel_mask, bit_depth_readout and
+// serial_number into a companion <prefix>_meta.h5, and that file is routinely not kept when a
+// dataset is archived or deposited. Asking only whether the LINK exists then answers yes and the
+// read that follows throws, which turns every optional-field guard in the reader into a hard
+// failure - measured on a deposited Eiger 16M set that could not be opened at all.
+TEST_CASE("HDF5Objects_dangling_external_link_does_not_exist", "[HDF5][Unit]") {
+    const std::string fname = "test_dangling_link.h5";
+    remove(fname.c_str());
+    {
+        HDF5File file(fname);
+        HDF5Group group(file, "/entry");
+        group.SaveScalar("present", static_cast<int64_t>(7));
+        // ...and a link into a file that does not exist, exactly as an orphaned Eiger master has.
+        REQUIRE(H5Lcreate_external("no_such_meta.h5", "/_dectris/whatever",
+                                   group.GetID(), "absent", H5P_DEFAULT, H5P_DEFAULT) >= 0);
+    }
+    {
+        HDF5ReadOnlyFile file(fname);
+        CHECK(file.Exists("/entry/present"));
+        CHECK(file.GetInt("/entry/present") == 7);
+        // The link is there, the object is not.
+        CHECK(H5Lexists(file.GetID(), "/entry/absent", H5P_DEFAULT) > 0);
+        CHECK_FALSE(file.Exists("/entry/absent"));
+        // ...so an optional read of it falls back instead of throwing.
+        CHECK(file.GetString("/entry/absent", "fallback") == "fallback");
+    }
+    remove(fname.c_str());
+    REQUIRE(H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_ALL) == 0);
+}

writer/HDF5Objects.cpp

@@ -1072,7 +1072,16 @@ std::string HDF5Object::GetString(const std::string &name, const std::string &de
 
 bool HDF5Object::Exists(const std::string &name) const {
     H5E_BEGIN_TRY {
-        return H5Lexists(GetID(), name.c_str(), H5P_DEFAULT) > 0;
+        // The LINK has to be there and the object it names has to be reachable. Those are two
+        // different questions for an external link, and every DECTRIS Eiger master asks the second
+        // one: it links saturation_value, pixel_mask, bit_depth_readout and serial_number into a
+        // companion <prefix>_meta.h5 that is routinely not kept when a dataset is archived or
+        // deposited. H5Lexists then says yes - the link is written in the master - and the read that
+        // follows throws, which turns every optional-field guard in the reader into a hard failure
+        // and refuses the whole file over a value it was prepared to do without.
+        if (H5Lexists(GetID(), name.c_str(), H5P_DEFAULT) <= 0)
+            return false;
+        return H5Oexists_by_name(GetID(), name.c_str(), H5P_DEFAULT) > 0;
     } H5E_END_TRY;
     return false;
 }