v1.0.0-rc.171 #81

Merged

leonarski_f merged 13 commits from rc171 into main 2026-09-17 14:42:53 +02:00

Conversation 0 Commits 13 Files Changed 211

+2857 -775

13 Commits

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	d6a2b9d19e	broker: error responses carry the content type the OpenAPI spec declares ... The 500 handlers built a correct error_message object but sent it through send_plain(), which labels the body text/plain. The spec declares 500 as application/json + error_message, and the generated clients key their deserialization off the content type: the Python client kept the body as a raw str, handed it to ErrorMessage.from_dict(), and pydantic raised - the exception swallowed by the finally: in response_deserialize, so the caller got a ServiceException with data=None and only the raw text in .body. send_plain becomes send_error, taking the content type alongside the code and the body; the two exception handlers now return it. 400 stays a plain-text exception string, exactly as the spec says; 500 is JSON. Two further 500s did not carry an error_message object at all and now do: the generic std::exception branch of handleParsingException, which returned a bare what(), and ProcessOutput's output-validation failure, which returned the validation dump. A catch-all set_exception_handler covers anything that escapes a route's own handler - httplib would otherwise answer with a bodyless 500, which fails to deserialize the same way. Verified against a running broker: POST /pedestal in the wrong state returns application/json and the generated Python client parses it into ErrorMessage(msg=..., reason='WrongDAQState'); a malformed body still returns 400 text/plain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 13:59:57 +02:00
leonarski_fandClaude Opus 5	51f51ec5bf	tests: the FFT indexers on a frame of pure noise ... The frame that works the indexer hardest is junk, not a crystal: a real lattice prunes the FFT shortlist to four or five distinct directions, while noise leaves dozens, which is what drives the length sort and the degenerate-plane fallback - where a production broker segfaulted twice. There was no test in that regime. Two cases, both over every FFT back-end the build has (GPU FFT under CUDA, CPU FFTW always), so a non-CUDA job covers them as well: - a cloud of 1500 reciprocal-space vectors of random direction and length, with no periodicity in it. Both back-ends return 70 candidate lattices from such a frame, which is the designed behaviour - Run() offers candidates and the caller scores them - so what the test pins is that none of them looks like a crystal: each takes about 1% of the cloud against the ~100% a real lattice takes, and the assertion allows 5%. - the same noise flattened onto a tilted plane. A coplanar shortlist cannot close a cell, so this is the branch that goes looking for the missing row in a 3 deg cap (SearchCap), verified with a temporary probe to enter it on both back-ends and not to enter it on the isotropic frame. The SAME indexer object then has to index a clean lattice correctly, which is the regression test for the cap search putting the direction grid back. The cloud is drawn from a fixed seed, and from the engine scaled by hand rather than through uniform_real_distribution, whose output is not specified to be the same in every standard library: a crash that needs one particular junk frame is no use as a regression test if the frame is redrawn on each machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 13:03:06 +02:00
leonarski_fandClaude Opus 5	4b5c6e234d	indexing: the cap search always puts the direction grid back, and every FFT upload is checked ... A production broker on rc.170 segfaulted again in the FilterFFTResults length sort, in a binary that does contain the rc.170 non-finite guard (verified from its disassembly: the two isfinite tests are compiled in and skip the insert). With magnitude and length guaranteed finite, the only remaining way that sort sees a NaN key is a NaN direction vector - and the one place that writes the direction grid is SearchCap. SearchCap swaps in a temporary cap grid but restored it around ExecuteFFT only. The fill can run out of memory, the upload can fail, and the .at() that reads the peak back throws on exactly the corrupted result row this search has to survive - any of those left the cap grid installed on an indexer the pool hands the next image. If the axis was ever not a direction, axis.Normalize() is NaN and the leaked grid is 16384 NaN directions, which is that crash on every image afterwards. The restore now covers the whole body, and an axis that is not finite and non-zero is refused instead of normalized into NaN. The peak is also tested against the minimum length before the .at() rather than after, so an unusable peak with a corrupted index no longer throws at all. DirectionsChanged uploaded the grid with three unchecked cudaMemcpy calls, and the spot upload in ExecuteFFT had three more: a silently failed upload leaves the device holding the previous grid, so the directions the host reads the results against are not the ones the kernel used. Checked like the rest of the file. Tests: [Indexing] (9 cases, 115 assertions) and RotationIndexer pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 12:25:09 +02:00
leonarski_fandClaude Opus 5	20977e89bc	viewer: draw a live grid scan as a grid, and the compact axis names upright ... Two fixes in the dataset-info panel. The live (HTTP) reader took the goniometer axis and the grid scan as alternatives, so a raster collected at a head position - both fields set, the axis with step 0 - lost its grid and was drawn as a line plot, with the Grid button disabled. The broker had the same bug and lost it in rc.162 (OpenAPIConvert.cpp); the HDF5 read-back path already read both. The reader now sets both, like the other two. Qt Charts always draws a vertical axis title rotated, which reads awkwardly beside the two numbers it names in a half-height plot and is what forced that plot's minimum height (Qt elided "Spots" to "Sp..." once the font outgrew it). The title is switched off and the view paints the name itself: upright, one letter under the next, in the series' colour, centred on the plot area, with the left margin reserving its width. It is re-measured on every rebuild, so a font change still fixes itself, and plotAreaChanged re-centres it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 11:34:57 +02:00
leonarski_fandClaude Opus 5	4bff5c2b01	v1.0.0-rc.171 ... VERSION bumped and update_version.sh run: the OpenAPI spec and the three clients it generates, the frontend and docs version strings, the Redoc page, and the FPGA and PCIe-driver version strings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 08:41:29 +02:00
leonarski_fandClaude Opus 5	29a90416b7	docs: condense the rc.171 changelog ... Three lines at the level the rest of the file keeps: what a user gains, not how it works. The detail lives in RUGNUX_FORMATS.md and in the commits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 08:38:42 +02:00
leonarski_fandClaude Opus 5	39a98692e2	marCCD/SMV: keep the default frame time when the header's exposure is not a time ... One deposited marCCD sweep carries -2093438692 in its exposure field. Passed on as written that is a negative count time, DiffractionExperiment refuses it, and the whole dataset fails to open - over a number that feeds no geometry, no indexing and no merge. With the field ignored when it is implausible, that sweep processes and matches its deposition: P2_12_12_1, cell within 0.12%, and 1.375 A against a deposited 2.0. Neither format guarantees the field. The same guard is in the SMV reader, whose TIME key is no better attested, and both say what they did rather than passing over it silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 08:34:49 +02:00
leonarski_fandClaude Opus 5	8288d4cbe6	viewer: the stacked spots/background plots hold together at any font size ... The two half-height plots were matched once per data load, from JFJochViewerDatasetInfo: it read both y-label widths and pushed the larger back into both. A font change never goes through that path - each chart re-measures its own labels in its FontChange handler - so the padding was still the width measured in the old font, collapsed to zero, and the two curves started at different x at 125 % and 150 %. The pairing now lives in the chart view: each compact plot holds a peer pointer, and every rebuild re-matches the pair, so a font change fixes itself. Matched along with the label width: - the minimum height, taken from the width of the rotated y-axis title (Qt elided "Spots" to "Sp..." once the font outgrew the 70 px that was hardcoded for it), and given to both plots so the layout does not hand the wider-titled one more of the dock; - the plot area height: the upper plot now paints its x labels transparent instead of hiding them, so Qt reserves the same label band on both. The background plot also stops snapping its y axis to zero - it is read for its shape, not its level, and half a dock is not much to draw it in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 08:28:08 +02:00
leonarski_fandClaude Opus 5	7fecd11337	docs: the 51 second-round external datasets, and XRDa ... EXTERNAL_TEST_DATA.md gains a row per dataset in the existing form - deposited beamline, resolution, space group, cell and title from the RCSB API, source repository with its citable DOI, and the detector read out of the image files themselves. Every DOI was resolved before it was written down; one differs from the pattern the rest follow (5EPE is 10.18430/m3159c, not the m3<pdbid> form), which is the reason the page insists on resolving them rather than constructing them. Seven of the 51 name a detector in the PDB entry that the files contradict, now listed with the others. Two are not merely a model or size: one deposited as a "MAR CCD 165 mm" writes a 225 mm plate, and one deposited as a "RAYONIX MX300-HS" writes SMV with 315 mm geometry. marCCD and SMV headers name no model at all, so the Detector column carries what those headers do state - plate size, serial - and the page explains that. A new section gives the composition of the round: repository, file format as the files are on disk, facility and crystal system. It is metadata about the depositions, not measurement: no quantity measured by this software appears on the page, as the page's own preamble promises. The format spread is the point of the round, and is why Rugnux now reads marCCD, SMV and gzipped miniCBF natively. ACKNOWLEDGEMENT.md adds XRDa, the archive three of these came from, in the form the preceding documentation sweep settled on for a repository with no canonical citation paper: name, link and what it asks to be cited, and no substitute reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 08:20:42 +02:00
leonarski_fandClaude Fable 5	70c4d871b6	Documentation sweep: Rugnux naming, repository citations, defect pass ... - Capitalize Rugnux as a proper noun throughout the prose; the command stays lowercase `rugnux` in code font. RUGNUX_OVERVIEW.md is retitled "What Rugnux does". - ACKNOWLEDGEMENT.md cites the raw-data repositories only: dataset counts and DOI prefixes moved out (EXTERNAL_TEST_DATA.md owns them), the ESRF data portal gains its citation (Dimper et al. 2019), and MXRDR remains name + link - it has no canonical citation paper. - RUGNUX_FORMATS.md: the CCD formats (marCCD, SMV) are supported as-is with very limited scope, and per-panel XFEL data is not read. - Fix wrong facts a reader would act on: nonexistent `make jfjoch` targets, invalid udev rules, PUSH sockets documented as PULL, swapped writer width/height, underload semantics, the transposed pixel-mask numpy example (the server checks width and height separately), the Durin/Neggia mask-bit table, FPGA threshold register addresses and the mailbox bit field, the I2C core's document number (PG090), an inverted MODEL_FIT_SIGMA formula, a self-inconsistent worked report example, and 11 cross-page anchors whose slugs carry MyST section numbers. - Unify CC1/2 spelling in prose; math notation and report keys unchanged. - Sweep grammar, typos and editing residue across the FPGA, deployment, streaming and analysis pages, including historical CHANGELOG typos. - rugnux_cli.cpp: the -S usage/error examples pair 96 with P43212; 92 names a different group. - Root THIRD_PARTY_NOTICES.md: scope the GPL-compatibility claim (CUDA EULA) and the vendored-table intro (traccc); the docs copy regenerates via update_version.sh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-09-17 08:07:45 +02:00
leonarski_fandClaude Opus 5	28f222f51d	marCCD: take the distance from the start/end pair when the first field is zero ... Four of the twelve marCCD sets in the new corpus were refused with "states no detector distance". Their headers carry xtal_to_detector (offset 640) as zero and the real distance only in start_xtal_to_detector and end_xtal_to_detector - 70, 170 and 200 mm on three BESSY Rayonix MX-225 sweeps, against a set where all three fields agree at 300 mm. Reading the first field alone is what refused them. The guard itself was right: a distance of zero collapses every resolution and every scattering vector, so refusing beats processing silently. It just fired on data that does state a distance, in the field the format also defines for it. With the fallback, all three process de novo and match their depositions - P2_1 at cell 0.06%, I422 at 0.40%, P1 at 0.07% - and each reaches finer than its deposited resolution (0.889 A against 1.09, 1.440 against 1.69, 1.757 against 1.93). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 06:39:39 +02:00
leonarski_fandClaude Opus 5	2ab8c55dfa	rugnux and the viewer read SMV and gzipped miniCBF ... Two more of the formats deposited data actually arrives in, found by processing a corpus of it: every PETRA III EMBL set is .cbf.gz, and NSRRC and the whole ADSC Quantum era are SMV. Both were previously "no native input". SMV is an ASCII "KEY=value;" block between braces, then the pixels - no container, no compression, nothing to decode by offset - so reader/SMV.{h,cpp} and JFJochSMVReader are a smaller job than the marCCD pair they sit beside, and need no new dependency at all. Two things the format does not give us, both said out loud rather than papered over: * It states no saturation value, so overloads are judged on the 16-bit container alone. That can only fail to call a pixel saturated, never condemn a good one, but a CCD at the top of its range does saturate, so the reader warns once. * Its beam centre is in MILLIMETRES and which of X/Y is the fast direction is a convention rather than a rule. Measured on one ALS ADSC sweep the file's value is TRANSPOSED: as stated it indexes 2/60 frames, and the run's own beam-centre measurement (which adopts the right one automatically) indexes 60/60. Swapping it here would fit that writer and might break another, so the header is read as the format defines it and the measurement stays the arbiter. Revisit with a second vendor's SMV in hand. .cbf.gz needed only Slurp() in MiniCBF.cpp, through which every read already passes: it sniffs the two-byte gzip magic - not the file name - and takes a zlib path when it is there, leaving the plain path free of zlib's buffer copy. zlib-ng is already in the build, so this is a link line, not a dependency. The sweep template grew a suffix, because ".cbf" and ".cbf.gz" are separate sweeps and std::filesystem cannot split the double extension on its own. The viewer's single cbf_reader becomes three, dispatched by CanRead() in the same order as rugnux. Dispatch is by CONTENT in both: ".img" is used by miniCBF, marCCD AND SMV depending on the writer, and a PDB detector label has now been wrong about the format four times, so an extension decides nothing. Measured, de novo, no flags: 9fcg (1800 gzipped frames) gives P4 and a cell 0.06% from the deposited one at 1.37 A against a deposited 1.54; 6oel (ADSC SMV) gives F4132 - 96 operations, the most a protein space group can have - and a cell 0.05% out, 100% indexed. Tests cover both formats and the transposed-beam-centre case with fixtures written byte for byte, so they need no external data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-16 22:35:05 +02:00
leonarski_fandClaude Opus 5	a1889c45e9	rugnux reads marCCD sweeps natively ... A decade of deposited CCD data is archived as marCCD - what Rayonix MX-series and mar Mosaic detectors write - and rugnux could not open any of it. reader/ had two formats, NXmx/HDF5 and PILATUS miniCBF, and rugnux_cli dispatched on the one CanRead(); this adds the third. The format needs no new dependency: a marCCD file is an ordinary uncompressed TIFF whose 3072-byte instrument header sits in the gap between the TIFF header and the pixels, so libtiff - already fetched for JFJochPreview in every build mode - reads the image, and the header is a fixed-offset block of little-endian int32. Two things differ from the miniCBF path and are worth naming: * The pixel size is NOT rounded to whole micrometres. A PILATUS pixel is exactly 172 um so the existing reader can afford lround(); a MAR300 pixel is 73.242 um, and rounding it to 73 is a 0.33% scale error on every cell edge reported. * The sweep template is the last run of digits in the whole file name rather than in the stem, which covers both schemes these detectors use - a numbered stem (xtal_1_00042.mccd) and the frame number as the extension (D1.042). A CCD frame marks no untrusted pixels, so the sweep starts with nothing masked, and the format has nowhere to state the rotation axis' direction, so the run settles its sign from the data exactly as it does for a miniCBF carrying no axis table. Measured on one deposited 300-frame Rayonix MX-300 sweep, de novo with no flags: 100% indexing, the deposited point group, cell within 0.045%, 99.5% complete at multiplicity 3.4, in 26 s. The chosen sweep is confirmed against the instrument header before it is opened, so a directory of ordinary TIFFs is refused rather than read with a pixel size of zero - a unit test covers that, both naming schemes, and the geometry conversion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-16 19:31:58 +02:00