Files
Jungfraujoch/RUGNUX_FORMATS.html
T
2026-10-06 20:38:39 +00:00

1 line
23 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html> <html lang=en data-content_root="./"> <meta charset=utf-8 /> <meta name=viewport content="width=device-width, initial-scale=1.0" /><meta name=viewport content="width=device-width, initial-scale=1" /> <meta name=viewport content="width=device-width,initial-scale=1"> <meta http-equiv=x-ua-compatible content="ie=edge"> <meta name="lang:clipboard.copy" content="Copy to clipboard"> <meta name="lang:clipboard.copied" content="Copied to clipboard"> <meta name="lang:search.language" content=en > <meta name="lang:search.pipeline.stopwords" content=True > <meta name="lang:search.pipeline.trimmer" content=True > <meta name="lang:search.result.none" content="No matching documents"> <meta name="lang:search.result.one" content="1 matching document"> <meta name="lang:search.result.other" content="# matching documents"> <meta name="lang:search.tokenizer" content="[\s\-]+"> <link href="https://fonts.gstatic.com/" rel=preconnect crossorigin> <link href="https://fonts.googleapis.com/css?family=Roboto+Mono:400,500,700|Roboto:300,400,400i,700&display=fallback" rel=stylesheet > <style> body, input { font-family: "Roboto", "Helvetica Neue", Helvetica, Arial, sans-serif } code, kbd, pre { font-family: "Roboto Mono", "Courier New", Courier, monospace } </style> <link rel=stylesheet href="_static/stylesheets/application.css"/> <link rel=stylesheet href="_static/stylesheets/application-palette.css"/> <link rel=stylesheet href="_static/stylesheets/application-fixes.css"/> <link rel=stylesheet href="_static/fonts/material-icons.css"/> <meta name=theme-color content="#3f51b5"> <script src="_static/javascripts/modernizr.js"></script> <title>What Rugnux reads &#8212; Jungfraujoch 1.0.0-rc.174 documentation</title> <link rel=stylesheet type="text/css" href="_static/pygments.css?v=83e35b93" /> <link rel=stylesheet type="text/css" href="_static/material.css?v=79c92029" /> <script src="_static/documentation_options.js?v=3dccfb20"></script> <script src="_static/doctools.js?v=9bcbadda"></script> <script src="_static/sphinx_highlight.js?v=dc90522c"></script> <link rel=icon href="_static/jfjoch.png"/> <link rel=index title=Index href=genindex.html /> <link rel=search title=Search href=search.html /> <link rel=next title="Running Rugnux" href=RUGNUX_TUTORIAL.html /> <link rel=prev title="Installing Rugnux" href=RUGNUX_INSTALL.html /> <body dir=ltr data-md-color-primary=indigo data-md-color-accent=lime> <svg class=md-svg > <defs data-children-count=0 > <svg xmlns="http://www.w3.org/2000/svg" width=500 height=500 viewBox="0 0 500 500" id=__gitlab ><path fill=currentColor d="M93.667 473.347l90.684-279.097H2.983l90.684 279.097z" transform="translate(156.198 1.16)"/><path fill=currentColor d="M221.333 473.345L130.649 194.25H3.557l217.776 279.095z" transform="translate(28.531 1.16)" opacity=.7 /><path fill=currentColor d="M32 195.155L4.441 279.97a18.773 18.773 0 0 0 6.821 20.99l238.514 173.29L32 195.155z" transform="translate(.089 .256)" opacity=.5 /><path fill=currentColor d="M2.667-84.844h127.092L75.14-252.942c-2.811-8.649-15.047-8.649-17.856 0L2.667-84.844z" transform="translate(29.422 280.256)"/><path fill=currentColor d="M2.667 473.345L93.351 194.25h127.092L2.667 473.345z" transform="translate(247.198 1.16)" opacity=.7 /><path fill=currentColor d="M221.334 195.155l27.559 84.815a18.772 18.772 0 0 1-6.821 20.99L3.557 474.25l217.777-279.095z" transform="translate(246.307 .256)" opacity=.5 /><path fill=currentColor d="M130.667-84.844H3.575l54.618-168.098c2.811-8.649 15.047-8.649 17.856 0l54.618 168.098z" transform="translate(336.974 280.256)"/></svg> </defs> </svg> <input class=md-toggle data-md-toggle=drawer type=checkbox id=__drawer > <input class=md-toggle data-md-toggle=search type=checkbox id=__search > <label class=md-overlay data-md-component=overlay for=__drawer ></label> <a href="#RUGNUX_FORMATS" tabindex=1 class=md-skip > Skip to content </a> <header class=md-header data-md-component=header > <nav class="md-header-nav md-grid"> <div class="md-flex navheader"> <div class="md-flex__cell md-flex__cell--shrink"> <a href=index.html title="Jungfraujoch 1.0.0-rc.174 documentation" class="md-header-nav__button md-logo"> <i class=md-icon >&#xe30d</i> </a> </div> <div class="md-flex__cell md-flex__cell--shrink"> <label class="md-icon md-icon--menu md-header-nav__button" for=__drawer ></label> </div> <div class="md-flex__cell md-flex__cell--stretch"> <div class="md-flex__ellipsis md-header-nav__title" data-md-component=title > <span class=md-header-nav__topic >PSI Jungfraujoch</span> <span class=md-header-nav__topic > What Rugnux reads </span> </div> </div> <div class="md-flex__cell md-flex__cell--shrink"> <label class="md-icon md-icon--search md-header-nav__button" for=__search ></label> <div class=md-search data-md-component=search role=dialog > <label class=md-search__overlay for=__search ></label> <div class=md-search__inner role=search > <form class=md-search__form action=search.html method=get name=search > <input type=text class=md-search__input name=q placeholder=""Search"" autocapitalize=off autocomplete=off spellcheck=false data-md-component=query data-md-state=active > <label class="md-icon md-search__icon" for=__search ></label> <button type=reset class="md-icon md-search__icon" data-md-component=reset tabindex=-1 > &#xE5CD; </button> </form> <div class=md-search__output > <div class=md-search__scrollwrap data-md-scrollfix> <div class=md-search-result data-md-component=result > <div class=md-search-result__meta > Type to start searching </div> <ol class=md-search-result__list ></ol> </div> </div> </div> </div> </div> </div> <div class="md-flex__cell md-flex__cell--shrink"> <div class=md-header-nav__source > <a href="https://gitea.psi.ch/mx/jungfraujoch" title="Go to repository" class=md-source data-md-source=github > <div class=md-source__icon > <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 24 24" width=28 height=28 > <use xlink:href="#__gitlab" width=24 height=24 ></use> </svg> </div> <div class=md-source__repository > Jungfraujoch </div> </a> </div> </div> <script src="_static/javascripts/version_dropdown.js"></script> <script> var json_loc = ""versions.json"", target_loc = "../", text = "Versions"; $( document ).ready( add_version_dropdown(json_loc, target_loc, text)); </script> </div> </nav> </header> <div class=md-container > <nav class=md-tabs data-md-component=tabs > <div class="md-tabs__inner md-grid"> <ul class=md-tabs__list > <li class=md-tabs__item ><a href=index.html class=md-tabs__link >Jungfraujoch 1.0.0-rc.174 documentation</a> </ul> </div> </nav> <main class=md-main > <div class="md-main__inner md-grid" data-md-component=container > <div class="md-sidebar md-sidebar--primary" data-md-component=navigation > <div class=md-sidebar__scrollwrap > <div class=md-sidebar__inner > <nav class="md-nav md-nav--primary" data-md-level=0 > <label class="md-nav__title md-nav__title--site" for=__drawer > <a href=index.html title="Jungfraujoch 1.0.0-rc.174 documentation" class="md-nav__button md-logo"> <i class=md-icon >&#xe30d</i> </a> <a href=index.html title="Jungfraujoch 1.0.0-rc.174 documentation">PSI Jungfraujoch</a> </label> <div class=md-nav__source > <a href="https://gitea.psi.ch/mx/jungfraujoch" title="Go to repository" class=md-source data-md-source=github > <div class=md-source__icon > <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 24 24" width=28 height=28 > <use xlink:href="#__gitlab" width=24 height=24 ></use> </svg> </div> <div class=md-source__repository > Jungfraujoch </div> </a> </div> <ul class=md-nav__list > <li class=md-nav__item > <span class="md-nav__link caption"><span class=caption-text >rugnux — data processing</span></span> <li class=md-nav__item > <a href=RUGNUX.html class=md-nav__link >Rugnux</a> <li class=md-nav__item > <a href=RUGNUX_OVERVIEW.html class=md-nav__link >What Rugnux does</a> <li class=md-nav__item > <a href=RUGNUX_INSTALL.html class=md-nav__link >Installing Rugnux</a> <li class=md-nav__item > <input class="md-toggle md-nav__toggle" data-md-toggle=toc type=checkbox id=__toc > <label class="md-nav__link md-nav__link--active" for=__toc > What Rugnux reads </label> <a href="#" class="md-nav__link md-nav__link--active">What Rugnux reads</a> <nav class="md-nav md-nav--secondary"> <ul class=md-nav__list data-md-scrollfix=""> </ul> </nav> <li class=md-nav__item > <a href=RUGNUX_TUTORIAL.html class=md-nav__link >Running Rugnux</a> <li class=md-nav__item > <a href=RUGNUX_INTEGRATION.html class=md-nav__link >Rugnux with other programs</a> <li class=md-nav__item > <a href=RUGNUX_REPORT.html class=md-nav__link >The results report</a> <li class=md-nav__item > <a href=RUGNUX_ADVANCED.html class=md-nav__link >Advanced Rugnux</a> <li class=md-nav__item > <a href=RUGNUX_CALIBRATION.html class=md-nav__link >Detector calibration from powder rings (<code class="docutils literal notranslate"><span class=pre >rugnux</span> <span class=pre >--mode</span> <span class=pre >calibration</span></code>)</a> <li class=md-nav__item > <a href=CPU_DATA_ANALYSIS.html class=md-nav__link >CPU-side crystallographic data analysis (Jungfraujoch)</a> <li class=md-nav__item > <a href=CPU_DATA_ANALYSIS_IMAGE.html class=md-nav__link >Data analysis: from images to spots (§0–§3)</a> <li class=md-nav__item > <a href=CPU_DATA_ANALYSIS_INDEXING.html class=md-nav__link >Data analysis: indexing and geometry (§4–§7)</a> <li class=md-nav__item > <a href=CPU_DATA_ANALYSIS_INTEGRATION.html class=md-nav__link >Data analysis: integration, scaling and merging (§8–§12)</a> <li class=md-nav__item > <a href=CPU_DATA_ANALYSIS_DECISIONS.html class=md-nav__link >Data analysis: space group and validation (§13–§14)</a> <li class=md-nav__item > <span class="md-nav__link caption"><span class=caption-text >Jungfraujoch — acquisition</span></span> <li class=md-nav__item > <a href=JFJOCH_BROKER.html class=md-nav__link >jfjoch_broker</a> <li class=md-nav__item > <a href=JFJOCH_WRITER.html class=md-nav__link >jfjoch_writer</a> <li class=md-nav__item > <a href=JFJOCH_VIEWER.html class=md-nav__link >jfjoch_viewer</a> <li class=md-nav__item > <a href=SOFTWARE_INTEGRATION.html class=md-nav__link >Integration with MX data processing software</a> <li class=md-nav__item > <a href=TOOLS.html class=md-nav__link >Tools</a> <li class=md-nav__item > <a href=DEPLOYMENT.html class=md-nav__link >Deployment</a> <li class=md-nav__item > <a href=DETECTORS.html class=md-nav__link >Supported detectors</a> <li class=md-nav__item > <a href=HARDWARE.html class=md-nav__link >Hardware requirements</a> <li class=md-nav__item > <a href=SOFTWARE.html class=md-nav__link >Software requirements</a> <li class=md-nav__item > <span class="md-nav__link caption"><span class=caption-text >FPGA</span></span> <li class=md-nav__item > <a href=FPGA.html class=md-nav__link >FPGA smartNIC</a> <li class=md-nav__item > <a href=FPGA_LICENSE.html class=md-nav__link >FPGA license</a> <li class=md-nav__item > <a href=FPGA_DESIGN.html class=md-nav__link >FPGA data flow</a> <li class=md-nav__item > <a href=FPGA_NETWORK.html class=md-nav__link >FPGA network</a> <li class=md-nav__item > <a href=FPGA_PCIE_DRIVER.html class=md-nav__link >FPGA PCIe driver</a> <li class=md-nav__item > <a href=FPGA_SETTINGS.html class=md-nav__link >FPGA advanced reference</a> <li class=md-nav__item > <a href=FPGA_DATA_ANALYSIS.html class=md-nav__link >FPGA data analysis</a> <li class=md-nav__item > <span class="md-nav__link caption"><span class=caption-text >Reference</span></span> <li class=md-nav__item > <a href=DETECTOR_GEOMETRY.html class=md-nav__link >Detector geometry</a> <li class=md-nav__item > <a href=OPENAPI.html class=md-nav__link >OpenAPI</a> <li class=md-nav__item > <a href=OPENAPI_SPECS.html class=md-nav__link >OpenAPI specification</a> <li class=md-nav__item > <a href=PYTHON_CLIENT.html class=md-nav__link >OpenAPI Python client</a> <li class=md-nav__item > <a href=CBOR.html class=md-nav__link >CBOR messages</a> <li class=md-nav__item > <a href=HDF5.html class=md-nav__link >HDF5 / NeXus data format</a> <li class=md-nav__item > <a href=IMAGE_STREAM.html class=md-nav__link >Data streams</a> <li class=md-nav__item > <a href=PIXEL_MASK.html class=md-nav__link >Pixel mask</a> <li class=md-nav__item > <a href=WEB_FRONTEND.html class=md-nav__link >Web frontend</a> <li class=md-nav__item > <a href=TESTS.html class=md-nav__link >Tests</a> <li class=md-nav__item > <span class="md-nav__link caption"><span class=caption-text >Project</span></span> <li class=md-nav__item > <a href=ACKNOWLEDGEMENT.html class=md-nav__link >Acknowledgements</a> <li class=md-nav__item > <a href=EXTERNAL_TEST_DATA.html class=md-nav__link >External test data</a> <li class=md-nav__item > <a href=LICENSE.html class=md-nav__link >License</a> <li class=md-nav__item > <a href=THIRD_PARTY_NOTICES.html class=md-nav__link >Third-party software notices</a> <li class=md-nav__item > <a href=VERSIONING.html class=md-nav__link >Semantic versioning</a> <li class=md-nav__item > <a href=SECURITY.html class=md-nav__link >Security</a> <li class=md-nav__item > <a href=RELEASE_CONTENTS.html class=md-nav__link >Release contents</a> <li class=md-nav__item > <a href=REPOSITORIES.html class=md-nav__link >Linux package repositories</a> <li class=md-nav__item > <a href=NAMING.html class=md-nav__link >Naming</a> <li class=md-nav__item > <a href=CHANGELOG.html class=md-nav__link >Changelog</a> </ul> </nav> </div> </div> </div> <div class="md-sidebar md-sidebar--secondary" data-md-component=toc > <div class=md-sidebar__scrollwrap > <div class=md-sidebar__inner > <nav class="md-nav md-nav--secondary"> <ul class=md-nav__list data-md-scrollfix=""> </ul> </nav> </div> </div> </div> <div class=md-content > <article class="md-content__inner md-typeset" role=main > <section class="tex2jax_ignore mathjax_ignore" id=what-rugnux-reads > <h1 id=rugnux-formats--page-root >What Rugnux reads<a class=headerlink href="#rugnux-formats--page-root" title="Link to this heading"></a></h1> <p>Which data <code class="docutils literal notranslate"><span class=pre >rugnux</span></code> opens, before anything is typed. The short answer: an HDF5 master (NXmx or DECTRIS, from any facility), a PILATUS miniCBF sweep, a marCCD sweep or an SMV sweep (ADSC or Rigaku d*TREK) — nothing else is read, so any other format has to be converted to one of these first.</p> <p><strong>Input</strong> is an HDF5 master file, or a directory of PILATUS miniCBF, marCCD or SMV frames. One input is <strong>one sweep of one crystal</strong> — Rugnux does not combine sweeps or crystals in a run; process each sweep to its own <code class="docutils literal notranslate"><span class=pre >_unmerged.mtz</span></code> and merge them downstream (see <a class="reference internal" href="RUGNUX_INTEGRATION.html#taking-the-data-onward"><span class="std std-ref">Taking the data onward</span></a>).</p> <ul class=simple > <li><p><strong>HDF5 master (NXmx-based)</strong> — a file written by <a class="reference internal" href=JFJOCH_WRITER.html ><span class="std std-doc"><code class="docutils literal notranslate"><span class=pre >jfjoch_writer</span></code></span></a>, a DECTRIS EIGER master, or an NXmx master written by another facility’s toolchain. Lengths are taken in the unit the file declares, the image size from the image array’s own shape, and each data file’s images at the path the master’s own link names, so masters that state single values as one-element arrays, compose their images as a virtual dataset over the master itself, or keep them somewhere other than <code class="docutils literal notranslate"><span class=pre >/entry/data/data</span></code> all open. Where the NXmx spellings are absent the pre-NXmx ones are tried, so a DECTRIS firmware 1.x master opens too — including its goniometer, which used to be missed and the sweep read as stills. Images may be <code class="docutils literal notranslate"><span class=pre >bitshuffle</span></code>+LZ4/Zstd or the HDF Group’s plain LZ4 (filter 32004); any other filter is named in the error.</p> <li><p><strong>PILATUS miniCBF sweep</strong> — one frame per file, read natively with no conversion and no libcbf. Name <strong>any frame of the sweep</strong>, or the directory holding it, and the whole sweep is processed: the frames are the ones matching that frame’s template (prefix plus digit count), so a directory holding two sweeps is not spliced into one crystal, and naming a directory takes the sweep with the most frames in it. The geometry, the rotation axis and the detector mounting come from the header, including the imgCIF axis table where the header carries one (see <a class="reference internal" href=DETECTOR_GEOMETRY.html ><span class="std std-doc">Detector geometry</span></a>). A raw CBF carries no analysis results, so <code class="docutils literal notranslate"><span class=pre >--mode</span> <span class=pre >scale</span></code> — which re-scales the reflections stored in a <code class="docutils literal notranslate"><span class=pre >_process.h5</span></code> — does not accept one. <strong>Gzipped frames (<code class="docutils literal notranslate"><span class=pre >.cbf.gz</span></code>) are read directly</strong>, with no decompression step: EMBL Hamburg’s beamlines write that form by default, and expanding a sweep first would double the disk it needs. The two forms count as separate sweeps.</p> <li><p><strong>marCCD sweep</strong> — what Rayonix MX-series and mar Mosaic detectors write, and what a decade of deposited CCD data is archived as: an uncompressed TIFF with the instrument header in the gap before the pixels. One frame per file, read natively. Naming a frame or its directory selects the sweep exactly as for miniCBF, under either naming scheme these detectors use — a numbered stem (<code class="docutils literal notranslate"><span class=pre >xtal_1_00042.mccd</span></code>) or the frame number as the file extension (<code class="docutils literal notranslate"><span class=pre >D1.042</span></code>). The distance, beam centre, pixel size, wavelength and the circle that turned come from the header; the rotation axis’ direction does not, because the format has nowhere to state it, so the run settles its sign from the data as it does for a miniCBF that carries no axis table. A CCD frame marks no untrusted pixels, so the sweep starts with nothing masked. Like a raw CBF, it carries no analysis results and <code class="docutils literal notranslate"><span class=pre >--mode</span> <span class=pre >scale</span></code> does not accept one.</p> <li><p><strong>SMV sweep</strong> — what ADSC Quantum detectors wrote and what Rayonix and others still write, so most archived CCD data from the 2000s is in this form: an ASCII <code class="docutils literal notranslate"><span class=pre >KEY=value;</span></code> block between braces, then the pixels. One frame per file, read natively, the sweep selected exactly as above. The beam centre it states is in <strong>millimetres</strong> and is converted here; its convention varies between writers, and a file that states it transposed indexes nothing until the run’s own beam-centre measurement replaces it, which it does automatically. The saturation value is read where the header states one (<code class="docutils literal notranslate"><span class=pre >CCD_IMAGE_SATURATION</span></code> or <code class="docutils literal notranslate"><span class=pre >SATURATED_VALUE</span></code>); where it does not, overloads are judged on the 16-bit container alone and the reader says so once per sweep. <strong>Rigaku d*TREK headers</strong> — what Saturn CCDs on rotating-anode home sources write, recognised by their <code class="docutils literal notranslate"><span class=pre >DETECTOR_NAMES</span></code> key — are read in their own vocabulary: the pixel size and the point of normal incidence from the spatial-distortion record, the image directions from the detector and distortion vectors (so a mirrored or turned image keeps its hand), the distance and the detector circles, including a swung <strong>2θ arm</strong>, from the detector goniometer, the spindle and its angles from the rotation record, and the wavelength from the scan record. Pixel values above 32767 are the format’s encoded overflows and are expanded with the header’s compression ratio.</p> </ul> <p>Support for the two CCD formats — marCCD and SMV — is <strong>very limited and provided as-is</strong>; they are read so that a CCD sweep does not need a conversion step to be processed. Only the header fields named above are read, and the headers vary by site and vintage, so no promise is made that every writer’s dialect opens. The HDF5 and miniCBF paths are the maintained ones.</p> <p><strong>XFEL data saved as individual panels is not read.</strong> The detectors of serial XFEL endstations — CSPAD, AGIPD, ePix and their kin — write each panel as its own array, with the panel positions kept in a separate geometry file rather than in the image file. Rugnux has no reader for those per-panel containers and no way to take an external geometry, so such data cannot be opened; it belongs in the pipelines built around panel geometries, such as CrystFEL or cctbx.xfel. An XFEL dataset that has already been assembled into single images in an NXmx master opens like any other HDF5 input.</p> <p>Spots are always found by <code class="docutils literal notranslate"><span class=pre >rugnux</span></code> itself, including for the two-pass rotation first pass — the spot lists a dataset may already carry were found online, at the acquisition’s threshold and with its ice-band spots already discarded, so reusing them would hide the spot-finding settings from the lattice search.</p> </section> </article> </div> </div> </main> </div> <footer class=md-footer > <div class=md-footer-nav > <nav class="md-footer-nav__inner md-grid"> <a href=RUGNUX_INSTALL.html title="Installing Rugnux" class="md-flex md-footer-nav__link md-footer-nav__link--prev" rel=prev > <div class="md-flex__cell md-flex__cell--shrink"> <i class="md-icon md-icon--arrow-back md-footer-nav__button"></i> </div> <div class="md-flex__cell md-flex__cell--stretch md-footer-nav__title"> <span class=md-flex__ellipsis > <span class=md-footer-nav__direction > "Previous" </span> Installing Rugnux </span> </div> </a> <a href=RUGNUX_TUTORIAL.html title="Running Rugnux" class="md-flex md-footer-nav__link md-footer-nav__link--next" rel=next > <div class="md-flex__cell md-flex__cell--stretch md-footer-nav__title"><span class=md-flex__ellipsis > <span class=md-footer-nav__direction > "Next" </span> Running Rugnux </span> </div> <div class="md-flex__cell md-flex__cell--shrink"><i class="md-icon md-icon--arrow-forward md-footer-nav__button"></i> </div> </a> </nav> </div> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class=md-footer-copyright > <div class=md-footer-copyright__highlight > &#169; Copyright 2024, Paul Scherrer Institute. </div> Created using <a href="http://www.sphinx-doc.org/">Sphinx</a> 8.1.3. and <a href="https://github.com/bashtage/sphinx-material/">Material for Sphinx</a> </div> </div> </div> </footer> <script src="_static/javascripts/application.js"></script> <script>app.initialize({version: "1.0.4", url: {base: ".."}})</script>