1 line
267 KiB
HTML
1 line
267 KiB
HTML
<!DOCTYPE html> <html lang=en data-content_root="./"> <meta charset=utf-8 /> <meta name=viewport content="width=device-width, initial-scale=1.0" /><meta name=viewport content="width=device-width, initial-scale=1" /> <meta name=viewport content="width=device-width,initial-scale=1"> <meta http-equiv=x-ua-compatible content="ie=edge"> <meta name="lang:clipboard.copy" content="Copy to clipboard"> <meta name="lang:clipboard.copied" content="Copied to clipboard"> <meta name="lang:search.language" content=en > <meta name="lang:search.pipeline.stopwords" content=True > <meta name="lang:search.pipeline.trimmer" content=True > <meta name="lang:search.result.none" content="No matching documents"> <meta name="lang:search.result.one" content="1 matching document"> <meta name="lang:search.result.other" content="# matching documents"> <meta name="lang:search.tokenizer" content="[\s\-]+"> <link href="https://fonts.gstatic.com/" rel=preconnect crossorigin> <link href="https://fonts.googleapis.com/css?family=Roboto+Mono:400,500,700|Roboto:300,400,400i,700&display=fallback" rel=stylesheet > <style> body, input { font-family: "Roboto", "Helvetica Neue", Helvetica, Arial, sans-serif } code, kbd, pre { font-family: "Roboto Mono", "Courier New", Courier, monospace } </style> <link rel=stylesheet href="_static/stylesheets/application.css"/> <link rel=stylesheet href="_static/stylesheets/application-palette.css"/> <link rel=stylesheet href="_static/stylesheets/application-fixes.css"/> <link rel=stylesheet href="_static/fonts/material-icons.css"/> <meta name=theme-color content="#3f51b5"> <script src="_static/javascripts/modernizr.js"></script> <title>CPU-side crystallographic data analysis (Jungfraujoch) — Jungfraujoch 1.0.0-rc.165 documentation</title> <link rel=stylesheet type="text/css" href="_static/pygments.css?v=83e35b93" /> <link rel=stylesheet type="text/css" href="_static/material.css?v=79c92029" /> <script src="_static/documentation_options.js?v=fd7baf7d"></script> <script src="_static/doctools.js?v=9bcbadda"></script> <script src="_static/sphinx_highlight.js?v=dc90522c"></script> <script>window.MathJax = {"tex": {"tags": "ams", "inlineMath": [["\\(", "\\)"], ["$", "$"]], "displayMath": [["\\[", "\\]"], ["$$", "$$"]], "processEscapes": true}, "options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script> <script defer=defer src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script> <link rel=icon href="_static/jfjoch.png"/> <link rel=index title=Index href=genindex.html /> <link rel=search title=Search href=search.html /> <link rel=next title=OpenAPI href=OPENAPI.html /> <link rel=prev title="Detector geometry" href=DETECTOR_GEOMETRY.html /> <body dir=ltr data-md-color-primary=indigo data-md-color-accent=lime> <svg class=md-svg > <defs data-children-count=0 > <svg xmlns="http://www.w3.org/2000/svg" width=500 height=500 viewBox="0 0 500 500" id=__gitlab ><path fill=currentColor d="M93.667 473.347l90.684-279.097H2.983l90.684 279.097z" transform="translate(156.198 1.16)"/><path fill=currentColor d="M221.333 473.345L130.649 194.25H3.557l217.776 279.095z" transform="translate(28.531 1.16)" opacity=.7 /><path fill=currentColor d="M32 195.155L4.441 279.97a18.773 18.773 0 0 0 6.821 20.99l238.514 173.29L32 195.155z" transform="translate(.089 .256)" opacity=.5 /><path fill=currentColor d="M2.667-84.844h127.092L75.14-252.942c-2.811-8.649-15.047-8.649-17.856 0L2.667-84.844z" transform="translate(29.422 280.256)"/><path fill=currentColor d="M2.667 473.345L93.351 194.25h127.092L2.667 473.345z" transform="translate(247.198 1.16)" opacity=.7 /><path fill=currentColor d="M221.334 195.155l27.559 84.815a18.772 18.772 0 0 1-6.821 20.99L3.557 474.25l217.777-279.095z" transform="translate(246.307 .256)" opacity=.5 /><path fill=currentColor d="M130.667-84.844H3.575l54.618-168.098c2.811-8.649 15.047-8.649 17.856 0l54.618 168.098z" transform="translate(336.974 280.256)"/></svg> </defs> </svg> <input class=md-toggle data-md-toggle=drawer type=checkbox id=__drawer > <input class=md-toggle data-md-toggle=search type=checkbox id=__search > <label class=md-overlay data-md-component=overlay for=__drawer ></label> <a href="#CPU_DATA_ANALYSIS" tabindex=1 class=md-skip > Skip to content </a> <header class=md-header data-md-component=header > <nav class="md-header-nav md-grid"> <div class="md-flex navheader"> <div class="md-flex__cell md-flex__cell--shrink"> <a href=index.html title="Jungfraujoch 1.0.0-rc.165 documentation" class="md-header-nav__button md-logo"> <i class=md-icon ></i> </a> </div> <div class="md-flex__cell md-flex__cell--shrink"> <label class="md-icon md-icon--menu md-header-nav__button" for=__drawer ></label> </div> <div class="md-flex__cell md-flex__cell--stretch"> <div class="md-flex__ellipsis md-header-nav__title" data-md-component=title > <span class=md-header-nav__topic >PSI Jungfraujoch</span> <span class=md-header-nav__topic > CPU-side crystallographic data analysis (Jungfraujoch) </span> </div> </div> <div class="md-flex__cell md-flex__cell--shrink"> <label class="md-icon md-icon--search md-header-nav__button" for=__search ></label> <div class=md-search data-md-component=search role=dialog > <label class=md-search__overlay for=__search ></label> <div class=md-search__inner role=search > <form class=md-search__form action=search.html method=get name=search > <input type=text class=md-search__input name=q placeholder=""Search"" autocapitalize=off autocomplete=off spellcheck=false data-md-component=query data-md-state=active > <label class="md-icon md-search__icon" for=__search ></label> <button type=reset class="md-icon md-search__icon" data-md-component=reset tabindex=-1 >  </button> </form> <div class=md-search__output > <div class=md-search__scrollwrap data-md-scrollfix> <div class=md-search-result data-md-component=result > <div class=md-search-result__meta > Type to start searching </div> <ol class=md-search-result__list ></ol> </div> </div> </div> </div> </div> </div> <div class="md-flex__cell md-flex__cell--shrink"> <div class=md-header-nav__source > <a href="https://gitea.psi.ch/mx/jungfraujoch" title="Go to repository" class=md-source data-md-source=github > <div class=md-source__icon > <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 24 24" width=28 height=28 > <use xlink:href="#__gitlab" width=24 height=24 ></use> </svg> </div> <div class=md-source__repository > Jungfraujoch </div> </a> </div> </div> <script src="_static/javascripts/version_dropdown.js"></script> <script> var json_loc = ""versions.json"", target_loc = "../", text = "Versions"; $( document ).ready( add_version_dropdown(json_loc, target_loc, text)); </script> </div> </nav> </header> <div class=md-container > <nav class=md-tabs data-md-component=tabs > <div class="md-tabs__inner md-grid"> <ul class=md-tabs__list > <li class=md-tabs__item ><a href=index.html class=md-tabs__link >Jungfraujoch 1.0.0-rc.165 documentation</a> </ul> </div> </nav> <main class=md-main > <div class="md-main__inner md-grid" data-md-component=container > <div class="md-sidebar md-sidebar--primary" data-md-component=navigation > <div class=md-sidebar__scrollwrap > <div class=md-sidebar__inner > <nav class="md-nav md-nav--primary" data-md-level=0 > <label class="md-nav__title md-nav__title--site" for=__drawer > <a href=index.html title="Jungfraujoch 1.0.0-rc.165 documentation" class="md-nav__button md-logo"> <i class=md-icon ></i> </a> <a href=index.html title="Jungfraujoch 1.0.0-rc.165 documentation">PSI Jungfraujoch</a> </label> <div class=md-nav__source > <a href="https://gitea.psi.ch/mx/jungfraujoch" title="Go to repository" class=md-source data-md-source=github > <div class=md-source__icon > <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 24 24" width=28 height=28 > <use xlink:href="#__gitlab" width=24 height=24 ></use> </svg> </div> <div class=md-source__repository > Jungfraujoch </div> </a> </div> <ul class=md-nav__list > <li class=md-nav__item > <span class="md-nav__link caption"><span class=caption-text >General</span></span> <li class=md-nav__item > <a href=ACKNOWLEDGEMENT.html class=md-nav__link >Acknowledgements</a> <li class=md-nav__item > <a href=LICENSE.html class=md-nav__link >License</a> <li class=md-nav__item > <a href=THIRD_PARTY_NOTICES.html class=md-nav__link >Third-party software notices</a> <li class=md-nav__item > <a href=DETECTORS.html class=md-nav__link >Supported detectors</a> <li class=md-nav__item > <a href="DETECTORS.html#dectris-detectors" class=md-nav__link >DECTRIS detectors</a> <li class=md-nav__item > <a href=HARDWARE.html class=md-nav__link >Hardware requirements</a> <li class=md-nav__item > <a href=SOFTWARE.html class=md-nav__link >Software requirements</a> <li class=md-nav__item > <a href=VERSIONING.html class=md-nav__link >Semantic versioning</a> <li class=md-nav__item > <a href=DEPLOYMENT.html class=md-nav__link >Deployment</a> <li class=md-nav__item > <a href=SECURITY.html class=md-nav__link >Security</a> <li class=md-nav__item > <a href=RELEASE_CONTENTS.html class=md-nav__link >Release contents</a> <li class=md-nav__item > <a href=REPOSITORIES.html class=md-nav__link >Linux package repositories</a> <li class=md-nav__item > <a href=NAMING.html class=md-nav__link >Naming</a> <li class=md-nav__item > <a href=CHANGELOG.html class=md-nav__link >Changelog</a> <li class=md-nav__item > <span class="md-nav__link caption"><span class=caption-text >Software</span></span> <li class=md-nav__item > <a href=JFJOCH_BROKER.html class=md-nav__link >jfjoch_broker</a> <li class=md-nav__item > <a href=JFJOCH_WRITER.html class=md-nav__link >jfjoch_writer</a> <li class=md-nav__item > <a href=RUGNUX.html class=md-nav__link >rugnux</a> <li class=md-nav__item > <a href=JFJOCH_VIEWER.html class=md-nav__link >jfjoch_viewer</a> <li class=md-nav__item > <a href=SOFTWARE_INTEGRATION.html class=md-nav__link >Integration with MX data processing software</a> <li class=md-nav__item > <a href=TOOLS.html class=md-nav__link >Tools</a> <li class=md-nav__item > <span class="md-nav__link caption"><span class=caption-text >FPGA</span></span> <li class=md-nav__item > <a href=FPGA.html class=md-nav__link >FPGA smartNIC</a> <li class=md-nav__item > <a href=FPGA_LICENSE.html class=md-nav__link >FPGA license</a> <li class=md-nav__item > <a href=FPGA_DESIGN.html class=md-nav__link >FPGA data flow</a> <li class=md-nav__item > <a href=FPGA_NETWORK.html class=md-nav__link >FPGA network</a> <li class=md-nav__item > <a href=FPGA_PCIE_DRIVER.html class=md-nav__link >FPGA PCIe driver</a> <li class=md-nav__item > <a href=FPGA_SETTINGS.html class=md-nav__link >FPGA advanced reference</a> <li class=md-nav__item > <a href=FPGA_DATA_ANALYSIS.html class=md-nav__link >FPGA data analysis</a> <li class=md-nav__item > <span class="md-nav__link caption"><span class=caption-text >Reference</span></span> <li class=md-nav__item > <a href=DETECTOR_GEOMETRY.html class=md-nav__link >Detector geometry</a> <li class=md-nav__item > <input class="md-toggle md-nav__toggle" data-md-toggle=toc type=checkbox id=__toc > <label class="md-nav__link md-nav__link--active" for=__toc > CPU-side crystallographic data analysis (Jungfraujoch) </label> <a href="#" class="md-nav__link md-nav__link--active">CPU-side crystallographic data analysis (Jungfraujoch)</a> <nav class="md-nav md-nav--secondary"> <ul class=md-nav__list data-md-scrollfix=""> </ul> </nav> <ul class=md-nav__list > <li class=md-nav__item > <a href="#references" class=md-nav__link >References</a> <li class=md-nav__item > <a href="#getting-the-image-onto-the-gpu-device-side-bitshuffle-lz4-decoding" class=md-nav__link >0. Getting the image onto the GPU: device-side bitshuffle+LZ4 decoding</a> <li class=md-nav__item > <a href="#geometry-reciprocal-space-mapping-and-basic-quantities" class=md-nav__link >1. Geometry, reciprocal-space mapping, and basic quantities</a> <li class=md-nav__item > <a href="#azimuthal-integration-radial-profiles" class=md-nav__link >2. Azimuthal integration (radial profiles)</a> <li class=md-nav__item > <a href="#spot-finding-strong-pixels-bragg-spots" class=md-nav__link >3. Spot finding (strong pixels → Bragg spots)</a> <li class=md-nav__item > <a href="#indexing-overview" class=md-nav__link >4. Indexing overview</a> <li class=md-nav__item > <a href="#fft-indexing-unknown-unit-cell" class=md-nav__link >5. FFT indexing (unknown unit cell)</a> <li class=md-nav__item > <a href="#bravais-lattice-centering-inference-lattice-search" class=md-nav__link >6. Bravais lattice / centering inference (“lattice search”)</a> <li class=md-nav__item > <a href="#geometry-and-lattice-refinement" class=md-nav__link >7. Geometry and lattice refinement</a> <li class=md-nav__item > <a href="#reflection-prediction" class=md-nav__link >8. Reflection prediction</a> <li class=md-nav__item > <a href="#d-bragg-integration-profile-fitting-over-a-three-ring-roi" class=md-nav__link >9. 2D Bragg integration (profile fitting over a three-ring ROI)</a> <li class=md-nav__item > <a href="#scaling-and-merging" class=md-nav__link >10. Scaling and merging</a> <li class=md-nav__item > <a href="#mosaicity-and-profile-radius-monitoring" class=md-nav__link >11. Mosaicity and “profile radius” monitoring</a> <li class=md-nav__item > <a href="#auxiliary-statistics-i-i-and-wilson-plot" class=md-nav__link >12. Auxiliary statistics: ⟨I/σ(I)⟩ and Wilson plot</a> <li class=md-nav__item > <a href="#space-group-determination-and-merge-level-decisions" class=md-nav__link >13. Space-group determination and merge-level decisions</a> <li class=md-nav__item > <a href="#model-based-validation-r-free-against-a-model-and-electron-density-maps" class=md-nav__link >14. Model-based validation: R-free against a model and electron-density maps</a> </ul> <li class=md-nav__item > <a href=OPENAPI.html class=md-nav__link >OpenAPI</a> <li class=md-nav__item > <a href=OPENAPI_SPECS.html class=md-nav__link >OpenAPI specification</a> <li class=md-nav__item > <a href=CBOR.html class=md-nav__link >CBOR messages</a> <li class=md-nav__item > <a href=HDF5.html class=md-nav__link >HDF5 / NeXus data format</a> <li class=md-nav__item > <a href=IMAGE_STREAM.html class=md-nav__link >Data streams</a> <li class=md-nav__item > <a href=PIXEL_MASK.html class=md-nav__link >Pixel mask</a> <li class=md-nav__item > <a href=WEB_FRONTEND.html class=md-nav__link >Web frontend</a> <li class=md-nav__item > <a href=TESTS.html class=md-nav__link >Tests</a> <li class=md-nav__item > <span class="md-nav__link caption"><span class=caption-text >OpenAPI Python client</span></span> <li class=md-nav__item > <a href="python_client/README.html" class=md-nav__link >jfjoch-client</a> <li class=md-nav__item > <a href="python_client/README.html#license-clarification" class=md-nav__link >License Clarification</a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html" class=md-nav__link >jfjoch_client.DefaultApi</a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#cancel-post" class=md-nav__link ><strong>cancel_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-azim-int-get" class=md-nav__link ><strong>config_azim_int_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-azim-int-put" class=md-nav__link ><strong>config_azim_int_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-bragg-integration-get" class=md-nav__link ><strong>config_bragg_integration_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-bragg-integration-put" class=md-nav__link ><strong>config_bragg_integration_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-dark-mask-get" class=md-nav__link ><strong>config_dark_mask_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-dark-mask-put" class=md-nav__link ><strong>config_dark_mask_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-detector-get" class=md-nav__link ><strong>config_detector_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-detector-put" class=md-nav__link ><strong>config_detector_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-file-writer-get" class=md-nav__link ><strong>config_file_writer_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-file-writer-put" class=md-nav__link ><strong>config_file_writer_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-image-format-conversion-post" class=md-nav__link ><strong>config_image_format_conversion_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-image-format-get" class=md-nav__link ><strong>config_image_format_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-image-format-put" class=md-nav__link ><strong>config_image_format_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-image-format-raw-post" class=md-nav__link ><strong>config_image_format_raw_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-indexing-get" class=md-nav__link ><strong>config_indexing_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-indexing-put" class=md-nav__link ><strong>config_indexing_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-instrument-get" class=md-nav__link ><strong>config_instrument_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-instrument-put" class=md-nav__link ><strong>config_instrument_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-internal-generator-image-put" class=md-nav__link ><strong>config_internal_generator_image_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-internal-generator-image-tiff-put" class=md-nav__link ><strong>config_internal_generator_image_tiff_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-mask-get" class=md-nav__link ><strong>config_mask_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-mask-tiff-get" class=md-nav__link ><strong>config_mask_tiff_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-roi-get" class=md-nav__link ><strong>config_roi_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-roi-put" class=md-nav__link ><strong>config_roi_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-select-detector-get" class=md-nav__link ><strong>config_select_detector_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-select-detector-put" class=md-nav__link ><strong>config_select_detector_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-spot-finding-get" class=md-nav__link ><strong>config_spot_finding_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-spot-finding-put" class=md-nav__link ><strong>config_spot_finding_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-user-mask-get" class=md-nav__link ><strong>config_user_mask_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-user-mask-put" class=md-nav__link ><strong>config_user_mask_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-user-mask-tiff-get" class=md-nav__link ><strong>config_user_mask_tiff_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-user-mask-tiff-put" class=md-nav__link ><strong>config_user_mask_tiff_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-zeromq-metadata-get" class=md-nav__link ><strong>config_zeromq_metadata_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-zeromq-metadata-put" class=md-nav__link ><strong>config_zeromq_metadata_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-zeromq-preview-get" class=md-nav__link ><strong>config_zeromq_preview_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-zeromq-preview-put" class=md-nav__link ><strong>config_zeromq_preview_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#deactivate-post" class=md-nav__link ><strong>deactivate_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#detector-status-get" class=md-nav__link ><strong>detector_status_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#fpga-status-get" class=md-nav__link ><strong>fpga_status_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-clear-post" class=md-nav__link ><strong>image_buffer_clear_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-image-cbor-get" class=md-nav__link ><strong>image_buffer_image_cbor_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-image-jpeg-get" class=md-nav__link ><strong>image_buffer_image_jpeg_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-image-tiff-get" class=md-nav__link ><strong>image_buffer_image_tiff_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-start-cbor-get" class=md-nav__link ><strong>image_buffer_start_cbor_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-status-get" class=md-nav__link ><strong>image_buffer_status_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-pusher-status-get" class=md-nav__link ><strong>image_pusher_status_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#initialize-post" class=md-nav__link ><strong>initialize_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#pedestal-post" class=md-nav__link ><strong>pedestal_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#preview-pedestal-tiff-get" class=md-nav__link ><strong>preview_pedestal_tiff_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#preview-plot-bin-get" class=md-nav__link ><strong>preview_plot_bin_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#preview-plot-get" class=md-nav__link ><strong>preview_plot_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#result-scan-get" class=md-nav__link ><strong>result_scan_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#start-post" class=md-nav__link ><strong>start_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#statistics-calibration-get" class=md-nav__link ><strong>statistics_calibration_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#statistics-data-collection-get" class=md-nav__link ><strong>statistics_data_collection_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#statistics-get" class=md-nav__link ><strong>statistics_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#status-get" class=md-nav__link ><strong>status_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#trigger-post" class=md-nav__link ><strong>trigger_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#version-get" class=md-nav__link ><strong>version_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#wait-till-done-post" class=md-nav__link ><strong>wait_till_done_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#wait-until-running-post" class=md-nav__link ><strong>wait_until_running_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#xfel-event-code-get" class=md-nav__link ><strong>xfel_event_code_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#xfel-pulse-id-get" class=md-nav__link ><strong>xfel_pulse_id_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/AzimIntSettings.html" class=md-nav__link >AzimIntSettings</a> <li class=md-nav__item > <a href="python_client/docs/BraggIntegrationSettings.html" class=md-nav__link >BraggIntegrationSettings</a> <li class=md-nav__item > <a href="python_client/docs/BrokerStatus.html" class=md-nav__link >BrokerStatus</a> <li class=md-nav__item > <a href="python_client/docs/CalibrationStatisticsInner.html" class=md-nav__link >CalibrationStatisticsInner</a> <li class=md-nav__item > <a href="python_client/docs/DarkMaskSettings.html" class=md-nav__link >DarkMaskSettings</a> <li class=md-nav__item > <a href="python_client/docs/DatasetSettings.html" class=md-nav__link >DatasetSettings</a> <li class=md-nav__item > <a href="python_client/docs/DatasetSettingsSmargon.html" class=md-nav__link >DatasetSettingsSmargon</a> <li class=md-nav__item > <a href="python_client/docs/DatasetSettingsXrayFluorescenceSpectrum.html" class=md-nav__link >DatasetSettingsXrayFluorescenceSpectrum</a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html" class=md-nav__link >jfjoch_client.DefaultApi</a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#cancel-post" class=md-nav__link ><strong>cancel_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-azim-int-get" class=md-nav__link ><strong>config_azim_int_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-azim-int-put" class=md-nav__link ><strong>config_azim_int_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-bragg-integration-get" class=md-nav__link ><strong>config_bragg_integration_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-bragg-integration-put" class=md-nav__link ><strong>config_bragg_integration_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-dark-mask-get" class=md-nav__link ><strong>config_dark_mask_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-dark-mask-put" class=md-nav__link ><strong>config_dark_mask_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-detector-get" class=md-nav__link ><strong>config_detector_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-detector-put" class=md-nav__link ><strong>config_detector_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-file-writer-get" class=md-nav__link ><strong>config_file_writer_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-file-writer-put" class=md-nav__link ><strong>config_file_writer_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-image-format-conversion-post" class=md-nav__link ><strong>config_image_format_conversion_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-image-format-get" class=md-nav__link ><strong>config_image_format_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-image-format-put" class=md-nav__link ><strong>config_image_format_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-image-format-raw-post" class=md-nav__link ><strong>config_image_format_raw_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-indexing-get" class=md-nav__link ><strong>config_indexing_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-indexing-put" class=md-nav__link ><strong>config_indexing_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-instrument-get" class=md-nav__link ><strong>config_instrument_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-instrument-put" class=md-nav__link ><strong>config_instrument_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-internal-generator-image-put" class=md-nav__link ><strong>config_internal_generator_image_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-internal-generator-image-tiff-put" class=md-nav__link ><strong>config_internal_generator_image_tiff_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-mask-get" class=md-nav__link ><strong>config_mask_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-mask-tiff-get" class=md-nav__link ><strong>config_mask_tiff_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-roi-get" class=md-nav__link ><strong>config_roi_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-roi-put" class=md-nav__link ><strong>config_roi_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-select-detector-get" class=md-nav__link ><strong>config_select_detector_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-select-detector-put" class=md-nav__link ><strong>config_select_detector_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-spot-finding-get" class=md-nav__link ><strong>config_spot_finding_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-spot-finding-put" class=md-nav__link ><strong>config_spot_finding_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-user-mask-get" class=md-nav__link ><strong>config_user_mask_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-user-mask-put" class=md-nav__link ><strong>config_user_mask_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-user-mask-tiff-get" class=md-nav__link ><strong>config_user_mask_tiff_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-user-mask-tiff-put" class=md-nav__link ><strong>config_user_mask_tiff_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-zeromq-metadata-get" class=md-nav__link ><strong>config_zeromq_metadata_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-zeromq-metadata-put" class=md-nav__link ><strong>config_zeromq_metadata_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-zeromq-preview-get" class=md-nav__link ><strong>config_zeromq_preview_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#config-zeromq-preview-put" class=md-nav__link ><strong>config_zeromq_preview_put</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#deactivate-post" class=md-nav__link ><strong>deactivate_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#detector-status-get" class=md-nav__link ><strong>detector_status_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#fpga-status-get" class=md-nav__link ><strong>fpga_status_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-clear-post" class=md-nav__link ><strong>image_buffer_clear_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-image-cbor-get" class=md-nav__link ><strong>image_buffer_image_cbor_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-image-jpeg-get" class=md-nav__link ><strong>image_buffer_image_jpeg_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-image-tiff-get" class=md-nav__link ><strong>image_buffer_image_tiff_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-start-cbor-get" class=md-nav__link ><strong>image_buffer_start_cbor_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-buffer-status-get" class=md-nav__link ><strong>image_buffer_status_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#image-pusher-status-get" class=md-nav__link ><strong>image_pusher_status_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#initialize-post" class=md-nav__link ><strong>initialize_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#pedestal-post" class=md-nav__link ><strong>pedestal_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#preview-pedestal-tiff-get" class=md-nav__link ><strong>preview_pedestal_tiff_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#preview-plot-bin-get" class=md-nav__link ><strong>preview_plot_bin_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#preview-plot-get" class=md-nav__link ><strong>preview_plot_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#result-scan-get" class=md-nav__link ><strong>result_scan_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#start-post" class=md-nav__link ><strong>start_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#statistics-calibration-get" class=md-nav__link ><strong>statistics_calibration_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#statistics-data-collection-get" class=md-nav__link ><strong>statistics_data_collection_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#statistics-get" class=md-nav__link ><strong>statistics_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#status-get" class=md-nav__link ><strong>status_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#trigger-post" class=md-nav__link ><strong>trigger_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#version-get" class=md-nav__link ><strong>version_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#wait-till-done-post" class=md-nav__link ><strong>wait_till_done_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#wait-until-running-post" class=md-nav__link ><strong>wait_until_running_post</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#xfel-event-code-get" class=md-nav__link ><strong>xfel_event_code_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/DefaultApi.html#xfel-pulse-id-get" class=md-nav__link ><strong>xfel_pulse_id_get</strong></a> <li class=md-nav__item > <a href="python_client/docs/Detector.html" class=md-nav__link >Detector</a> <li class=md-nav__item > <a href="python_client/docs/DetectorList.html" class=md-nav__link >DetectorList</a> <li class=md-nav__item > <a href="python_client/docs/DetectorListElement.html" class=md-nav__link >DetectorListElement</a> <li class=md-nav__item > <a href="python_client/docs/DetectorModule.html" class=md-nav__link >DetectorModule</a> <li class=md-nav__item > <a href="python_client/docs/DetectorModuleDirection.html" class=md-nav__link >DetectorModuleDirection</a> <li class=md-nav__item > <a href="python_client/docs/DetectorPowerState.html" class=md-nav__link >DetectorPowerState</a> <li class=md-nav__item > <a href="python_client/docs/DetectorSelection.html" class=md-nav__link >DetectorSelection</a> <li class=md-nav__item > <a href="python_client/docs/DetectorSettings.html" class=md-nav__link >DetectorSettings</a> <li class=md-nav__item > <a href="python_client/docs/DetectorState.html" class=md-nav__link >DetectorState</a> <li class=md-nav__item > <a href="python_client/docs/DetectorStatus.html" class=md-nav__link >DetectorStatus</a> <li class=md-nav__item > <a href="python_client/docs/DetectorTiming.html" class=md-nav__link >DetectorTiming</a> <li class=md-nav__item > <a href="python_client/docs/DetectorType.html" class=md-nav__link >DetectorType</a> <li class=md-nav__item > <a href="python_client/docs/ErrorMessage.html" class=md-nav__link >ErrorMessage</a> <li class=md-nav__item > <a href="python_client/docs/FileWriterFormat.html" class=md-nav__link >FileWriterFormat</a> <li class=md-nav__item > <a href="python_client/docs/FileWriterSettings.html" class=md-nav__link >FileWriterSettings</a> <li class=md-nav__item > <a href="python_client/docs/FpgaStatusInner.html" class=md-nav__link >FpgaStatusInner</a> <li class=md-nav__item > <a href="python_client/docs/GeomRefinementAlgorithm.html" class=md-nav__link >GeomRefinementAlgorithm</a> <li class=md-nav__item > <a href="python_client/docs/GridScan.html" class=md-nav__link >GridScan</a> <li class=md-nav__item > <a href="python_client/docs/ImageBufferStatus.html" class=md-nav__link >ImageBufferStatus</a> <li class=md-nav__item > <a href="python_client/docs/ImageFormatSettings.html" class=md-nav__link >ImageFormatSettings</a> <li class=md-nav__item > <a href="python_client/docs/ImagePusherStatus.html" class=md-nav__link >ImagePusherStatus</a> <li class=md-nav__item > <a href="python_client/docs/ImagePusherType.html" class=md-nav__link >ImagePusherType</a> <li class=md-nav__item > <a href="python_client/docs/IndexingAlgorithm.html" class=md-nav__link >IndexingAlgorithm</a> <li class=md-nav__item > <a href="python_client/docs/IndexingSettings.html" class=md-nav__link >IndexingSettings</a> <li class=md-nav__item > <a href="python_client/docs/InstrumentMetadata.html" class=md-nav__link >InstrumentMetadata</a> <li class=md-nav__item > <a href="python_client/docs/IntegrationModel.html" class=md-nav__link >IntegrationModel</a> <li class=md-nav__item > <a href="python_client/docs/JfjochSettings.html" class=md-nav__link >JfjochSettings</a> <li class=md-nav__item > <a href="python_client/docs/JfjochStatistics.html" class=md-nav__link >JfjochStatistics</a> <li class=md-nav__item > <a href="python_client/docs/MeasurementStatistics.html" class=md-nav__link >MeasurementStatistics</a> <li class=md-nav__item > <a href="python_client/docs/PcieDevicesInner.html" class=md-nav__link >PcieDevicesInner</a> <li class=md-nav__item > <a href="python_client/docs/PixelMaskStatistics.html" class=md-nav__link >PixelMaskStatistics</a> <li class=md-nav__item > <a href="python_client/docs/Plot.html" class=md-nav__link >Plot</a> <li class=md-nav__item > <a href="python_client/docs/PlotUnitX.html" class=md-nav__link >PlotUnitX</a> <li class=md-nav__item > <a href="python_client/docs/Plots.html" class=md-nav__link >Plots</a> <li class=md-nav__item > <a href="python_client/docs/RoiAzimList.html" class=md-nav__link >RoiAzimList</a> <li class=md-nav__item > <a href="python_client/docs/RoiAzimuthal.html" class=md-nav__link >RoiAzimuthal</a> <li class=md-nav__item > <a href="python_client/docs/RoiBox.html" class=md-nav__link >RoiBox</a> <li class=md-nav__item > <a href="python_client/docs/RoiBoxList.html" class=md-nav__link >RoiBoxList</a> <li class=md-nav__item > <a href="python_client/docs/RoiCircle.html" class=md-nav__link >RoiCircle</a> <li class=md-nav__item > <a href="python_client/docs/RoiCircleList.html" class=md-nav__link >RoiCircleList</a> <li class=md-nav__item > <a href="python_client/docs/RoiDefinitions.html" class=md-nav__link >RoiDefinitions</a> <li class=md-nav__item > <a href="python_client/docs/RotationAxis.html" class=md-nav__link >RotationAxis</a> <li class=md-nav__item > <a href="python_client/docs/ScanResult.html" class=md-nav__link >ScanResult</a> <li class=md-nav__item > <a href="python_client/docs/ScanResultImagesInner.html" class=md-nav__link >ScanResultImagesInner</a> <li class=md-nav__item > <a href="python_client/docs/SpotFindingSettings.html" class=md-nav__link >SpotFindingSettings</a> <li class=md-nav__item > <a href="python_client/docs/StandardDetectorGeometry.html" class=md-nav__link >StandardDetectorGeometry</a> <li class=md-nav__item > <a href="python_client/docs/TcpSettings.html" class=md-nav__link >TcpSettings</a> <li class=md-nav__item > <a href="python_client/docs/UnitCell.html" class=md-nav__link >UnitCell</a> <li class=md-nav__item > <a href="python_client/docs/ZeromqMetadataSettings.html" class=md-nav__link >ZeromqMetadataSettings</a> <li class=md-nav__item > <a href="python_client/docs/ZeromqPreviewSettings.html" class=md-nav__link >ZeromqPreviewSettings</a> <li class=md-nav__item > <a href="python_client/docs/ZeromqSettings.html" class=md-nav__link >ZeromqSettings</a> </ul> </nav> </div> </div> </div> <div class="md-sidebar md-sidebar--secondary" data-md-component=toc > <div class=md-sidebar__scrollwrap > <div class=md-sidebar__inner > <nav class="md-nav md-nav--secondary"> <ul class=md-nav__list data-md-scrollfix=""> </ul> </nav> </div> </div> </div> <div class=md-content > <article class="md-content__inner md-typeset" role=main > <section class="tex2jax_ignore mathjax_ignore" id=cpu-side-crystallographic-data-analysis-jungfraujoch > <h1 id=cpu-data-analysis--page-root >CPU-side crystallographic data analysis (Jungfraujoch)<a class=headerlink href="#cpu-data-analysis--page-root" title="Link to this heading">¶</a></h1> <p>This document describes the crystallographic algorithms implemented in Jungfraujoch for <strong>CPU</strong>- and <strong>GPU</strong>-side real‑time and near‑real‑time data analysis.</p> <p><strong>Scope.</strong> The pipeline covered here comprises:</p> <ol class="arabic simple"> <li><p>geometry mapping and corrections,</p> <li><p>azimuthal integration (powder/radial profiles),</p> <li><p>Bragg spot finding (strong pixels → connected components → spot descriptors),</p> <li><p>indexing (still and rotation modes),</p> <li><p>Bravais lattice / centering inference,</p> <li><p>geometry and lattice refinement,</p> <li><p>reflection prediction (still and rotation),</p> <li><p>Bragg integration by either 2D box summation or profile fitting (Kabsch, reference-free),</p> <li><p>scaling and merging,</p> <li><p>merge-level error modelling, outlier rejection and the resolution cutoff,</p> <li><p>space-group determination from the merged intensities (Laue group, screw axes, centering) and the twinning check,</p> <li><p>auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref),</p> <li><p>amplitude estimation (French–Wilson) and R-free test-set flagging,</p> <li><p>optional model-based validation: R-free against a supplied model, 2Fo−Fc / Fo−Fc electron-density maps, and an anomalous difference map with the strongest anomalous sites named.</p> </ol> <nav class="contents local" id=on-this-page > <p class=topic-title >On this page</p> <ul class=simple > <li><p><a class="reference internal" href="#references" id=id1 >References</a></p> <li><p><a class="reference internal" href="#getting-the-image-onto-the-gpu-device-side-bitshuffle-lz4-decoding" id=id2 >0. Getting the image onto the GPU: device-side bitshuffle+LZ4 decoding</a></p> <li><p><a class="reference internal" href="#geometry-reciprocal-space-mapping-and-basic-quantities" id=id3 >1. Geometry, reciprocal-space mapping, and basic quantities</a></p> <ul> <li><p><a class="reference internal" href="#coordinate-conventions" id=id4 >1.1 Coordinate conventions</a></p> <li><p><a class="reference internal" href="#two-theta-azimuth-resolution-and-q" id=id5 >1.2 Two-theta, azimuth, resolution and <span class="math notranslate nohighlight">\(q\)</span></a></p> <li><p><a class="reference internal" href="#distance-from-the-ewald-sphere" id=id6 >1.3 Distance from the Ewald sphere</a></p> <li><p><a class="reference internal" href="#measuring-the-direct-beam-before-indexing" id=id7 >1.4 Measuring the direct beam before indexing</a></p> <li><p><a class="reference internal" href="#finding-the-beam-stop" id=id8 >1.5 Finding the beam stop</a></p> </ul> <li><p><a class="reference internal" href="#azimuthal-integration-radial-profiles" id=id9 >2. Azimuthal integration (radial profiles)</a></p> <ul> <li><p><a class="reference internal" href="#histogram-estimator" id=id10 >2.1 Histogram estimator</a></p> <li><p><a class="reference internal" href="#corrections-applied" id=id11 >2.2 Corrections applied</a></p> <li><p><a class="reference internal" href="#background-estimate-for-profiles" id=id12 >2.3 Background estimate for profiles</a></p> </ul> <li><p><a class="reference internal" href="#spot-finding-strong-pixels-bragg-spots" id=id13 >3. Spot finding (strong pixels → Bragg spots)</a></p> <ul> <li><p><a class="reference internal" href="#strong-pixel-detection-by-local-statistics" id=id14 >3.1 Strong-pixel detection by local statistics</a></p> <li><p><a class="reference internal" href="#adaptive-self-calibrating-detection" id=id15 >3.2 Adaptive (self-calibrating) detection</a></p> <li><p><a class="reference internal" href="#resolution-and-ice-ring-handling" id=id16 >3.3 Resolution and ice-ring handling</a></p> <li><p><a class="reference internal" href="#connected-component-labeling-ccl" id=id17 >3.4 Connected-component labeling (CCL)</a></p> <li><p><a class="reference internal" href="#adaptive-per-image-minimum-spot-size" id=id18 >3.5 Adaptive per-image minimum spot size</a></p> <li><p><a class="reference internal" href="#predicting-the-resolution-the-merged-data-will-reach" id=id19 >3.6 Predicting the resolution the merged data will reach</a></p> </ul> <li><p><a class="reference internal" href="#indexing-overview" id=id20 >4. Indexing overview</a></p> <ul> <li><p><a class="reference internal" href="#indexed-spot-decision-inlier-test" id=id21 >4.1 Indexed-spot decision (inlier test)</a></p> </ul> <li><p><a class="reference internal" href="#fft-indexing-unknown-unit-cell" id=id22 >5. FFT indexing (unknown unit cell)</a></p> <ul> <li><p><a class="reference internal" href="#directional-projections-and-histograms" id=id23 >5.1 Directional projections and histograms</a></p> <li><p><a class="reference internal" href="#fft-peak-picking-and-candidate-vectors" id=id24 >5.2 FFT peak picking and candidate vectors</a></p> <li><p><a class="reference internal" href="#lattice-reduction-and-cell-candidates" id=id25 >5.3 Lattice reduction and cell candidates</a></p> <li><p><a class="reference internal" href="#robust-refinement-and-best-cell-selection" id=id26 >5.4 Robust refinement and best-cell selection</a></p> </ul> <li><p><a class="reference internal" href="#bravais-lattice-centering-inference-lattice-search" id=id27 >6. Bravais lattice / centering inference (“lattice search”)</a></p> <li><p><a class="reference internal" href="#geometry-and-lattice-refinement" id=id28 >7. Geometry and lattice refinement</a></p> <ul> <li><p><a class="reference internal" href="#parameterization" id=id29 >7.1 Parameterization</a></p> <li><p><a class="reference internal" href="#residuals-and-objective" id=id30 >7.2 Residuals and objective</a></p> <li><p><a class="reference internal" href="#rotation-datasets-bringing-observations-to-a-common-reference-frame" id=id31 >7.3 Rotation datasets: bringing observations to a common reference frame</a></p> <li><p><a class="reference internal" href="#multi-stage-tightening-of-inlier-tolerance" id=id32 >7.4 Multi-stage tightening of inlier tolerance</a></p> <li><p><a class="reference internal" href="#rotation-geometry-post-refinement-two-pass" id=id33 >7.5 Rotation geometry post-refinement (two-pass)</a></p> <li><p><a class="reference internal" href="#detector-geometry-from-powder-rings" id=id34 >7.6 Detector geometry from powder rings</a></p> </ul> <li><p><a class="reference internal" href="#reflection-prediction" id=id35 >8. Reflection prediction</a></p> <ul> <li><p><a class="reference internal" href="#enumerating-reciprocal-lattice-points" id=id36 >8.1 Enumerating reciprocal lattice points</a></p> <li><p><a class="reference internal" href="#still-prediction-excitation-error-cutoff" id=id37 >8.2 Still prediction (excitation-error cutoff)</a></p> <li><p><a class="reference internal" href="#rotation-prediction-laue-equation-partiality-model" id=id38 >8.3 Rotation prediction (Laue equation + partiality model)</a></p> <li><p><a class="reference internal" href="#systematic-absences-centering" id=id39 >8.4 Systematic absences (centering)</a></p> </ul> <li><p><a class="reference internal" href="#d-bragg-integration-profile-fitting-over-a-three-ring-roi" id=id40 >9. 2D Bragg integration (profile fitting over a three-ring ROI)</a></p> <ul> <li><p><a class="reference internal" href="#regions-of-interest" id=id41 >9.1 Regions of interest</a></p> <li><p><a class="reference internal" href="#box-summation-seed-and-fallback" id=id42 >9.2 Box summation (seed and fallback)</a></p> <li><p><a class="reference internal" href="#profile-fitted-extraction-default" id=id43 >9.3 Profile-fitted extraction (default)</a></p> <li><p><a class="reference internal" href="#lorentzpolarization-factor-handling" id=id44 >9.4 Lorentz–polarization factor handling</a></p> <li><p><a class="reference internal" href="#choosing-the-signal-radius-from-the-crystals-own-spots-rotation" id=id45 >9.5 Choosing the signal radius from the crystal’s own spots (rotation)</a></p> </ul> <li><p><a class="reference internal" href="#scaling-and-merging" id=id46 >10. Scaling and merging</a></p> <ul> <li><p><a class="reference internal" href="#observation-model" id=id47 >10.1 Observation model</a></p> <li><p><a class="reference internal" href="#partiality-models" id=id48 >10.2 Partiality models</a></p> <li><p><a class="reference internal" href="#smoothing-of-per-frame-scales" id=id49 >10.3 Smoothing of per-frame scales</a></p> <li><p><a class="reference internal" href="#merging-estimator" id=id50 >10.4 Merging estimator</a></p> <li><p><a class="reference internal" href="#merging-statistics" id=id51 >10.5 Merging statistics</a></p> <li><p><a class="reference internal" href="#rotation-datasets-combining-partials-into-fulls-3d-integration" id=id52 >10.6 Rotation datasets: combining partials into fulls (3D integration)</a></p> <li><p><a class="reference internal" href="#r-free-test-set-flags" id=id53 >10.7 R-free test-set flags</a></p> <li><p><a class="reference internal" href="#frenchwilson-amplitudes" id=id54 >10.8 French–Wilson amplitudes</a></p> <li><p><a class="reference internal" href="#reference-data-fixing-the-space-group-and-resolving-the-indexing-ambiguity" id=id55 >10.9 Reference data: fixing the space group and resolving the indexing ambiguity</a></p> <li><p><a class="reference internal" href="#ice-rings-at-the-scale-and-merge-stages" id=id56 >10.10 Ice rings at the scale and merge stages</a></p> </ul> <li><p><a class="reference internal" href="#mosaicity-and-profile-radius-monitoring" id=id57 >11. Mosaicity and “profile radius” monitoring</a></p> <ul> <li><p><a class="reference internal" href="#profile-radius-intrinsic-excitation-error-width" id=id58 >11.1 Profile radius (intrinsic excitation-error width)</a></p> <li><p><a class="reference internal" href="#mosaicity-from-rotation-data" id=id59 >11.2 Mosaicity from rotation data</a></p> </ul> <li><p><a class="reference internal" href="#auxiliary-statistics-i-i-and-wilson-plot" id=id60 >12. Auxiliary statistics: ⟨I/σ(I)⟩ and Wilson plot</a></p> <ul> <li><p><a class="reference internal" href="#per-shell-i-i" id=id61 >12.1 Per-shell ⟨I/σ(I)⟩</a></p> <li><p><a class="reference internal" href="#wilson-plot-b-factor-proxy" id=id62 >12.2 Wilson plot (B-factor proxy)</a></p> </ul> <li><p><a class="reference internal" href="#space-group-determination-and-merge-level-decisions" id=id63 >13. Space-group determination and merge-level decisions</a></p> <ul> <li><p><a class="reference internal" href="#space-group-determination" id=id64 >13.1 Space-group determination</a></p> <li><p><a class="reference internal" href="#twinning-check" id=id65 >13.2 Twinning check</a></p> <li><p><a class="reference internal" href="#outlier-rejection" id=id66 >13.3 Outlier rejection</a></p> <li><p><a class="reference internal" href="#automatic-resolution-cutoff" id=id67 >13.4 Automatic resolution cutoff</a></p> <li><p><a class="reference internal" href="#diffraction-anisotropy" id=id68 >13.5 Diffraction anisotropy</a></p> <li><p><a class="reference internal" href="#practical-notes-and-limitations" id=id69 >13.6 Practical notes and limitations</a></p> </ul> <li><p><a class="reference internal" href="#model-based-validation-r-free-against-a-model-and-electron-density-maps" id=id70 >14. Model-based validation: R-free against a model and electron-density maps</a></p> <ul> <li><p><a class="reference internal" href="#model-structure-factors" id=id71 >14.1 Model structure factors</a></p> <li><p><a class="reference internal" href="#bulk-solvent-and-scaling" id=id72 >14.2 Bulk solvent and scaling</a></p> <li><p><a class="reference internal" href="#r-work-and-r-free" id=id73 >14.3 R-work and R-free</a></p> <li><p><a class="reference internal" href="#electron-density-maps" id=id74 >14.4 Electron-density maps</a></p> <li><p><a class="reference internal" href="#aligning-the-data-to-the-model-enantiomorph-and-indexing-ambiguity" id=id75 >14.5 Aligning the data to the model: enantiomorph and indexing ambiguity</a></p> <li><p><a class="reference internal" href="#anomalous-difference-map-and-the-sites-it-names" id=id76 >14.6 Anomalous difference map and the sites it names</a></p> </ul> </ul> </nav> <section id=references > <h2 id=references ><a class=toc-backref href="#id1" role=doc-backlink >References</a><a class=headerlink href="#references" title="Link to this heading">¶</a></h2> <p>The methods draw on, and in places reimplement, solutions from:</p> <ul class=simple > <li><p>W. Kabsch, “XDS”, <em>Acta Cryst.</em> <strong>D66</strong> (2010), 125–132 and related XDS papers (rotation geometry, partiality, scaling concepts).</p> <li><p>W. Kabsch, “Integration, scaling, space-group assignment and post-refinement”, <em>Acta Cryst.</em> <strong>D66</strong> (2010), 133–144 (mosaicity/partiality likelihood treatment; notation such as ζ and rotation factors).</p> <li><p>T. A. White et al., CrystFEL method papers (spot finding, three‑ring integration, serial/still diffraction processing concepts).</p> <li><p>J. Kieffer & J. P. Wright, “PyFAI: a Python library for high performance azimuthal integration on GPU”, <em>Powder Diffraction</em> <strong>28</strong> (2013), S339-S350 (detector geometry definition, azimuthal integration)</p> <li><p>H. Powell, “The Rossmann Fourier autoindexing algorithm in MOSFLM”, <em>Acta Cryst.</em> <strong>D55</strong> (1999), 1690-1695 (FFT indexing)</p> <li><p>S. French & K. Wilson, “On the treatment of negative intensity observations”, <em>Acta Cryst.</em> <strong>A34</strong> (1978), 517-525 (Bayesian amplitude estimation from intensities).</p> <li><p>A. T. Brünger, “Free R value: a novel statistical quantity for assessing the accuracy of crystal structures”, <em>Nature</em> <strong>355</strong> (1992), 472-475 (R-free cross-validation).</p> <li><p>M. Wojdyr, “GEMMI: A library for structural biology”, <em>J. Open Source Softw.</em> <strong>7</strong> (2022), 4200 (model / structure-factor / map machinery used in §14).</p> <li><p>J. P. Wright, “Experiences with GPU decompression for bitshuffle + LZ4 data”, HDF5 User Group meeting (2021), and <a class="reference external" href="https://github.com/jonwright/bslz4decoders">github.com/jonwright/bslz4decoders</a> (device-side decoding of bitshuffle+LZ4 images, §0).</p> <li><p>A. Thorn & G. M. Sheldrick, “ANODE: anomalous and heavy-atom density calculation”, <em>J. Appl. Cryst.</em> <strong>44</strong> (2011), 1285-1287 (anomalous difference density read at the model’s sites).</p> <li><p>Z. Otwinowski & W. Minor, “Processing of X-ray diffraction data collected in oscillation mode”, <em>Methods Enzymol.</em> <strong>276</strong> (1997), 307-326 (reweighted, de-biased profile-fit variances).</p> <li><p>G. Winter et al., “DIALS: implementation and evaluation of a new integration package”, <em>Acta Cryst.</em> <strong>D74</strong> (2018), 85-97, and J. Beilsten-Edmands et al., <em>Acta Cryst.</em> <strong>D76</strong> (2020), 385-399 (CC1/2 resolution cutoff, merge outlier rejection, scaling error model).</p> <li><p>P. Evans, “Scaling and assessment of data quality”, <em>Acta Cryst.</em> <strong>D62</strong> (2006), 72-82, and P. R. Evans, <em>Acta Cryst.</em> <strong>D67</strong> (2011), 282-292 (POINTLESS: operator-by-operator point-group scoring, and the axial-zone screw-absence test).</p> <li><p>A. G. W. Leslie & H. R. Powell, “Processing diffraction data with MOSFLM” (2007), NATO Science Series II <strong>245</strong>, 41-51 (post-refinement practice: what is refined per image and what over a wedge).</p> <li><p>D. W. Moreau, H. Atakisi & R. E. Thorne, “Ice in biomolecular cryocrystallography”, <em>Acta Cryst.</em> <strong>D77</strong> (2021), 540-554 (measured hexagonal-ice ring positions, used by the ice-ring score, the ice flagging and the ice calibrant).</p> <li><p>S. Sheriff & W. A. Hendrickson, “Description of overall anisotropy in diffraction from macromolecular crystals”, <em>Acta Cryst.</em> <strong>A43</strong> (1987), 118-121, and A. N. Popov & G. P. Bourenkov, <em>Acta Cryst.</em> <strong>D59</strong> (2003), 1145-1153 (the overall anisotropic B tensor, its symmetry constraints, and its estimation from the observed intensities).</p> <li><p>P. R. Evans & G. N. Murshudov, “How good are my data and what is the resolution?”, <em>Acta Cryst.</em> <strong>D69</strong> (2013), 1204-1214 (AIMLESS: the anisotropic deltaB as the range of the principal components, and diffraction limits from a cone about each principal direction).</p> <li><p>K. Diederichs & P. A. Karplus, <em>Nat. Struct. Biol.</em> <strong>4</strong> (1997), 269-275, and P. A. Karplus & K. Diederichs, <em>Science</em> <strong>336</strong> (2012), 1030-1033 (R_meas / R_pim, CC1/2 and CC*).</p> <li><p>IUCr Commission on Crystallographic Nomenclature, “Statistical descriptors in crystallography”, <em>Acta Cryst.</em> <strong>A45</strong> (1989), 63-75, and <em>Acta Cryst.</em> <strong>A51</strong> (1995), 565-569 (uncertainty conventions).</p> </ul> <p>(list is not exhaustive; the full citations, with DOIs, are in <a class="reference internal" href=ACKNOWLEDGEMENT.html ><span class="std std-doc">ACKNOWLEDGEMENT.md</span></a>)</p> </section> <section id=getting-the-image-onto-the-gpu-device-side-bitshuffle-lz4-decoding > <h2 id=getting-the-image-onto-the-gpu-device-side-bitshuffle-lz4-decoding ><a class=toc-backref href="#id2" role=doc-backlink >0. Getting the image onto the GPU: device-side bitshuffle+LZ4 decoding</a><a class=headerlink href="#getting-the-image-onto-the-gpu-device-side-bitshuffle-lz4-decoding" title="Link to this heading">¶</a></h2> <p>Images arrive bitshuffle+LZ4 compressed (HDF5 filter 32008), and everything from §1 onwards runs on the GPU when one is present. Instead of decompressing on the host and uploading the image, the compressed chunk is uploaded — a few MB rather than tens of MB — and decoded on the device. The approach follows Jon Wright (ESRF); the kernels are Jungfraujoch’s own.</p> <p>One kernel does the work: one CUDA block owns one bitshuffle block, from the compressed payload through to finished pixels.</p> <ol class="arabic simple"> <li><p><strong>LZ4 into shared memory, one warp per bitshuffle block.</strong> Blocks are independent, so the parallelism is across them; within the warp every lane runs the same sequence parser over the same bytes, and the literal and match copies are split across the 32 lanes so the stores coalesce. An overlapping match is treated as a pattern of period <code class="docutils literal notranslate"><span class=pre >offset</span></code> sourced from bytes that already precede the write position, which keeps it parallel rather than a serial byte loop; <code class="docutils literal notranslate"><span class=pre >offset</span> <span class=pre >==</span> <span class=pre >1</span></code> (a run of one repeated byte, the common case in sparse detector data) and power-of-two offsets avoid the modulo altogether. Because the lanes cooperate on the copies, each one is followed by <code class="docutils literal notranslate"><span class=pre >__syncwarp()</span></code> — a later match can read bytes another lane wrote, and since Volta that ordering is not implicit.</p> <li><p><strong>The bitshuffle inverse fused with preprocessing.</strong> The whole CUDA block then reads that shared buffer back: one thread owns one group of 8 elements across every byte-plane, so once it has transposed its 8 bytes out of each plane it holds 8 complete elements — and it applies the pixel mask, the error marker and the saturation cap and emits 8 finished <code class="docutils literal notranslate"><span class=pre >int32</span></code> pixels directly. Nothing of the block reaches device memory but the pixels — neither the bitshuffled bytes nor the decompressed image is ever materialised. For 8-bit images there is a single plane and the assembly degenerates to a copy.</p> </ol> <p>Decoding into shared memory is worth more than the bandwidth it saves: an LZ4 match reads back bytes written a few sequences earlier, so every copy step is a dependent round trip — tens of cycles in shared memory against hundreds in device memory. It is paid for in residency, because the buffer holds a whole bitshuffle block, and a block larger than 16 kB (larger than either writer this pipeline reads produces) falls back to a pair of kernels instead, the first writing the shuffled image to device memory and the second un-transposing and preprocessing out of it.</p> <p>The block offsets inside the container can only be discovered by reading the block lengths in order, so that scan stays on the host.</p> <p>Only <code class="docutils literal notranslate"><span class=pre >BSHUF_LZ4</span></code> is decoded on the device. For the zstd variants (<code class="docutils literal notranslate"><span class=pre >BSHUF_ZSTD</span></code>, <code class="docutils literal notranslate"><span class=pre >BSHUF_ZSTD_RLE</span></code>, <code class="docutils literal notranslate"><span class=pre >BSHUF_ZSTD_RLE_HUFF</span></code>), and for uncompressed or float images, <code class="docutils literal notranslate"><span class=pre >BSLZ4DecoderGPU::Supports()</span></code> returns false and the pipeline decompresses on the host and uploads as before.</p> <p>The container arrives off the network or off disk and is not trusted. Everything checkable on the host — declared sizes, the block scan, a block size that is not a multiple of 8 elements, a block count the chunk could not hold, trailing bytes — is rejected before any work is queued; the kernel additionally flags a block that did not decode to exactly its declared length, which becomes an exception once the caller has synchronised. That last check matters because the decode buffers are reused frame to frame: a block that stopped early would leave the <em>previous</em> image’s most significant byte-plane in place, which reads not as a missing corner but as real pixels several powers of two too bright.</p> </section> <section id=geometry-reciprocal-space-mapping-and-basic-quantities > <h2 id=geometry-reciprocal-space-mapping-and-basic-quantities ><a class=toc-backref href="#id3" role=doc-backlink >1. Geometry, reciprocal-space mapping, and basic quantities</a><a class=headerlink href="#geometry-reciprocal-space-mapping-and-basic-quantities" title="Link to this heading">¶</a></h2> <section id=coordinate-conventions > <h3 id=coordinate-conventions ><a class=toc-backref href="#id4" role=doc-backlink >1.1 Coordinate conventions</a><a class=headerlink href="#coordinate-conventions" title="Link to this heading">¶</a></h3> <p>For a pixel coordinate <span class="math notranslate nohighlight">\((x,y)\)</span> (in pixels), Jungfraujoch converts to a laboratory direction vector via:</p> <ol class="arabic simple"> <li><p>shift by direct-beam position <span class="math notranslate nohighlight">\((x_\mathrm{beam}, y_\mathrm{beam})\)</span>,</p> <li><p>scale by pixel size <span class="math notranslate nohighlight">\(p\)</span> (mm),</p> <li><p>set detector distance <span class="math notranslate nohighlight">\(D\)</span> (mm),</p> <li><p>apply detector orientation rotation <span class="math notranslate nohighlight">\(R_\mathrm{det}\)</span> (PyFAI-like parameterization).</p> </ol> <p>The unnormalized detector coordinate (mm) is: <span class="math notranslate nohighlight">\( \mathbf{r}_\mathrm{det}(x,y) = \begin{pmatrix} (x-x_\mathrm{beam})p\\ (y-y_\mathrm{beam})p\\ D \end{pmatrix}. \)</span></p> <p>The lab-frame vector is: <span class="math notranslate nohighlight">\( \mathbf{r}_\mathrm{lab} = R_\mathrm{det}\,\mathbf{r}_\mathrm{det}. \)</span></p> <p>Let the incident wavevector magnitude be <span class="math notranslate nohighlight">\(k = 1/\lambda\)</span> in Å<span class="math notranslate nohighlight">\(^{-1}\)</span>, and define: <span class="math notranslate nohighlight">\( \mathbf{S}_0 = (0,0,k). \)</span></p> <p>The <strong>reciprocal-space scattering vector</strong> associated with pixel <span class="math notranslate nohighlight">\((x,y)\)</span> is: <span class="math notranslate nohighlight">\( \mathbf{s}(x,y) = k\,\frac{\mathbf{r}_\mathrm{lab}}{\lVert \mathbf{r}_\mathrm{lab}\rVert} - \mathbf{S}_0. \)</span></p> <p>This <span class="math notranslate nohighlight">\(\mathbf{s}\)</span> is the fundamental quantity used for spot finding (resolution filters), indexing, and refinement.</p> </section> <section id=two-theta-azimuth-resolution-and-q > <h3 id=two-theta-azimuth-resolution-and-q ><a class=toc-backref href="#id5" role=doc-backlink >1.2 Two-theta, azimuth, resolution and <span class="math notranslate nohighlight">\(q\)</span></a><a class=headerlink href="#two-theta-azimuth-resolution-and-q" title="Link to this heading">¶</a></h3> <p>The scattering angle <span class="math notranslate nohighlight">\(2\theta\)</span> is computed from <span class="math notranslate nohighlight">\(\mathbf{r}_\mathrm{lab}\)</span> via: <span class="math notranslate nohighlight">\( 2\theta = \arctan\!\left(\frac{\sqrt{x_\mathrm{lab}^2 + y_\mathrm{lab}^2}}{z_\mathrm{lab}}\right). \)</span></p> <p>Resolution (Å) at a pixel is: <span class="math notranslate nohighlight">\( d = \frac{\lambda}{2\sin\theta}. \)</span></p> <p>The magnitude <span class="math notranslate nohighlight">\(q = 2\pi/d\)</span> is used for radial binning and ice-ring handling.</p> </section> <section id=distance-from-the-ewald-sphere > <h3 id=distance-from-the-ewald-sphere ><a class=toc-backref href="#id6" role=doc-backlink >1.3 Distance from the Ewald sphere</a><a class=headerlink href="#distance-from-the-ewald-sphere" title="Link to this heading">¶</a></h3> <p>For a reciprocal lattice point <span class="math notranslate nohighlight">\(\mathbf{p}\)</span> (Å<span class="math notranslate nohighlight">\(^{-1}\)</span>), define: <span class="math notranslate nohighlight">\( \Delta_\mathrm{Ewald}(\mathbf{p}) = \lVert \mathbf{p} + \mathbf{S}_0\rVert - k. \)</span> Jungfraujoch uses <span class="math notranslate nohighlight">\(|\Delta_\mathrm{Ewald}|\)</span> as an operational proxy for excitation error. This appears in:</p> <ul class=simple > <li><p>still prediction (accept if <span class="math notranslate nohighlight">\(|\Delta_\mathrm{Ewald}|\le \Delta_\mathrm{cut}\)</span>),</p> <li><p>profile radius estimation (see §11.1),</p> <li><p>still partiality option in scaling/merging (§10.2).</p> </ul> </section> <section id=measuring-the-direct-beam-before-indexing > <h3 id=measuring-the-direct-beam-before-indexing ><a class=toc-backref href="#id7" role=doc-backlink >1.4 Measuring the direct beam before indexing</a><a class=headerlink href="#measuring-the-direct-beam-before-indexing" title="Link to this heading">¶</a></h3> <p>The beam centre in the file is often a placeholder, and nothing else measures it until post-refinement (§7.5) — by which time a wrong centre has already chosen the lattice. With <code class="docutils literal notranslate"><span class=pre >--estimate-beam-center</span></code> it is measured from spot positions alone, before anything is indexed. Two exact facts about a rotation sweep supply the two coordinates:</p> <p><strong>Friedel mates half a turn apart.</strong> The Laue condition fixes the component of <span class="math notranslate nohighlight">\(\mathbf{q}\)</span> along the beam, <span class="math notranslate nohighlight">\(q_\parallel = -\lVert\mathbf{q}\rVert^2\lambda/2\)</span>. Rotating 180° about the spindle <span class="math notranslate nohighlight">\(\mathbf{m}\)</span> negates the two components perpendicular to <span class="math notranslate nohighlight">\(\mathbf{m}\)</span> and taking <span class="math notranslate nohighlight">\(-h\)</span> negates all three, so together they negate <strong>only</strong> the component along <span class="math notranslate nohighlight">\(\mathbf{m}\)</span> and leave <span class="math notranslate nohighlight">\(q_\parallel\)</span> untouched. With the spindle perpendicular to the beam, <span class="math notranslate nohighlight">\(-h\)</span> therefore diffracts at <span class="math notranslate nohighlight">\(\varphi+180°\)</span> exactly where <span class="math notranslate nohighlight">\(h\)</span> diffracts at <span class="math notranslate nohighlight">\(\varphi\)</span>, and its spot sits at the mirror image of <span class="math notranslate nohighlight">\(h\)</span>’s along the spindle. This gives the beam coordinate <strong>along</strong> the spindle. Only the reciprocal lattice’s centrosymmetry is needed for the geometry; Friedel’s law <span class="math notranslate nohighlight">\(|F(h)|=|F(-h)|\)</span> is used separately, to tell a true pairing from an accidental one.</p> <p><strong>The second crossing.</strong> The same reflection meets the Ewald sphere twice, at two angles that are generally <em>not</em> 180° apart, differing only in the sign of the lab component perpendicular to both <span class="math notranslate nohighlight">\(\mathbf{m}\)</span> and the beam. This gives the remaining coordinate. The two crossings are separated by a sweep angle fixed by the reflection’s own position, which is what identifies genuine pairs.</p> <p>Neither observable requires a cell or an orientation matrix: each candidate pairing votes for a beam coordinate, and the true value accumulates while wrong pairings scatter. A Friedel pair needs both <span class="math notranslate nohighlight">\(\varphi\)</span> and <span class="math notranslate nohighlight">\(\varphi+180°\)</span> recorded, so a sweep of <span class="math notranslate nohighlight">\(S°\)</span> yields only <span class="math notranslate nohighlight">\(S-180\)</span> degrees’ worth of pairs — a sweep of exactly half a turn yields none, and the estimator is refused below a floor on that span.</p> <p>The mirror is exact in the <strong>laboratory</strong> frame, so it is sensitive to the spindle direction. A skew of the spindle about the beam <em>spreads</em> the vote rather than shifting it, and is fitted alongside the centre (<code class="docutils literal notranslate"><span class=pre >--no-fit-spindle</span></code> keeps the axis from the file); a tilt of the spindle towards the beam is measured and reported but not applied, being confounded with the detector tilt until that is fitted too. Nothing inside the fit can tell that the vote settled on the wrong periodic maximum — every frame pair agrees with every other — so the answer is accepted only if it does not move when the search is started from a different position. Frames are sampled away from both ends of the sweep, where shutter synchronisation can spoil an image.</p> <p>Where the sweep is shorter than half a turn the spot symmetry cannot be formed, and the centre is taken instead from the centroid of the radial background profile, which needs only a few images. Where neither method can measure the centre, the value from the file is kept.</p> </section> <section id=finding-the-beam-stop > <h3 id=finding-the-beam-stop ><a class=toc-backref href="#id8" role=doc-backlink >1.5 Finding the beam stop</a><a class=headerlink href="#finding-the-beam-stop" title="Link to this heading">¶</a></h3> <p>The beam stop and its holder arm shadow part of the detector. A reflection behind them is attenuated but otherwise ordinary — it integrates low, with a plausible <span class="math notranslate nohighlight">\(\sigma\)</span>, and no outlier test catches it — so the shadow is found and masked instead. <code class="docutils literal notranslate"><span class=pre >--detect-beam-stop[=N|off]</span></code> (on by default, <span class="math notranslate nohighlight">\(N=60\)</span> frames; it reads its own frames, so it is not affected by, and does not affect, the beam-centre pre-scan of §1.4) projects those frames to a per-pixel mean and maximum, and writes the result into the pixel mask as bit 9, from where it excludes those pixels from every later stage.</p> <p>The shadow is a place where the background is <em>missing</em>, so it is found by comparing each pixel’s background against the background at the same radius. The mean is pooled over a small box first — a single pixel of a sparse background carries too few counts to tell a shadow from a Poisson hole, and the stop is much wider than the box — and the comparison is against the median of the pixel’s own radius ring, taken over the pixels not already excluded and iterated a few times so the shadow stays out of the baseline it is measured against. A pixel is shadow when the ratio falls below 0.35, and only where the ring has accumulated enough counts for the dip to mean anything. Nothing assumes the stop and the beam are concentric, because only the per-ring comparison is used; a ring lying wholly inside the stop has no unshadowed pixel for its median to find, which is exactly the case where the comparison must fail, and such a ring is shadow in its entirety.</p> <p>What survives is then shaped into a region: the low pixels connected to the beam centre, bridged across the module gaps the holder arm crosses, grown outward through the partially shadowed penumbra, closed, and with the interior of the disk filled. Last, any pixel that ever recorded a real reflection — a maximum over the frames well above background, in a cluster, so that a single-frame zinger does not count — is given back, because a beam stop cannot have blocked a reflection that was measured.</p> </section> </section> <hr class=docutils /> <section id=azimuthal-integration-radial-profiles > <h2 id=azimuthal-integration-radial-profiles ><a class=toc-backref href="#id9" role=doc-backlink >2. Azimuthal integration (radial profiles)</a><a class=headerlink href="#azimuthal-integration-radial-profiles" title="Link to this heading">¶</a></h2> <p>Azimuthal integration produces a radial profile <span class="math notranslate nohighlight">\(I(q)\)</span> or <span class="math notranslate nohighlight">\(I(d)\)</span> by histogramming pixels into radial bins. Pixels are <strong>not split</strong> across bins; each pixel contributes wholly to a single bin. By default the profile is purely radial (a single azimuthal bin), but the azimuth can optionally be split into up to 512 <span class="math notranslate nohighlight">\(\phi\)</span> sectors (<code class="docutils literal notranslate"><span class=pre >azim_bins</span></code>, <code class="docutils literal notranslate"><span class=pre >--azim-phi-bins</span></code>), giving a <strong>2D <span class="math notranslate nohighlight">\(q\times\phi\)</span> profile</strong> that exposes azimuthal anisotropy such as detector shadowing or sample texture.</p> <section id=histogram-estimator > <h3 id=histogram-estimator ><a class=toc-backref href="#id10" role=doc-backlink >2.1 Histogram estimator</a><a class=headerlink href="#histogram-estimator" title="Link to this heading">¶</a></h3> <p>Let bin index <span class="math notranslate nohighlight">\(b(x,y)\)</span> be precomputed from <span class="math notranslate nohighlight">\(q(x,y)\)</span> (or equivalently from <span class="math notranslate nohighlight">\(d(x,y)\)</span>) and, when <span class="math notranslate nohighlight">\(\phi\)</span> sectors are enabled, the azimuth <span class="math notranslate nohighlight">\(\phi(x,y)\)</span> — so <span class="math notranslate nohighlight">\(b = b_q + b_\phi B_q\)</span>. For each bin <span class="math notranslate nohighlight">\(b\)</span>:</p> <ul class=simple > <li><p>accumulate corrected intensity and its square: <span class="math notranslate nohighlight">\( S_b = \sum_{(x,y):\,b(x,y)=b} I(x,y)\,C(x,y),\qquad S^{(2)}_b = \sum I(x,y)^2\,C(x,y)^2, \)</span></p> <li><p>and count: <span class="math notranslate nohighlight">\( N_b = \#\{(x,y):\,b(x,y)=b \text{ and pixel is valid}\}. \)</span></p> </ul> <p>The profile reports both the mean <span class="math notranslate nohighlight">\(\bar{I}_b = S_b / N_b\)</span> (when <span class="math notranslate nohighlight">\(N_b>0\)</span>) and a per-bin sample standard deviation <span class="math notranslate nohighlight">\(\sigma_b = \sqrt{(S^{(2)}_b - S_b^2/N_b)/(N_b-1)}\)</span> (a spread/error estimate for each radial point). Invalid pixels (masked, saturated, detector error codes) are excluded.</p> </section> <section id=corrections-applied > <h3 id=corrections-applied ><a class=toc-backref href="#id11" role=doc-backlink >2.2 Corrections applied</a><a class=headerlink href="#corrections-applied" title="Link to this heading">¶</a></h3> <p>Two standard corrections are available:</p> <p><strong>(i) Solid angle / geometric correction.</strong> A flat pixel’s solid angle falls off with the <strong>incidence angle <span class="math notranslate nohighlight">\(\alpha\)</span> between the scattered ray and the detector normal</strong>. With the in-plane detector offsets <span class="math notranslate nohighlight">\(u=(x-x_\mathrm{beam})p\)</span> and <span class="math notranslate nohighlight">\(v=(y-y_\mathrm{beam})p\)</span> (§1.1) and detector distance <span class="math notranslate nohighlight">\(D\)</span>, <span class="math notranslate nohighlight">\( \cos\alpha = \frac{D}{\sqrt{u^2+v^2+D^2}},\qquad C_\Omega = \cos^3\alpha, \)</span> applied — like the polarization term below — as a <strong>divisor</strong> (intensities are scaled by <span class="math notranslate nohighlight">\(1/\cos^3\alpha\)</span>), so pixels at oblique incidence, which subtend a smaller solid angle, are boosted. Because <span class="math notranslate nohighlight">\(\alpha\)</span> is evaluated in the detector’s own frame it is <strong>invariant under detector tilt</strong> (<span class="math notranslate nohighlight">\(\mathrm{rot1}/\mathrm{rot2}/\mathrm{rot3}\)</span>), matching PyFAI’s <code class="docutils literal notranslate"><span class=pre >solidAngleArray</span></code> and MAX IV azint. It reduces to the commonly quoted <span class="math notranslate nohighlight">\(\cos^3(2\theta)\)</span> form only for an untilted detector, where the incidence angle coincides with the scattering angle.</p> <p><strong>(ii) Polarization correction.</strong> With polarization coefficient <span class="math notranslate nohighlight">\(P\)</span> (beamline dependent) and azimuth <span class="math notranslate nohighlight">\(\phi\)</span>: <span class="math notranslate nohighlight">\( C_\mathrm{pol}(2\theta,\phi) = \frac{1}{2}\left(1+\cos^2(2\theta) - P\cos(2\phi)\left(1-\cos^2(2\theta)\right)\right), \)</span> applied as a divisor to intensities (i.e. scale by <span class="math notranslate nohighlight">\(1/C_\mathrm{pol}\)</span>) when enabled.</p> </section> <section id=background-estimate-for-profiles > <h3 id=background-estimate-for-profiles ><a class=toc-backref href="#id12" role=doc-backlink >2.3 Background estimate for profiles</a><a class=headerlink href="#background-estimate-for-profiles" title="Link to this heading">¶</a></h3> <p>A background estimate is derived from the profile as its mean intensity over a fixed low-to-mid <span class="math notranslate nohighlight">\(Q\)</span> window (default <span class="math notranslate nohighlight">\(2\pi/5\)</span> to <span class="math notranslate nohighlight">\(2\pi/3\)</span> Å<span class="math notranslate nohighlight">\(^{-1}\)</span>). This background is used for monitoring and diagnostics; it is <strong>not</strong> the same as the local Bragg-spot background used in summation integration (§9.2).</p> </section> </section> <hr class=docutils /> <section id=spot-finding-strong-pixels-bragg-spots > <h2 id=spot-finding-strong-pixels-bragg-spots ><a class=toc-backref href="#id13" role=doc-backlink >3. Spot finding (strong pixels → Bragg spots)</a><a class=headerlink href="#spot-finding-strong-pixels-bragg-spots" title="Link to this heading">¶</a></h2> <p>Spot finding is a two-stage process:</p> <ol class="arabic simple"> <li><p><strong>Strong-pixel selection</strong> using intensity and/or local signal-to-noise criteria.</p> <li><p><strong>Connected-component labeling (CCL)</strong> to group strong pixels into candidate spots, followed by spot-level filtering and feature extraction.</p> </ol> <section id=strong-pixel-detection-by-local-statistics > <h3 id=strong-pixel-detection-by-local-statistics ><a class=toc-backref href="#id14" role=doc-backlink >3.1 Strong-pixel detection by local statistics</a><a class=headerlink href="#strong-pixel-detection-by-local-statistics" title="Link to this heading">¶</a></h3> <p>For each pixel <span class="math notranslate nohighlight">\(i\)</span> with value <span class="math notranslate nohighlight">\(v_i\)</span>, consider a square window (nominally <span class="math notranslate nohighlight">\(31\times 31\)</span> pixels) around it. Let the window contain <span class="math notranslate nohighlight">\(n\)</span> valid pixels (excluding masked/bad/saturated), and define: <span class="math notranslate nohighlight">\( \Sigma = \sum v,\qquad \Sigma_2 = \sum v^2. \)</span></p> <p>To avoid biasing the local statistics by the test pixel itself, Jungfraujoch evaluates the pixel against the window with the pixel removed: <span class="math notranslate nohighlight">\( \Sigma' = \Sigma - v_i,\quad \Sigma_2' = \Sigma_2 - v_i^2,\quad n' = n-1. \)</span></p> <p>A variance-like quantity proportional to <span class="math notranslate nohighlight">\(n'^2\)</span> is formed: <span class="math notranslate nohighlight">\( V = n'\Sigma_2' - (\Sigma')^2, \)</span> and the deviation-from-mean quantity: <span class="math notranslate nohighlight">\( \Delta = v_i n' - \Sigma'. \)</span></p> <p>A pixel is considered strong if:</p> <ul class=simple > <li><p>it is above a photon/count threshold, and</p> <li><p>its window contains enough valid neighbours (more than 100), so the local statistics are meaningful, and</p> <li><p><span class="math notranslate nohighlight">\(\Delta>0\)</span>, and</p> <li><p>the squared deviation exceeds a scaled variance: <span class="math notranslate nohighlight">\( \Delta^2 > V\cdot T^2, \)</span> where <span class="math notranslate nohighlight">\(T\)</span> is the configured signal-to-noise threshold.</p> </ul> <p>This is equivalent to a local z-score criterion but implemented in integer arithmetic to be robust and fast.</p> <p>The test is applied in <strong>two passes</strong> over the image. The first is as described above. The second repeats it with every pixel found strong by the first excluded from the local background — it is treated exactly like a saturated pixel, so it contributes to no window it falls into and stays strong itself. This matters for any spot wide enough to reach into its own background box: on a single pass such a spot inflates the mean and variance it is then tested against, and its outer pixels fail the criterion. Excluding the core recovers them, so the spot is reported with its true extent rather than its brightest few pixels. Both the CPU and GPU implementations run these two passes and return the same spot list for the same frame.</p> <p>Special cases:</p> <ul class=simple > <li><p>saturated pixels can be forced to “strong” (useful for detecting overloaded Bragg spots),</p> <li><p>invalid pixels are never strong.</p> </ul> </section> <section id=adaptive-self-calibrating-detection > <h3 id=adaptive-self-calibrating-detection ><a class=toc-backref href="#id15" role=doc-backlink >3.2 Adaptive (self-calibrating) detection</a><a class=headerlink href="#adaptive-self-calibrating-detection" title="Link to this heading">¶</a></h3> <p>The local-statistics test above needs a fixed photon/count threshold whose correct value depends on the background level, which varies between datasets. The <strong>adaptive</strong> mode (<code class="docutils literal notranslate"><span class=pre >--adaptive-spots</span></code>; the default in <code class="docutils literal notranslate"><span class=pre >rugnux</span></code> and in the viewer for both stills and rotation data, <code class="docutils literal notranslate"><span class=pre >--no-adaptive-spots</span></code> reverts) derives that threshold from each image’s own noise, per resolution ring, so no per-dataset value is needed. It admits more spots than the fixed threshold, including genuine reflections that belong to no indexed lattice; these are down-weighted rather than filtered in the per-image geometry fit (§7.4).</p> <p>Pixels are binned into the same resolution rings as the azimuthal integrator (§2). For each ring a robust background is estimated in three passes: one plain pass over all valid pixels, then two <span class="math notranslate nohighlight">\(\sigma\)</span>-clipping passes that keep only pixels within <span class="math notranslate nohighlight">\(\pm 3\sigma\)</span> of the current ring mean (removing the Bragg peaks from the background estimate). This yields a per-ring background mean <span class="math notranslate nohighlight">\(\mu_b\)</span> and scatter <span class="math notranslate nohighlight">\(\sigma_b\)</span>.</p> <p>The ring’s detection threshold is the larger of two arms, <span class="math notranslate nohighlight">\( t_b = \max\!\big(\;\mu_b + z\,\sqrt{\sigma_b^2 + \sigma_\mathrm{read}^2}\;,\;\; k_\mathrm{Poisson}(\mu_b, p)\;\big), \)</span> where <span class="math notranslate nohighlight">\(k_\mathrm{Poisson}(\mu_b,p)\)</span> is the smallest count whose Poisson<span class="math notranslate nohighlight">\((\mu_b)\)</span> upper tail is <span class="math notranslate nohighlight">\(\le p\)</span>. The Poisson arm is correct where the background is countable (a bright low-resolution ring gets a high threshold); the Gaussian arm — floored by a detector-level excess-noise constant <span class="math notranslate nohighlight">\(\sigma_\mathrm{read}\)</span> — takes over on near-empty high-resolution rings, where the Poisson arm degenerates to “one photon is significant” and would flood. The operating point <span class="math notranslate nohighlight">\(p = E/N\)</span> is set from a single portable knob <span class="math notranslate nohighlight">\(E\)</span>, the expected number of false pixels tolerated per frame (<code class="docutils literal notranslate"><span class=pre >--spot-false-pixels</span></code>, default 100), with <span class="math notranslate nohighlight">\(N\)</span> the number of valid pixels. Because <span class="math notranslate nohighlight">\(p\)</span> and every <span class="math notranslate nohighlight">\(\mu_b,\sigma_b\)</span> come from the image itself, the same <span class="math notranslate nohighlight">\(E\)</span> lands a sensible photon threshold on strong and weak datasets alike, with no per-dataset tuning. Rings too sparse to characterise (fewer than ~40 pixels) fall back to a whole-frame background. A pixel is strong when <span class="math notranslate nohighlight">\(v_i \ge t_b\)</span> for its ring (saturated pixels are still forced strong); the strong pixels then feed the same CCL stage (§3.4). The signal-to-noise and photon-count criteria of §3.1 are not used in this mode.</p> <p>Because detection reads the pixel’s ring, a pixel that falls outside the azimuthal-integration <span class="math notranslate nohighlight">\(q\)</span> range has no ring and can never be strong: the integration range bounds what adaptive detection can see. Both upper limits are therefore optional and default to the detector itself — the azimuthal integration runs to the highest <span class="math notranslate nohighlight">\(q\)</span> any pixel of the detector reaches (<code class="docutils literal notranslate"><span class=pre >--azim-max-q</span></code> unset), and spot finding is not clipped in resolution (<code class="docutils literal notranslate"><span class=pre >--spot-high-resolution</span></code> unset), for rotation data as well as stills. Setting either one narrows detection accordingly — appropriate for weak, high-background data, where the spots admitted at the detector edge are dominated by noise.</p> <p><strong>Fused GPU engine.</strong> The per-ring reduction the adaptive threshold needs is the <em>same</em> reduction the azimuthal integrator performs. On the GPU path the two are fused into a single image pass (<code class="docutils literal notranslate"><span class=pre >AdaptiveSpotFinderGPU</span></code>): one reduction accumulates the corrected per-ring sums for the azimuthal profile (§2) <em>and</em> the raw per-ring statistics for the threshold, after which a light kernel flags the strong pixels. One GPU pass therefore replaces both the separate azimuthal-integration pass and the host-side adaptive spot-finding pass, at a small fraction of the CPU finder’s cost per frame and producing the same spot list and azimuthal profile. It is enabled by default in the offline <code class="docutils literal notranslate"><span class=pre >rugnux</span></code> path, the interactive viewer and the online receiver.</p> <p><strong>Online.</strong> <code class="docutils literal notranslate"><span class=pre >spot_finding_settings</span></code> in the REST API carries <code class="docutils literal notranslate"><span class=pre >adaptive_threshold</span></code> and <code class="docutils literal notranslate"><span class=pre >false_pixels_per_frame</span></code>, so the mode is reachable from the broker and from the web frontend as well as from <code class="docutils literal notranslate"><span class=pre >rugnux</span></code> and the viewer. It defaults to <em>off</em> online, unlike <code class="docutils literal notranslate"><span class=pre >rugnux</span></code> and the viewer, because the broker serves both workflows and only one of them can run it: spots are found in software only on the DECTRIS/SIMPLON path, while the JUNGFRAU and EIGER workflows find them on the FPGA at its own fixed threshold. Setting <code class="docutils literal notranslate"><span class=pre >adaptive_threshold</span></code> on those is refused with an error rather than accepted and ignored, so a detection setting that had no effect cannot be mistaken for one that did.</p> </section> <section id=resolution-and-ice-ring-handling > <h3 id=resolution-and-ice-ring-handling ><a class=toc-backref href="#id16" role=doc-backlink >3.3 Resolution and ice-ring handling</a><a class=headerlink href="#resolution-and-ice-ring-handling" title="Link to this heading">¶</a></h3> <p>Spot finding can be restricted to a resolution range <span class="math notranslate nohighlight">\([d_\mathrm{high}, d_\mathrm{low}]\)</span> by masking pixels outside the range. Optionally, spots in identified ice-ring regions can be tagged so that subsequent indexing/refinement may include or exclude them (see §4 and §6).</p> <p>A single per-image <strong>ice-ring score</strong> is derived from a radial profile: for each hexagonal-ice powder ring (positions <span class="math notranslate nohighlight">\(d\)</span> from Moreau <em>et al.</em>, Acta Cryst D77, 2021), the profile intensity at the ring is divided by a smooth background estimated from the <em>whole</em> profile — a running median of the non-ice bins, interpolated under each ring — and the strongest ring’s ratio is reported (1 = no ice, <span class="math notranslate nohighlight">\(>1\)</span> = ice above background). A whole-profile background is used rather than a couple of adjacent shoulder bins so the estimate is robust to the radial binning: at a coarse Q-spacing a local shoulder can be only ~1 bin and would double-count the ring’s own edge (offline processing defaults to a fine 0.01 1/Å spacing, <code class="docutils literal notranslate"><span class=pre >--azim-q-spacing</span></code>, so the rings are well resolved). The reported quantity is the ice <em>magnitude</em> rather than a significance: with many photons any real ice ring is statistically significant, so significance does not discriminate.</p> <p>The profile the score is read off is the <strong>peak-excluded</strong> one, not the plain azimuthal integration: where adaptive spot finding runs (§3.2 — the offline and viewer default), the score uses the sigma-clipped per-resolution-ring background that finder already computes for its threshold. This matters more than it sounds. A plain azimuthal profile is a per-ring <em>mean</em>, so a few strong low-resolution reflections landing in a ring’s bin raise it exactly as ice would; measured over a rotation battery, that alone ranked ice-free crystals above crystals that really are iced. An ice ring is azimuthally smooth and survives the sigma clip, while Bragg peaks do not, so on the clipped profile ice-free crystals sit near 1 and crystals with confirmed ice above 2. Only where no adaptive finder ran (the FPGA workflow) does the score fall back to the plain profile.</p> <p>The radial profile sees ice only as a <strong>smooth powder ring</strong>. Ice in large crystallites diffracts as discrete spots, leaves the profile flat, and is invisible to the score above, so a second channel is read from the spot list itself: the spots found on the ice bands are counted against the spots found in the ice-free flanks <span class="math notranslate nohighlight">\([w,2w)\)</span> either side of each band, rescaled to the bands’ own <span class="math notranslate nohighlight">\(q\)</span> width (a flank landing on another ring is dropped with its width). The indicator is the ratio pooled over the run — per image the control is a handful of spots and the ratio means nothing — and it is taken before the spot-count filter, which orders ice spots last and would discard them first. The two channels barely overlap: smooth ice reads high on the profile and ~1 on the spots, textured ice the reverse, and a clean crystal ~1 on both. Both counts are stored per image (<code class="docutils literal notranslate"><span class=pre >spot_count_ice_rings</span></code>, <code class="docutils literal notranslate"><span class=pre >spot_count_ice_control</span></code>; HDF5 <code class="docutils literal notranslate"><span class=pre >/entry/MX/peakCountIceRingRes</span></code>, <code class="docutils literal notranslate"><span class=pre >/entry/MX/peakCountIceRingControl</span></code>).</p> <p>Both channels are used offline as a <strong>gate</strong> on ice handling: unless the run reaches <code class="docutils literal notranslate"><span class=pre >--ice-min-score</span></code> (default 1.5) on the profile or <code class="docutils literal notranslate"><span class=pre >--ice-min-spot-ratio</span></code> (default 2.0) on the spots — 0 disables a channel — ice-ring flagging and the exclusion from the scale fit (§10.10) are skipped. The eleven fixed bands cover 16–26 % of the unique reflections at typical resolutions whether or not the crystal has ice, so handling ice on a clean crystal is a pure loss.</p> <p>A further optional safeguard removes isolated high-resolution “spur” spots by detecting large gaps in <span class="math notranslate nohighlight">\(1/d\)</span> (or <span class="math notranslate nohighlight">\(q\)</span>) space and discarding spots beyond the gap. This is intended for macromolecular diffraction where edge-of-detector backgrounds can be extremely low.</p> </section> <section id=connected-component-labeling-ccl > <h3 id=connected-component-labeling-ccl ><a class=toc-backref href="#id17" role=doc-backlink >3.4 Connected-component labeling (CCL)</a><a class=headerlink href="#connected-component-labeling-ccl" title="Link to this heading">¶</a></h3> <p>Strong pixels are grouped into connected components (adjacent strong pixels) using a CCL algorithm. Each component yields a candidate spot with:</p> <ul class=simple > <li><p>centroid <span class="math notranslate nohighlight">\((x,y)\)</span> (often intensity-weighted),</p> <li><p>pixel count (spot size),</p> <li><p>integrated spot intensity proxy (sum of pixel values),</p> <li><p>resolution <span class="math notranslate nohighlight">\(d\)</span> at the centroid (or mean over pixels),</p> <li><p>and quality flags (e.g. ice-ring classification).</p> </ul> <p>Spot-level filters include minimum/maximum pixel count and resolution limits.</p> <p>The host implementation (<code class="docutils literal notranslate"><span class=pre >StrongPixelSet::sparseccl</span></code>) is the SparseCCL of the ACTS/traccc project: it runs over the strong pixels sorted row-major, uses a sliding window over the previous line and a union-find whose root is each component’s lowest index. On the GPU the same labelling runs <strong>on the device</strong> (<code class="docutils literal notranslate"><span class=pre >SpotExtractorGPU</span></code>): the packed strong-pixel bitmask is compacted into that same sorted list without leaving the card, each pixel finds its at most four earlier 8-neighbours by binary search, and a lock-free union-find with path halving labels them. Only the finished spot list — a few hundred entries — comes back to the host, instead of the whole bitmask (2.26 MB per frame at 18 MP). The two implementations produce the same components, in the same order, with the same pixel counts and intensities; <code class="docutils literal notranslate"><span class=pre >tests/SpotExtractorGPUParityTest.cpp</span></code> holds them to it. The device version is also insensitive to frame content: the host sliding window becomes quadratic when many pixels light up in one detector line — a hot module, or a diffraction ring where it runs tangent to a row — which costs tens to hundreds of milliseconds on such a frame, while the device version stays under a millisecond.</p> </section> <section id=adaptive-per-image-minimum-spot-size > <h3 id=adaptive-per-image-minimum-spot-size ><a class=toc-backref href="#id18" role=doc-backlink >3.5 Adaptive per-image minimum spot size</a><a class=headerlink href="#adaptive-per-image-minimum-spot-size" title="Link to this heading">¶</a></h3> <p>The minimum-pixels-per-spot filter (§3.4) trades sensitivity against noise: a small value keeps faint one- or two-pixel spots — real signal on strong data, but detector noise on high-background frames — while a larger value keeps only well-formed spots. The best value is dataset-dependent, so for serial-stills indexing it can be chosen <strong>per image</strong> rather than fixed. The frame is indexed three times, at min-pix 3, 2 and 1, and the setting that maximises</p> <div class="math notranslate nohighlight"> \[ \frac{n_\mathrm{indexed}^2}{n_\mathrm{total}} \quad\text{(indexed-spot count weighted by indexed fraction)} \]</div> <p>is kept; the frame is then integrated once at that min-pix. The fraction factor discounts the extra spots a smaller min-pix admits <em>unless the lattice actually explains them</em>, so strong frames keep their real weak spots (extending resolution) while noise-flooded frames stay strict. Because min-pix filters the connected components <em>after</em> detection, strong-pixel detection AND the connected-component labelling both run <strong>once</strong> per frame, and the three attempts only repeat the spot-level filter; the azimuthal profile is the one that single detection pass computed. The winning attempt’s spot list is kept rather than re-extracted, so the frame that is integrated is exactly the frame that was scored. This is a <strong>stills-only, indexing-path</strong> option — rotation indexing builds one global lattice from all frames and keeps a fixed min-pix. In <code class="docutils literal notranslate"><span class=pre >rugnux</span></code> it is the default; giving an explicit <code class="docutils literal notranslate"><span class=pre >--min-pix-per-spot</span></code> pins a fixed value instead.</p> </section> <section id=predicting-the-resolution-the-merged-data-will-reach > <h3 id=predicting-the-resolution-the-merged-data-will-reach ><a class=toc-backref href="#id19" role=doc-backlink >3.6 Predicting the resolution the merged data will reach</a><a class=headerlink href="#predicting-the-resolution-the-merged-data-will-reach" title="Link to this heading">¶</a></h3> <p>A per-image <strong>resolution estimate</strong> is read off the finished spot list. It predicts how far the <em>merged</em> data will reach, not how far the furthest spot on this image lies. Each non-ice spot is weighted by <span class="math notranslate nohighlight">\(\sqrt{I}\)</span> — the intensity is a summed photon count, so <span class="math notranslate nohighlight">\(\sqrt{I}\)</span> is its Poisson significance — the <span class="math notranslate nohighlight">\(1/d^2\)</span> is found beyond which a fraction <span class="math notranslate nohighlight">\(f=0.30\)</span> of that weight lies, and the estimate is that resolution taken <span class="math notranslate nohighlight">\(2.25\times\)</span> further in <span class="math notranslate nohighlight">\(1/d\)</span>, clamped so it can never beat the corner of the detector. The dataset value is the median over images.</p> <p>Both constants carry a mechanism. A quantile from the middle of the distribution measures the <em>shape</em> of the fall-off, which is the crystal’s own <span class="math notranslate nohighlight">\(\exp(-B/2d^{2})\)</span>, where the extreme end of it measures where detection stops — a threshold that moves with the exposure and with how many reflections the unit cell puts on a frame. And merging averages many observations of each reflection, so intensities go on being measurable a fixed factor in <span class="math notranslate nohighlight">\(1/d\)</span> past the point at which one image’s spot finder still detects them; that factor is the <span class="math notranslate nohighlight">\(2.25\)</span>. Both are calibrated on rotation data against the resolution at which per-shell CC1/2 falls through 0.30, and the estimate is good to about 0.2 Å there. It is a prediction and not a measurement of what a run achieved: nothing downstream is cut on it, and it is reported alone (rugnux <code class="docutils literal notranslate"><span class=pre >SPOT_RESOLUTION_ESTIMATE</span></code>, and per image in the stream, the plots and HDF5).</p> </section> </section> <hr class=docutils /> <section id=indexing-overview > <h2 id=indexing-overview ><a class=toc-backref href="#id20" role=doc-backlink >4. Indexing overview</a><a class=headerlink href="#indexing-overview" title="Link to this heading">¶</a></h2> <p>Indexing maps observed reciprocal-space vectors <span class="math notranslate nohighlight">\(\mathbf{s}_i\)</span> to a lattice such that: <span class="math notranslate nohighlight">\( \mathbf{s}_i \approx h_i\mathbf{a}^* + k_i\mathbf{b}^* + l_i\mathbf{c}^*, \)</span> with integer <span class="math notranslate nohighlight">\((h_i,k_i,l_i)\)</span>.</p> <p>Jungfraujoch supports two complementary indexing strategies:</p> <ol class="arabic simple"> <li><p><strong>FFT-based indexing</strong> (Rossmann-type): does not require an a priori unit cell; suitable for unknown samples.</p> <li><p><strong>Fast-feedback indexing</strong> (TORO-like): requires an approximate unit cell; optimized for speed and feedback.</p> </ol> <p>Both feed into a common robust refinement/selection stage which maximizes the number of inliers under an indexing tolerance, and which can return <strong>more than one lattice</strong> per image (multi-lattice indexing; see §5.4).</p> <section id=indexed-spot-decision-inlier-test > <h3 id=indexed-spot-decision-inlier-test ><a class=toc-backref href="#id21" role=doc-backlink >4.1 Indexed-spot decision (inlier test)</a><a class=headerlink href="#indexed-spot-decision-inlier-test" title="Link to this heading">¶</a></h3> <p>Given a trial lattice with direct basis vectors <span class="math notranslate nohighlight">\(\mathbf{a},\mathbf{b},\mathbf{c}\)</span> (used here as reciprocal-space dot-test vectors), fractional indices are estimated by: <span class="math notranslate nohighlight">\( h_f = \mathbf{s}\cdot\mathbf{a},\quad k_f = \mathbf{s}\cdot\mathbf{b},\quad l_f = \mathbf{s}\cdot\mathbf{c}. \)</span> Let <span class="math notranslate nohighlight">\((h,k,l)=(\mathrm{round}(h_f),\mathrm{round}(k_f),\mathrm{round}(l_f))\)</span> and define the fractional residual: <span class="math notranslate nohighlight">\( \delta^2 = (h_f-h)^2 + (k_f-k)^2 + (l_f-l)^2. \)</span> A spot is indexed if <span class="math notranslate nohighlight">\(\delta^2 < \tau^2\)</span>, where <span class="math notranslate nohighlight">\(\tau\)</span> is the configured tolerance.</p> <p>For indexed spots, the reciprocal lattice point <span class="math notranslate nohighlight">\(\mathbf{p} = h\mathbf{a}^*+k\mathbf{b}^*+l\mathbf{c}^*\)</span> is used to compute <span class="math notranslate nohighlight">\(\Delta_\mathrm{Ewald}(\mathbf{p})\)</span> (stored as a diagnostic and later used in profile-radius estimation).</p> </section> </section> <hr class=docutils /> <section id=fft-indexing-unknown-unit-cell > <h2 id=fft-indexing-unknown-unit-cell ><a class=toc-backref href="#id22" role=doc-backlink >5. FFT indexing (unknown unit cell)</a><a class=headerlink href="#fft-indexing-unknown-unit-cell" title="Link to this heading">¶</a></h2> <p>FFT indexing follows a classical approach: detect dominant periodicities by projecting reciprocal-space points onto many directions and Fourier transforming the resulting 1D histograms.</p> <section id=directional-projections-and-histograms > <h3 id=directional-projections-and-histograms ><a class=toc-backref href="#id23" role=doc-backlink >5.1 Directional projections and histograms</a><a class=headerlink href="#directional-projections-and-histograms" title="Link to this heading">¶</a></h3> <p>Choose a set of unit vectors <span class="math notranslate nohighlight">\(\{\mathbf{u}_d\}\)</span> on a half-sphere (a near-uniform distribution generated via a golden-angle construction). For each direction <span class="math notranslate nohighlight">\(d\)</span>, form a histogram in the scalar projection: <span class="math notranslate nohighlight">\( t_{id} = \left|\mathbf{u}_d\cdot \mathbf{s}_i\right|. \)</span></p> <p>Bin width is chosen approximately as: <span class="math notranslate nohighlight">\( \Delta t \approx \frac{1}{2 L_\mathrm{max}}, \)</span> where <span class="math notranslate nohighlight">\(L_\mathrm{max}\)</span> is the maximum expected real-space unit-cell edge (Å). The histogram extent is tied to the maximum <span class="math notranslate nohighlight">\(q\)</span> used (set by a high-resolution cutoff for indexing).</p> </section> <section id=fft-peak-picking-and-candidate-vectors > <h3 id=fft-peak-picking-and-candidate-vectors ><a class=toc-backref href="#id24" role=doc-backlink >5.2 FFT peak picking and candidate vectors</a><a class=headerlink href="#fft-peak-picking-and-candidate-vectors" title="Link to this heading">¶</a></h3> <p>For each direction, the FFT magnitude spectrum is computed; peaks correspond to periodicities along <span class="math notranslate nohighlight">\(\mathbf{u}_d\)</span>. Each direction yields a candidate real-space length <span class="math notranslate nohighlight">\(L\)</span> chosen <strong>not</strong> by raw magnitude but by <strong>maximum prominence above a running-mean local background</strong> (subtracting the broad low-frequency envelope that otherwise dominates on weak or pink-beam frames), subject to <span class="math notranslate nohighlight">\(L\ge L_\mathrm{min}\)</span>.</p> <p>Candidate vectors are <span class="math notranslate nohighlight">\(\mathbf{v}_d = L_d\,\mathbf{u}_d\)</span>.</p> <p>A collinearity filter removes nearly parallel vectors (e.g. within 5°) and attempts to resolve harmonic ambiguity: shorter “fundamental” vectors may be preferred over longer harmonics if their peak magnitude is sufficiently strong relative to the dominant peak.</p> </section> <section id=lattice-reduction-and-cell-candidates > <h3 id=lattice-reduction-and-cell-candidates ><a class=toc-backref href="#id25" role=doc-backlink >5.3 Lattice reduction and cell candidates</a><a class=headerlink href="#lattice-reduction-and-cell-candidates" title="Link to this heading">¶</a></h3> <p>Triples of candidate vectors are combined to form candidate bases <span class="math notranslate nohighlight">\((\mathbf{A},\mathbf{B},\mathbf{C})\)</span>, each reduced to its <strong>Niggli-reduced cell</strong> (Gruber-vector reduction) before comparison, and filtered by allowed length and angle ranges. Two passes are run: a standard pass forms shortest-vector triples from the ~30 strongest filtered directions; if the best cell then indexes fewer than half the spots, a <strong>widened fallback</strong> anchors the two shortest axes and lets the third range over up to ~60 candidate vectors (deduplicated by Niggli cell), catching large, elongated or superstructure cells the first pass misses.</p> </section> <section id=robust-refinement-and-best-cell-selection > <h3 id=robust-refinement-and-best-cell-selection ><a class=toc-backref href="#id26" role=doc-backlink >5.4 Robust refinement and best-cell selection</a><a class=headerlink href="#robust-refinement-and-best-cell-selection" title="Link to this heading">¶</a></h3> <p>Candidate bases are refined against observed spots using an iterative inlier‑focused least‑squares procedure (trimmed/contracting threshold). Candidates are then ranked:</p> <ol class="arabic simple"> <li><p>more indexed spots wins — <strong>unless</strong> two candidates index within ~10 % of each other, in which case</p> <li><p>the <strong>smaller-volume</strong> cell is preferred (when the volumes differ by more than ~5 %), avoiding a doubled supercell, then</p> <li><p>the smaller refinement score, then the spot count again.</p> </ol> <p>Selection is <strong>not limited to a single lattice</strong>: after the best cell is accepted, further lattices are added as separate crystals provided fewer than ~40 % of their indexed spots overlap an already-accepted lattice (up to two extra by default), so split or multi-lattice crystals are indexed rather than discarded.</p> <p>An optional reference unit cell (if supplied) restricts acceptance to cells within a relative distance tolerance in edge lengths (permutation-invariant).</p> </section> </section> <hr class=docutils /> <section id=bravais-lattice-centering-inference-lattice-search > <h2 id=bravais-lattice-centering-inference-lattice-search ><a class=toc-backref href="#id27" role=doc-backlink >6. Bravais lattice / centering inference (“lattice search”)</a><a class=headerlink href="#bravais-lattice-centering-inference-lattice-search" title="Link to this heading">¶</a></h2> <p>If the space group is supplied by the user, its lattice constraints are assumed for refinement and subsequent processing.</p> <p>If not, Jungfraujoch attempts to infer the most plausible Bravais lattice type from the metric tensor after Niggli reduction:</p> <ol class="arabic simple"> <li><p><strong>Niggli reduction</strong> is performed to obtain a reduced cell in <span class="math notranslate nohighlight">\(G^6\)</span> representation (Gruber vector).</p> <li><p>The reduced cell is compared against a list of Niggli classes corresponding to Bravais lattices and centerings.</p> <li><p>The highest-symmetry class that matches within tolerances is selected (relative metric tolerance and angular tolerance).</p> </ol> <p>The output includes:</p> <ul class=simple > <li><p>a conventional cell,</p> <li><p>crystal system (triclinic, monoclinic, …),</p> <li><p>centering symbol (one of <span class="math notranslate nohighlight">\(P, C, I, F, R\)</span>; the <span class="math notranslate nohighlight">\(A/B\)</span> variants are not emitted here — they are handled only later as prediction absences, §8.4).</p> </ul> <p>This stage provides centering information used for systematic absences in prediction (§8.4) and for reporting.</p> <p><strong>A metric symmetry has to earn itself.</strong> The class is chosen from the <em>unrefined</em> candidate against a fixed angular tolerance (3°), so a lattice that is pseudo-symmetric to a few tenths of a degree is promoted a class too far — and the constraint then snaps a real angle to the ideal one, which throws nearly every reflection of every frame outside tolerance. Measured on a monoclinic crystal pseudo-C-orthorhombic to 0.42°, the promoted cell indexed 2 of 60 frames where its own primitive cell indexed 39: the same lattice, <span class="math notranslate nohighlight">\(\mathbf{b}_{oC}=-(\mathbf{a}+2\mathbf{c})\)</span>, at exactly twice the volume. A more accurate candidate is more likely to be promoted, not less: a run escapes the promotion only when the raw candidate misses the tolerance window.</p> <p>For rotation data the first pass therefore refines the constrained cell <em>and</em> an unconstrained (triclinic) one from the same spots — which it finds itself, over a sample spread across the sweep, rather than reading what the acquisition wrote — and settles the two on how many of a fixed set of validation frames each actually indexes. The bar is a clear majority, not a margin. An unconstrained refinement holds no cell parameter fixed, so it can only index at least as many frames as the constrained one, and on genuine symmetry it indexes a few more — a percentage margin therefore demotes real lattices (measured: a genuine <span class="math notranslate nohighlight">\(I\)</span>-centred orthorhombic to <span class="math notranslate nohighlight">\(P1\)</span>). Only a constrained cell that fails outright while its unconstrained cell works distinguishes a false promotion. The intensities settle the space group later regardless (§13).</p> <p><strong>Note.</strong> In ambiguous or special cases, forcing space group to <span class="math notranslate nohighlight">\(P1\)</span> (no symmetry assumptions) is recommended.</p> </section> <hr class=docutils /> <section id=geometry-and-lattice-refinement > <h2 id=geometry-and-lattice-refinement ><a class=toc-backref href="#id28" role=doc-backlink >7. Geometry and lattice refinement</a><a class=headerlink href="#geometry-and-lattice-refinement" title="Link to this heading">¶</a></h2> <p>Refinement adjusts experimental geometry and crystal parameters to minimize discrepancies between observed spot reciprocal vectors and those predicted by a lattice model with integer indices.</p> <section id=parameterization > <h3 id=parameterization ><a class=toc-backref href="#id29" role=doc-backlink >7.1 Parameterization</a><a class=headerlink href="#parameterization" title="Link to this heading">¶</a></h3> <p>The refinement jointly optimizes, depending on mode and constraints:</p> <ul class=simple > <li><p>beam center <span class="math notranslate nohighlight">\((x_\mathrm{beam}, y_\mathrm{beam})\)</span>,</p> <li><p>detector distance <span class="math notranslate nohighlight">\(D\)</span>,</p> <li><p>detector tilt angles (two-angle model; third rotation often held at 0),</p> <li><p>rotation axis direction (for rotation datasets),</p> <li><p>crystal orientation (a global rotation),</p> <li><p>unit-cell parameters, with constraints determined by inferred crystal system.</p> </ul> <p>The detector distance is not refined against one crystal’s spots at all: the positional residual leaves it degenerate with the cell scale, so it is fitted separately - by the rotation post-refinement, which holds the cell at the value its own cell/axis step settled on, and by the stills <code class="docutils literal notranslate"><span class=pre >--refine-geometry</span></code> bundle. Per image, the beam centre and the crystal orientation are refined, and the unit cell as well for stills. The first-pass rotation indexing refines the detector tilt and the rotation-axis direction too, against the spots accumulated across the sweep; everywhere else both are held fixed, because on a single crystal a tilt is absorbed almost exactly by the crystal orientation. A lighter <strong>orientation-only</strong> mode refines just the crystal orientation, for stills whose geometry is already trusted. It carries a weak small-rotation prior penalising the whole angle-axis vector (all three components, at a low weight); what it is there for is the poorly-determined out-of-plane component, which is the one the data barely constrain.</p> <p>For higher symmetries, constraints are enforced, e.g.</p> <ul class=simple > <li><p>cubic: <span class="math notranslate nohighlight">\(a=b=c,\ \alpha=\beta=\gamma=90^\circ\)</span>,</p> <li><p>tetragonal: <span class="math notranslate nohighlight">\(a=b\)</span>,</p> <li><p>hexagonal: <span class="math notranslate nohighlight">\(a=b,\ \gamma=120^\circ\)</span>,</p> <li><p>monoclinic (unique axis <span class="math notranslate nohighlight">\(b\)</span>): <span class="math notranslate nohighlight">\(\alpha=\gamma=90^\circ\)</span>, <span class="math notranslate nohighlight">\(\beta\)</span> refined.</p> </ul> </section> <section id=residuals-and-objective > <h3 id=residuals-and-objective ><a class=toc-backref href="#id30" role=doc-backlink >7.2 Residuals and objective</a><a class=headerlink href="#residuals-and-objective" title="Link to this heading">¶</a></h3> <p>For each indexed spot assigned integer <span class="math notranslate nohighlight">\((h,k,l)\)</span>, compute:</p> <ul class=simple > <li><p>observed reciprocal vector <span class="math notranslate nohighlight">\(\mathbf{s}_\mathrm{obs}\)</span> from its detector position and current geometry,</p> <li><p>predicted reciprocal vector <span class="math notranslate nohighlight">\(\mathbf{s}_\mathrm{pred}(h,k,l;\ \text{lattice params})\)</span>.</p> </ul> <p>Residual is: <span class="math notranslate nohighlight">\( \mathbf{r} = \mathbf{s}_\mathrm{obs} - \mathbf{s}_\mathrm{pred}. \)</span></p> <p>A non-linear least squares solver minimizes <span class="math notranslate nohighlight">\(\sum \|\mathbf{r}\|^2\)</span> over all selected inlier spots.</p> </section> <section id=rotation-datasets-bringing-observations-to-a-common-reference-frame > <h3 id=rotation-datasets-bringing-observations-to-a-common-reference-frame ><a class=toc-backref href="#id31" role=doc-backlink >7.3 Rotation datasets: bringing observations to a common reference frame</a><a class=headerlink href="#rotation-datasets-bringing-observations-to-a-common-reference-frame" title="Link to this heading">¶</a></h3> <p>For oscillation/rotation data, each image corresponds to a rotation angle <span class="math notranslate nohighlight">\(\phi\)</span> about an axis <span class="math notranslate nohighlight">\(\mathbf{m}_2\)</span>. Observed reciprocal vectors are rotated “back to start” so that all images are refined in a single reference crystal frame: <span class="math notranslate nohighlight">\( \mathbf{s}_\mathrm{obs,ref} = R(\phi)\,\mathbf{s}_\mathrm{obs}, \)</span> with <span class="math notranslate nohighlight">\(R(\phi)\)</span> constructed from the axis-angle representation of the goniometer model. The angle <span class="math notranslate nohighlight">\(\phi\)</span> is taken at the centre of each frame’s oscillation (the frame angle plus half the oscillation width).</p> </section> <section id=multi-stage-tightening-of-inlier-tolerance > <h3 id=multi-stage-tightening-of-inlier-tolerance ><a class=toc-backref href="#id32" role=doc-backlink >7.4 Multi-stage tightening of inlier tolerance</a><a class=headerlink href="#multi-stage-tightening-of-inlier-tolerance" title="Link to this heading">¶</a></h3> <p>Refinement is performed in stages with decreasing acceptance tolerance for including reflections (three stages, indexing tolerance <span class="math notranslate nohighlight">\(0.3\to0.2\to0.1\)</span>), which stabilizes convergence when starting from imperfect indexing and approximate geometry.</p> <p>The loose first stage necessarily admits some spots that are not reflections of this lattice — the fraction of <em>randomly</em> placed spots inside a fractional-Miller tolerance <span class="math notranslate nohighlight">\(t\)</span> is <span class="math notranslate nohighlight">\(\tfrac{4}{3}\pi t^3\)</span>, i.e. 11 % at <span class="math notranslate nohighlight">\(t=0.3\)</span> — and an unweighted fit lets them pull the orientation. Each residual is therefore weighted by how strong its spot is <strong>for its resolution</strong>: the frame’s spots are cut into equal-count resolution shells and each intensity is divided by its shell median, mapped to <span class="math notranslate nohighlight">\(w^2=r/(1+r)\)</span>. The shell normalisation is what makes this safe — genuine high-resolution spots are legitimately weaker and carry the cell and distance information, so an un-normalised intensity weight would suppress exactly the spots the fit needs. The weight is a property of the spot and never of the current residual, so it does not depend on how far the geometry is from convergence.</p> </section> <section id=rotation-geometry-post-refinement-two-pass > <h3 id=rotation-geometry-post-refinement-two-pass ><a class=toc-backref href="#id33" role=doc-backlink >7.5 Rotation geometry post-refinement (two-pass)</a><a class=headerlink href="#rotation-geometry-post-refinement-two-pass" title="Link to this heading">¶</a></h3> <p>The refinement above (§7.2) runs per image against that image’s spots. For rotation data an additional <strong>post-refinement</strong> (on by default; <code class="docutils literal notranslate"><span class=pre >--rotation-no-postrefine</span></code> disables it) improves the detector distance, beam centre and crystal cell/axis using <strong>all</strong> frames at once, then re-integrates:</p> <ol class=arabic > <li><p><strong>Pass 1</strong> integrates, scales and merges at the header geometry.</p> <li><p>From pass-1’s integrated reflections, the geometry is refined over all frames (Ceres, robust loss) in <strong>two separate steps</strong> rather than one joint fit:</p> <ul class=simple > <li><p><strong>Step A</strong>: crystal cell scale + goniometer-axis direction, from the observed rotation angles (a distance-independent excitation residual).</p> <li><p><strong>Step B</strong>: shared detector distance + beam centre, from the observed spot positions, with the cell held at step A — so the positional residual is no longer degenerate with the cell scale.</p> </ul> <p>Each step is <strong>cross-validated</strong> on a deterministic split of the <em>reflections</em> (an avalanche-mixed <span class="math notranslate nohighlight">\(hkl\)</span> hash, not a frame split and not an <span class="math notranslate nohighlight">\(h+k+l\)</span> parity, which would collide with a centering condition and leave the held-out half empty): fitted on one half, committed only if it lowers the held-out residual, otherwise left at nominal. The solver bounds the move — distance within ±5 %, beam centre within ±15 px — and detector tilt is held fixed, being gauge-coupled to the crystal orientation on a single crystal.</p> <li><p><strong>Pass 2</strong> re-indexes de novo and re-integrates at the committed geometry, reusing pass-1’s space group for the merge only. Only the <strong>detector distance and beam centre</strong> carry over: the refined cell and axis are used to make step B well-posed, but pass 2 re-indexes from scratch, so they are not propagated.</p> </ol> <p>Only pass 2 is written, as the canonical <code class="docutils literal notranslate"><span class=pre ><prefix>_*</span></code> output. Pass 1’s merge exists to choose the space group and to give the guard something to judge pass 2 against, so it stops short of the parts of the merge that only fill in a file — the correction surfaces, the twinning and radiation-damage analyses, the R-free flags and the amplitudes — and writes no merged files of its own.</p> <p><strong>Goniometer rotation scale (report only).</strong> A stage that turns further than it was commanded to leaves no trace in the file, because the stored <span class="math notranslate nohighlight">\(\omega\)</span> values <em>are</em> the commanded ones; the excess then presents as the crystal drifting, in this program and in others. Step A already measures it without a new degree of freedom: its residual rotates by <span class="math notranslate nohighlight">\(-\phi\,\mathbf{u}\)</span> with <span class="math notranslate nohighlight">\(\mathbf{u}\)</span> an <strong>unnormalised</strong> 3-vector, so <span class="math notranslate nohighlight">\(|\mathbf{u}|\)</span> is the factor by which the stage actually turned, and normalising the axis throws it away. It is reported, and warned about beyond 0.5 %, under the same cross-validation that gates the cell move — a fold that merely soaked up noise cannot raise the flag. It is a detector, not a calibration: nothing corrects the data, and it <strong>under-reads</strong> the true magnitude, because the fit only sees reflections that indexed at the nominal angle and per-frame orientation refinement has already absorbed part of the error.</p> </section> <section id=detector-geometry-from-powder-rings > <h3 id=detector-geometry-from-powder-rings ><a class=toc-backref href="#id34" role=doc-backlink >7.6 Detector geometry from powder rings</a><a class=headerlink href="#detector-geometry-from-powder-rings" title="Link to this heading">¶</a></h3> <p>Everything above fits the geometry to <em>Bragg</em> data, where the beam centre is the weakest parameter: it is gauge-coupled to the crystal orientation, which is why §7.5 restrains it toward the header value and commits only a sub-1 % move. A <strong>powder ring has no orientation to be coupled to</strong>. Where it falls on the detector depends on the geometry and on nothing else, which makes a calibrant — LaB₆, silver behenate, CeO₂, silicon — or even ice an independent constraint on exactly the quantity Bragg data cannot pin.</p> <p>The ring positions are matched to the observed rings and the geometry is refined (Ceres, five parameters: beam centre, distance, and the two detector tilts) so that the <span class="math notranslate nohighlight">\(|s|\)</span> predicted at each observed ring point matches the ring it belongs to.</p> <p><strong>Calibrants.</strong> LaB₆, silver behenate, CeO₂ and silicon are held as unit cells and their rings enumerated from them. Ice is held as the eleven <strong>measured</strong> hexagonal-ice ring positions of §3.3 instead, because hexagonal ice is <span class="math notranslate nohighlight">\(P6_3/mmc\)</span> and enumerating <span class="math notranslate nohighlight">\(hkl\)</span> from its cell would emit rings that are systematically absent. A calibrant is therefore a list of ring <span class="math notranslate nohighlight">\(q\)</span> values throughout, not a cell.</p> <p><strong>What a ring can and cannot determine.</strong> A ring is a conic centred on the beam, so a wrong centre makes its apparent radius oscillate once per turn, <span class="math notranslate nohighlight">\(r(\phi)=R+\delta_x\cos\phi+\delta_y\sin\phi\)</span>, with the <strong>same amplitude on every ring</strong>. A detector tilt <span class="math notranslate nohighlight">\(\beta\)</span> produces a <span class="math notranslate nohighlight">\(\cos\phi\)</span> term too — not the <span class="math notranslate nohighlight">\(\cos2\phi\)</span> one might expect — but one that grows as the ring’s radius <em>squared</em>, <span class="math notranslate nohighlight">\(r(\phi)=R+(R^2/F)(\beta_x\cos\phi+\beta_y\sin\phi)\)</span>; the true <span class="math notranslate nohighlight">\(\cos2\phi\)</span> term is <span class="math notranslate nohighlight">\(O(R^3\beta^2/F^2)\)</span>, hundredths of a pixel. The two are therefore separated by how the amplitude scales with radius, which needs <strong>at least two rings</strong> — on a single ring they are exactly degenerate. None of this uses the calibrant’s <span class="math notranslate nohighlight">\(d\)</span>-spacings, so the centre is determined without assuming anything about the standard.</p> <p>The <strong>distance</strong> is different: it follows from <span class="math notranslate nohighlight">\(r=F\tan2\theta\)</span> with <span class="math notranslate nohighlight">\(\sin\theta=\lambda/2d\)</span>, so a fractional error in the lattice constant passes straight into it, and the <span class="math notranslate nohighlight">\(\lambda\)</span>–<span class="math notranslate nohighlight">\(F\)</span> pair is separated only by the curvature of <span class="math notranslate nohighlight">\(\tan(2\arcsin(\lambda/2d))\)</span> across the rings — <span class="math notranslate nohighlight">\(\partial\ln r/\partial\ln F=1\)</span> at every ring against <span class="math notranslate nohighlight">\(\partial\ln r/\partial\ln\lambda=4\tan\theta/\sin4\theta\)</span>, which runs from about 1.05 at low angle to 1.43 at high. That lever collapses as the detector moves back and the rings crowd into small <span class="math notranslate nohighlight">\(2\theta\)</span>, so distance is a short-distance measurement and the wavelength is better calibrated by other means.</p> <p><strong>Reading the rings.</strong> The ring points come from one of two measurements, both accumulated over <strong>every processed image</strong> rather than one. The default reads the <strong>azimuthally-binned profile (§2) summed over the run</strong>: for each ring and each azimuthal sector, the radial peak is fitted against a locally interpolated background and the measured <span class="math notranslate nohighlight">\((q,\phi)\)</span> mapped back through the current geometry to the pixel it came from. The alternative pools the <strong>spot lists</strong>, which samples each arc wherever the spot finder’s threshold happens to bite. The accumulated profile is the same size however many images went into it; the pooled spot list is capped, each image contributing an equal share.</p> <p>A plain radial profile — one azimuthal sector — has averaged the ring over every direction and carries no centre at all, so the profile route requires at least four sectors and uses 32 by default. Sixteen to thirty-two are enough; beyond that the limit is the ring’s own texture, not counting statistics.</p> <p>The extraction window around a ring is capped at half the gap to its neighbour, because the background under a peak is taken from the ends of that window: hexagonal ice has three rings within 0.06 Å⁻¹ of one another, which a fixed window merges into a single peak. Where only one ring is in reach the two tilts are held at their input values rather than fitted, since on a single ring they are degenerate with the centre (above) and the fit would otherwise trade the centre away for them.</p> </section> </section> <hr class=docutils /> <section id=reflection-prediction > <h2 id=reflection-prediction ><a class=toc-backref href="#id35" role=doc-backlink >8. Reflection prediction</a><a class=headerlink href="#reflection-prediction" title="Link to this heading">¶</a></h2> <p>Jungfraujoch predicts reflection positions for integration by enumerating Miller indices within a resolution cutoff and accepting those that satisfy a diffraction condition model.</p> <section id=enumerating-reciprocal-lattice-points > <h3 id=enumerating-reciprocal-lattice-points ><a class=toc-backref href="#id36" role=doc-backlink >8.1 Enumerating reciprocal lattice points</a><a class=headerlink href="#enumerating-reciprocal-lattice-points" title="Link to this heading">¶</a></h3> <p>For a maximum resolution <span class="math notranslate nohighlight">\(d_\mathrm{min}\)</span>, accept <span class="math notranslate nohighlight">\((h,k,l)\)</span> such that: <span class="math notranslate nohighlight">\( \lVert \mathbf{p}(h,k,l)\rVert^2 = \lVert h\mathbf{a}^* + k\mathbf{b}^* + l\mathbf{c}^*\rVert^2 \le \left(\frac{1}{d_\mathrm{min}}\right)^2. \)</span></p> </section> <section id=still-prediction-excitation-error-cutoff > <h3 id=still-prediction-excitation-error-cutoff ><a class=toc-backref href="#id37" role=doc-backlink >8.2 Still prediction (excitation-error cutoff)</a><a class=headerlink href="#still-prediction-excitation-error-cutoff" title="Link to this heading">¶</a></h3> <p>For still images, the diffracting condition is approximated by an excitation-error cutoff: <span class="math notranslate nohighlight">\( \left|\Delta_\mathrm{Ewald}(\mathbf{p})\right| \le \Delta_\mathrm{cut}. \)</span> Accepted reflections are projected to the detector by intersecting the diffracted direction <span class="math notranslate nohighlight">\(\mathbf{S}=\mathbf{S}_0+\mathbf{p}\)</span> with the detector plane, using the current geometry.</p> <p>When the beam has a finite energy bandwidth, this window is <strong>broadened radially per reflection</strong>: the cutoff is combined in quadrature with a bandwidth smear, <span class="math notranslate nohighlight">\(\sqrt{\Delta_\mathrm{cut}^2 + (3\,\sigma_\mathrm{bw})^2}\)</span>, where <span class="math notranslate nohighlight">\(\sigma_\mathrm{bw}\propto|p_z|\)</span> (the reciprocal-space depth along the beam, growing as <span class="math notranslate nohighlight">\(\sim 1/d^2\)</span>). This keeps high-resolution reflections — smeared by the bandwidth into radial streaks — from being clipped. The same <span class="math notranslate nohighlight">\(\sigma_\mathrm{bw}\)</span> is deconvolved from the measured profile radius (§11.1), so it is not double-counted.</p> </section> <section id=rotation-prediction-laue-equation-partiality-model > <h3 id=rotation-prediction-laue-equation-partiality-model ><a class=toc-backref href="#id38" role=doc-backlink >8.3 Rotation prediction (Laue equation + partiality model)</a><a class=headerlink href="#rotation-prediction-laue-equation-partiality-model" title="Link to this heading">¶</a></h3> <p>For rotation/oscillation datasets, Jungfraujoch solves for rotation angles <span class="math notranslate nohighlight">\(\phi\)</span> where the rotated reciprocal lattice point satisfies the Ewald-sphere condition. In an XDS-like notation, define:</p> <ul class=simple > <li><p>rotation axis unit vector <span class="math notranslate nohighlight">\(\mathbf{m}_2\)</span>,</p> <li><p><span class="math notranslate nohighlight">\(\mathbf{S}_0\)</span> incident vector,</p> <li><p><span class="math notranslate nohighlight">\(\mathbf{S}(\phi)=\mathbf{S}_0+\mathbf{p}(\phi)\)</span>.</p> </ul> <p>A key quantity is: <span class="math notranslate nohighlight">\( \zeta = \left|\mathbf{m}_2\cdot \mathbf{e}_1\right|,\quad \mathbf{e}_1 = \frac{\mathbf{S}\times \mathbf{S}_0}{\lVert \mathbf{S}\times \mathbf{S}_0\rVert}, \)</span> which also appears in XDS as the Lorentz component linked to the rotation axis.</p> <p>A Gaussian mosaicity model yields a partiality fraction over an oscillation width <span class="math notranslate nohighlight">\(\Delta\phi\)</span>:</p> <p><span class="math notranslate nohighlight">\( P(\phi;\sigma_M,\zeta,\Delta\phi) = \frac{1}{2}\left[\mathrm{erf}\!\left(\frac{\phi+\Delta\phi/2}{\sqrt{2}\,\sigma_M/\zeta}\right) - \mathrm{erf}\!\left(\frac{\phi-\Delta\phi/2}{\sqrt{2}\,\sigma_M/\zeta}\right)\right], \)</span></p> <p>with mosaicity <span class="math notranslate nohighlight">\(\sigma_M\)</span> in radians.</p> <p>Reflections are predicted if they meet minimum <span class="math notranslate nohighlight">\(\zeta\)</span> and mosaicity-window criteria, and their predicted detector coordinates fall on the active detector area.</p> </section> <section id=systematic-absences-centering > <h3 id=systematic-absences-centering ><a class=toc-backref href="#id39" role=doc-backlink >8.4 Systematic absences (centering)</a><a class=headerlink href="#systematic-absences-centering" title="Link to this heading">¶</a></h3> <p>Systematic absences are applied at the centering level (prior to full space-group symmetry) <strong>when the space group is supplied by the user</strong>. With no user-fixed space group, prediction runs in <span class="math notranslate nohighlight">\(P\)</span> regardless of the centering the lattice search inferred: the centering-absent reflections are integrated so that the space-group search (§13) can confirm or disprove the centering from the measured intensities, and so that a missed superstructure shows up. For centering symbol <span class="math notranslate nohighlight">\(C\)</span>:</p> <ul class=simple > <li><p><span class="math notranslate nohighlight">\(I\)</span>: absent if <span class="math notranslate nohighlight">\(h+k+l\)</span> odd,</p> <li><p><span class="math notranslate nohighlight">\(A\)</span>: absent if <span class="math notranslate nohighlight">\(k+l\)</span> odd,</p> <li><p><span class="math notranslate nohighlight">\(B\)</span>: absent if <span class="math notranslate nohighlight">\(h+l\)</span> odd,</p> <li><p><span class="math notranslate nohighlight">\(C\)</span>: absent if <span class="math notranslate nohighlight">\(h+k\)</span> odd,</p> <li><p><span class="math notranslate nohighlight">\(F\)</span>: absent if any of <span class="math notranslate nohighlight">\(h+k, h+l, k+l\)</span> is odd,</p> <li><p><span class="math notranslate nohighlight">\(R\)</span>: absent if <span class="math notranslate nohighlight">\((-h+k+l)\bmod 3 \ne 0\)</span>,</p> <li><p><span class="math notranslate nohighlight">\(P\)</span>: no centering absences.</p> </ul> </section> </section> <hr class=docutils /> <section id=d-bragg-integration-profile-fitting-over-a-three-ring-roi > <h2 id=d-bragg-integration-profile-fitting-over-a-three-ring-roi ><a class=toc-backref href="#id40" role=doc-backlink >9. 2D Bragg integration (profile fitting over a three-ring ROI)</a><a class=headerlink href="#d-bragg-integration-profile-fitting-over-a-three-ring-roi" title="Link to this heading">¶</a></h2> <p>Jungfraujoch integrates each predicted reflection in the detector plane over a CrystFEL-inspired “three-ring” region of interest (§9.1). The <strong>default</strong> extraction is <strong>profile fitting</strong> (Kabsch; §9.3), which weights each pixel by a fitted spot profile and so recovers weak reflections far better than plain summation; plain box summation (§9.2) is retained as the seed for the profile and as a fallback. Both methods share the same ROI and background model, and emit the same per-reflection <span class="math notranslate nohighlight">\((I,\sigma,\text{partiality},d)\)</span>, so scaling, the rotation combine (§10.6) and merging consume either unchanged.</p> <section id=regions-of-interest > <h3 id=regions-of-interest ><a class=toc-backref href="#id41" role=doc-backlink >9.1 Regions of interest</a><a class=headerlink href="#regions-of-interest" title="Link to this heading">¶</a></h3> <p>For each predicted reflection at <span class="math notranslate nohighlight">\((x_p,y_p)\)</span>, define three radii:</p> <ul class=simple > <li><p><span class="math notranslate nohighlight">\(r_1\)</span>: inner signal radius,</p> <li><p><span class="math notranslate nohighlight">\(r_2\)</span>: inner background radius,</p> <li><p><span class="math notranslate nohighlight">\(r_3\)</span>: outer background radius.</p> </ul> <p>The defaults are <span class="math notranslate nohighlight">\(4,6,13\)</span> px for rotation data and <span class="math notranslate nohighlight">\(6,8,14\)</span> px for stills, which have a sparser pattern and can afford the wider ring. <code class="docutils literal notranslate"><span class=pre >--integration-radius</span></code> sets them by hand; on rotation data <span class="math notranslate nohighlight">\(r_1\)</span> is otherwise measured from the crystal’s own spots (§9.5).</p> <p>Pixels are classified by their squared distance <span class="math notranslate nohighlight">\(r^2=(x-x_p)^2+(y-y_p)^2\)</span>:</p> <ul class=simple > <li><p><strong>signal region:</strong> <span class="math notranslate nohighlight">\(r^2 < r_1^2\)</span>,</p> <li><p><strong>background annulus:</strong> <span class="math notranslate nohighlight">\(r_2^2 \le r^2 < r_3^2\)</span>.</p> </ul> <p>Invalid pixels (masked/bad/saturated) are excluded from both sums. In addition, pixels lying inside the signal disk (<span class="math notranslate nohighlight">\(r<r_2\)</span>) of any <em>other</em> predicted reflection are removed from this reflection’s background annulus, so a neighbouring spot cannot leak into the background estimate. (Both the annulus and that exclusion become ellipses when the option below is used; with it off, which is the default, they are the circles just described.)</p> <p><strong>Radially elongated background ring (opt-in, <code class="docutils literal notranslate"><span class=pre >--integration-stencil</span> <span class=pre ><k></span></code>, default 0).</strong> The three radii above are one triple for the whole run, identical for every reflection at every resolution. A reflection is not round, though: a finite bandwidth streaks it radially by <span class="math notranslate nohighlight">\(\sigma_\mathrm{bw}=\text{bandwidth}\cdot R_\mathrm{px}\)</span>. On a radially smeared spot the fixed <span class="math notranslate nohighlight">\(6\ldots13\)</span> px ring therefore sits only <span class="math notranslate nohighlight">\(\approx1.3\)</span>–<span class="math notranslate nohighlight">\(2.2\)</span> radial <span class="math notranslate nohighlight">\(\sigma\)</span> from the centre — on the reflection’s own tails, which it then measures as background.</p> <p>With <span class="math notranslate nohighlight">\(k>0\)</span> the <strong>background ring becomes an ellipse</strong>, elongated along the beam→reflection direction by <span class="math notranslate nohighlight">\(k\sigma_\mathrm{bw}\)</span>. The <strong>radial</strong> semi-axes become <span class="math notranslate nohighlight">\(r_2+k\sigma_\mathrm{bw}\)</span> and <span class="math notranslate nohighlight">\(r_3+k\sigma_\mathrm{bw}\)</span>; the <strong>tangential</strong> half-widths stay <span class="math notranslate nohighlight">\(r_2\)</span> and <span class="math notranslate nohighlight">\(r_3\)</span>; and the growth is capped at <span class="math notranslate nohighlight">\(2r_3\)</span>, which bounds what a mis-declared bandwidth can do to the bounding box. Pixels are then classified as</p> <ul class=simple > <li><p><strong>signal region:</strong> <span class="math notranslate nohighlight">\(r^2 < r_1^2\)</span> — a circle, unchanged,</p> <li><p><strong>background ring:</strong> <span class="math notranslate nohighlight">\(r^2-q_\mathrm{in}\rho^2 \ge r_2^2\)</span> <strong>and</strong> <span class="math notranslate nohighlight">\(r^2-q_\mathrm{out}\rho^2 < r_3^2\)</span>,</p> </ul> <p>where <span class="math notranslate nohighlight">\(\rho\)</span> is the pixel’s radial offset (its projection on the beam→reflection direction), <span class="math notranslate nohighlight">\(g=\min(k\sigma_\mathrm{bw},\,2r_3)\)</span> is the capped growth, and <span class="math notranslate nohighlight">\(q=1-\big(r/(r+g)\big)^2\)</span> for the boundary concerned. Written this way <span class="math notranslate nohighlight">\(k=0\)</span> gives <span class="math notranslate nohighlight">\(q=0\)</span> and both tests collapse onto <span class="math notranslate nohighlight">\(r^2\)</span> <strong>exactly in floating point</strong>, so the default classifies every pixel exactly as the circular stencil did. The neighbour exclusion above follows: each neighbour’s <strong>inner ellipse</strong>, taken in that neighbour’s own radial frame, is what is masked out of this reflection’s ring.</p> <p>The width is the bandwidth streak alone, and deliberately <strong>not</strong> the profile’s full radial variance of §9.3, which also carries the sensor parallax and weak-spot capture terms. Those two are the only terms there are on a monochromatic beam, and widening the ring by them was measured on the rotation battery: it neither helped the crystals with clean high-resolution shells nor left the weak ones alone. The bandwidth streak, by contrast, is a measured elongation of the recorded spot — principal axis along the radius to within a couple of degrees, and azimuth-independent. Keeping only it also makes the option exactly inert on a monochromatic beam, where <span class="math notranslate nohighlight">\(\sigma_\mathrm{bw}\)</span> is zero.</p> <p>Growing the ring also grows the neighbour exclusion, so on a crowded pattern fewer background pixels survive; a reflection left with too few is rejected outright. On the data this was measured on the loss is under 0.1% of reflections, but it is not structurally zero.</p> <p>Only the ring moves. The signal disk <span class="math notranslate nohighlight">\(r_1\)</span> stays circular, deliberately: it sets <span class="math notranslate nohighlight">\(n_S\)</span>, it sets <span class="math notranslate nohighlight">\(\mathrm{var}(\hat b)\)</span>, it is the domain the profile <em>width</em> is learned over (§9.3), and with <code class="docutils literal notranslate"><span class=pre >--integrator</span> <span class=pre >boxsum</span></code> it drives the all-or-nothing “every signal pixel valid” acceptance gate (§9.2), so growing it would reject any box sum carrying a single bad pixel anywhere along a long streak. In the default <code class="docutils literal notranslate"><span class=pre >gaussian</span></code> mode <span class="math notranslate nohighlight">\(r_1\)</span> does not set the intensity at all — the fit grid, <span class="math notranslate nohighlight">\(\lceil r_2\rceil\)</span>, does.</p> <p>What a circular <span class="math notranslate nohighlight">\(r_1\)</span> loses is flux, and that loss is <strong>not</strong> a function of resolution alone: measured per reflection, it carries a directional component worth several Ų with a definite principal axis, on top of the isotropic part. Nor is there anything in the merge to absorb it. There is <strong>no per-shell scale</strong>, and there cannot usefully be one: every scale in §10 is fitted against a reference built from a reflection’s own symmetry equivalents, and equivalents share <span class="math notranslate nohighlight">\(s^2\)</span> exactly, so any function of <span class="math notranslate nohighlight">\(s^2\)</span> lies in the exact null space of the whole scaling model — a per-shell parameter would have zero residual to fit against. (XDS and DIALS have the same null space, for the same reason.) The isotropic part of the loss is instead degenerate with the overall Wilson <span class="math notranslate nohighlight">\(B\)</span> and is silently reported as part of it, so <strong>the reported <code class="docutils literal notranslate"><span class=pre >WILSON_B</span></code> / <code class="docutils literal notranslate"><span class=pre >_reflns.B_iso_Wilson_estimate</span></code> carries an <span class="math notranslate nohighlight">\(r_1\)</span>-dependent contribution</strong>: measured across a constant-ring-area radius sweep it falls monotonically as the disk grows, by 0.5 Ų on sharp strong data and by up to ~10 Ų on weak wide-spot data. What this costs the <em>data</em> is much less than what it costs the flux, because most of the loss is matched by a proportional <span class="math notranslate nohighlight">\(\sigma\)</span>: it moves no CC<span class="math notranslate nohighlight">\(_{1/2}\)</span> and no <span class="math notranslate nohighlight">\(R_\text{meas}\)</span>, and — to within a few hundredths of an ångström — no resolution cut.</p> </section> <section id=box-summation-seed-and-fallback > <h3 id=box-summation-seed-and-fallback ><a class=toc-backref href="#id42" role=doc-backlink >9.2 Box summation (seed and fallback)</a><a class=headerlink href="#box-summation-seed-and-fallback" title="Link to this heading">¶</a></h3> <p>Let:</p> <ul class=simple > <li><p><span class="math notranslate nohighlight">\(S = \sum I(x,y)\)</span> over signal pixels,</p> <li><p><span class="math notranslate nohighlight">\(n_S\)</span> = number of valid signal pixels,</p> <li><p><span class="math notranslate nohighlight">\(B = \sum I(x,y)\)</span> over background pixels,</p> <li><p><span class="math notranslate nohighlight">\(n_B\)</span> = number of valid background pixels.</p> </ul> <p>Background per pixel and integrated intensity: <span class="math notranslate nohighlight">\( \hat{b} = \frac{B}{n_B},\qquad \hat{I} = S - n_S \hat{b}, \)</span> with a Poisson-like uncertainty <span class="math notranslate nohighlight">\(\sigma(\hat{I})=\max\!\big(1,\ r_\sigma\hat{I},\ \sqrt{S + n_S^2\,\mathrm{var}(\hat{b})}\big)\)</span>, i.e. <span class="math notranslate nohighlight">\(\sqrt{S}\)</span> floored both at 1 and at a small fraction <span class="math notranslate nohighlight">\(r_\sigma\)</span> of the intensity. The second term under the root is the <strong>uncertainty of the background estimate itself</strong>: <span class="math notranslate nohighlight">\(\hat b\)</span> is measured from a finite number of ring pixels, <span class="math notranslate nohighlight">\(\mathrm{var}(\hat b)=\hat b/n_B\)</span>, and it is subtracted <span class="math notranslate nohighlight">\(n_S\)</span> times over, so it enters squared. Omitting it understates <span class="math notranslate nohighlight">\(\sigma\)</span> by <span class="math notranslate nohighlight">\(\sqrt{1+n_S/n_B}\)</span> — 1.109 with the shipped circular stencil; with an elongated ring <span class="math notranslate nohighlight">\(n_B\)</span> grows with resolution, so the factor is no longer one number for a run — uniformly, on every reflection of every dataset. The same term is carried into the profile fit (§9.3), where it adds <span class="math notranslate nohighlight">\((\sum wP/\sum P^2/v)^2\,\mathrm{var}(\hat b)\)</span>; <span class="math notranslate nohighlight">\(n_B\)</span> is the count of pixels behind the <em>final</em> background value, so a clip or trim that discards ring pixels raises it. A box sum is accepted as “observed” only if all signal pixels were valid and <span class="math notranslate nohighlight">\(n_B\)</span> exceeds a minimum — it measures what is in the disk with no model of what should be there, so it cannot renormalise a disk it has lost pixels out of. The profile modes can, and do (§9.3). This box sum is the classical estimator; it is used directly with <code class="docutils literal notranslate"><span class=pre >--integrator</span> <span class=pre >boxsum</span></code>, and otherwise seeds the profile fit below, where <span class="math notranslate nohighlight">\(S\)</span> and <span class="math notranslate nohighlight">\(n_S\)</span> then count only the pixels that were actually read.</p> <p><strong>High-side clipped background (default on).</strong> Because <span class="math notranslate nohighlight">\(\hat{I}=S-n_S\hat{b}\)</span> is a small difference of large numbers for weak reflections, a per-pixel background bias <span class="math notranslate nohighlight">\(\delta\hat{b}\)</span> becomes a <em>fractional</em> intensity bias <span class="math notranslate nohighlight">\(\approx n_S\,\delta\hat{b}/\hat{I}\)</span> that grows as <span class="math notranslate nohighlight">\(\hat{I}\)</span> shrinks — worst at the resolution edge. A plain ring mean reads high there, because neighbour-spot wings that survive the signal-disk mask, tails and zingers are one-sided (positive) contaminants. The ring mean is therefore made robust: pixels above <span class="math notranslate nohighlight">\(\hat{b}+n\sqrt{\hat{b}}\)</span> are rejected and the mean recomputed, with <span class="math notranslate nohighlight">\(n=4\)</span> (<code class="docutils literal notranslate"><span class=pre >--background-clip</span></code>; <span class="math notranslate nohighlight">\(n=0\)</span> disables), lowered by <code class="docutils literal notranslate"><span class=pre >rugnux</span></code> to <span class="math notranslate nohighlight">\(n=3\)</span> on broadband (non-zero bandwidth: pink-beam / DMM) data, where a bandwidth-streaked high-resolution spot leaks into the ring more readily. That is only a default — the flag sets <span class="math notranslate nohighlight">\(n\)</span> whatever the bandwidth is. A clean Poisson ring is essentially unchanged by the cut (measured false-rejection rate 0.04–0.39 % at <span class="math notranslate nohighlight">\(4\sigma\)</span>), while a 40-pixel neighbour core at <span class="math notranslate nohighlight">\(+100\)</span> counts shifts the estimate by <span class="math notranslate nohighlight">\(+0.009\)</span> ct/px.</p> <p>The clip cuts only the high tail, which matters: the <strong>symmetric</strong> trimmed mean it replaced (drop the lowest and highest fraction <span class="math notranslate nohighlight">\(f\)</span> of ring pixels, <span class="math notranslate nohighlight">\(f=0.10\)</span>; still reachable with <code class="docutils literal notranslate"><span class=pre >--background-trim</span></code>, which switches the clip off) is <em>not</em> a consistent estimator of the mean of a right-skewed Poisson sample. It sits <span class="math notranslate nohighlight">\(\approx0.1\)</span> ct/px <strong>below</strong> the true mean at every level, and with <span class="math notranslate nohighlight">\(n_S\approx50\)</span> signal pixels in the <span class="math notranslate nohighlight">\(r_1\)</span> disk that under-estimate adds <span class="math notranslate nohighlight">\(\approx5\)</span> counts to <strong>every</strong> partial — negligible at low resolution, but a large fraction of a partial in the outermost shell. The trim also collapses once contamination exceeds <span class="math notranslate nohighlight">\(\approx10\,\%\)</span> of the ring, where the clip does not. Note that removing a positive background bias <em>lowers</em> <span class="math notranslate nohighlight">\(\langle I/\sigma\rangle\)</span> and <em>raises</em> edge <span class="math notranslate nohighlight">\(R_\text{meas}\)</span>, because both are inflated by information-free counts — so neither may be read as evidence against the change.</p> <p>Both estimators are computed in the shared background pass, but only the trim reaches plain box summation: the high-side clip is skipped for <code class="docutils literal notranslate"><span class=pre >--integrator</span> <span class=pre >boxsum</span></code>, which therefore uses the plain ring mean unless <code class="docutils literal notranslate"><span class=pre >--background-trim</span></code> is given.</p> <p><strong>Radial background correction (opt-in).</strong> A ring mean estimates the background <em>under</em> the signal disk correctly only if the background is flat there. The signal disk and the ring are concentric, so for a background that is <strong>linear</strong> in position <span class="math notranslate nohighlight">\(\langle B\rangle_\mathrm{ring}=\langle B\rangle_\mathrm{disk}\)</span> identically — a plane or gradient fit buys exactly nothing. The leading error is the <strong>curvature</strong> of the radial background, which is negligible on a smooth background but reaches tens of counts on a single reflection sitting on a sharp powder ring. That error is a kernel over radial offset, <span class="math notranslate nohighlight">\( \delta \hat b \;=\; \textstyle\sum_k \kappa_k\, \bar B(r_0+k), \)</span> with <span class="math notranslate nohighlight">\(\kappa\)</span> the annulus-minus-disk histogram of the stencil over radial offset, averaged over azimuth, and <span class="math notranslate nohighlight">\(\bar B(r)\)</span> the image’s own radial background curve. With the fixed circular stencil (<span class="math notranslate nohighlight">\(k=0\)</span>, §9.1) that single kernel serves every reflection. An elongated ring does not: its radial-offset histogram depends on how far that particular reflection’s ring was grown, so <span class="math notranslate nohighlight">\(\kappa\)</span> becomes a small table of kernels, indexed by the growth rounded to whole pixels. The azimuthal average survives the change unaltered, because the stencil is rebuilt in the reflection’s own frame at each azimuth and so stays radially aligned: what is averaged over is the sub-pixel phase of the detector grid against the radius, which is what genuinely differs between reflections. Applying it costs one short dot product per reflection and no extra pixel reads; correcting the background <em>scalar</em> means the box sum, the profile fit and the variance all pick it up. The curve is accumulated from the same annulus pixels the background pass already reads (a pixel’s radius is the reflection’s radius plus the pixel’s projection on the beam→reflection direction, so no per-pixel square root is needed) and specifically from the <strong>clipped</strong> pixels, or it would carry neighbour tails and zingers — which is why the correction is inert under <code class="docutils literal notranslate"><span class=pre >--integrator</span> <span class=pre >boxsum</span></code>, that path having no clip pass.</p> <p>The model is a function of <strong>radius alone</strong>, so it is applied only where that is true of the background. <code class="docutils literal notranslate"><span class=pre >--background-radial</span></code> takes <code class="docutils literal notranslate"><span class=pre >on</span></code>, <code class="docutils literal notranslate"><span class=pre >off</span></code> or <code class="docutils literal notranslate"><span class=pre >auto</span></code>. It is <strong>off by default</strong>; under <code class="docutils literal notranslate"><span class=pre >auto</span></code> each image’s peak-excluded ice score (§3.3) is taken after spot detection and before integration, and the correction is applied to that image when the score reaches the same <code class="docutils literal notranslate"><span class=pre >--ice-min-score</span></code> gate. Smooth powder ice <em>is</em> a radial feature and is corrected; ice made of discrete crystallite spots — which the profile channel is blind to and the spot channel catches — leaves no smooth ring to model, and correcting it makes matters worse. Measured against a fixed atomic model, comparing ice bands with resolution-matched decoy bands carrying no ice: on a crystal with pure smooth ice the correction removes <strong>43 % of the bands’ excess amplitude</strong>, and the improvement is <strong>7× larger inside the bands than outside</strong>, which is its stated mechanism; on a crystal whose ice is textured the same correction <em>increased</em> the excess amplitude by half; on a clean crystal it is inert to four decimal places. Auto engages only where a peak-excluded score exists (adaptive spot finding, §3.2) — a plain azimuthal profile carries the Bragg peaks and cannot support an absolute threshold, so without one auto leaves the correction off.</p> </section> <section id=profile-fitted-extraction-default > <h3 id=profile-fitted-extraction-default ><a class=toc-backref href="#id43" role=doc-backlink >9.3 Profile-fitted extraction (default)</a><a class=headerlink href="#profile-fitted-extraction-default" title="Link to this heading">¶</a></h3> <p>A fixed signal disk captures a <em>width-dependent</em> fraction of each spot, which puts a multiplicative floor on the per-observation precision of strong reflections and weights weak reflections poorly. Profile fitting removes this by extracting each intensity against a fitted spot shape, without needing reference intensities. Per frame:</p> <ol class=arabic > <li><p><strong>Seed.</strong> Box-sum every reflection (§9.2) to get a rough intensity and observed centroid, and select strong spots (significance <span class="math notranslate nohighlight">\(\ge 5\)</span>).</p> <li><p><strong>Build the profile.</strong> For <code class="docutils literal notranslate"><span class=pre >gaussian</span></code> (the default) the width is taken <strong>per resolution shell</strong> from the measured second moments of the strong spots (shell-dependent because spot size grows with resolution). The moments are <strong>anisotropic</strong>: each strong spot’s pixels are rotated into its <em>own</em> radial/tangential frame before being accumulated, giving <span class="math notranslate nohighlight">\(\sigma^2_r\)</span> and <span class="math notranslate nohighlight">\(\sigma^2_t\)</span> separately. Stacking the spots in the detector frame instead — they sit at every azimuth — averages the two directions away, leaving only <span class="math notranslate nohighlight">\(\sigma_r^2+\sigma_t^2\)</span>, so radial smearing is read back as a wider <em>tangential</em> spot. For <code class="docutils literal notranslate"><span class=pre >empirical</span></code> the profile is instead the averaged, background-subtracted pixel grid of the shell’s strong spots, accumulated in the detector frame on their <strong>rounded predicted</strong> positions. For <code class="docutils literal notranslate"><span class=pre >gaussian</span></code> only, the profile is then <strong>rebuilt for each reflection</strong>, centred on its <strong>sub-pixel predicted position</strong> (the noise-free geometric centre, not the observed centroid) and, where needed, <strong>elongated only along the radial direction</strong> (away from the beam centre) — because two effects stretch a spot radially but not tangentially:</p> <ul class=simple > <li><p>a finite energy <strong>bandwidth</strong> smears each spot by <span class="math notranslate nohighlight">\(\sigma_\mathrm{bw}=\text{bandwidth}\cdot R_\mathrm{px}\)</span> (<span class="math notranslate nohighlight">\(R_\mathrm{px}\)</span> = distance from the beam centre, large at high resolution), and</p> <li><p>sensor <strong>parallax</strong> — the depth over which a photon converts in a thick Si/CdTe sensor — adds a term <span class="math notranslate nohighlight">\(\propto\tan^2(2\theta)\)</span> (material- and energy-dependent), plus a small fixed weak-spot capture term.</p> </ul> <p>The two enter as a floor on the measured radial excess: <span class="math notranslate nohighlight">\(\sigma^2_\mathrm{radial}=\sigma^2_t+\max\!\left(\sigma^2_r-\sigma^2_t,\ \sigma_\mathrm{bw}^2+c_\mathrm{par}\tan^2(2\theta)\right)\)</span>, tangential unchanged at <span class="math notranslate nohighlight">\(\sigma^2_t\)</span>. The measured excess is what the signal disk can resolve; the analytic term takes over for a streak too long to be measured there. The Gaussian is built on a grid grown to hold the streak — capturing it without the tangential background an isotropic widening would add. The <code class="docutils literal notranslate"><span class=pre >empirical</span></code> profile keeps the fixed per-shell grid and gets none of this.</p> <li><p><strong>Fit (Kabsch).</strong> With profile <span class="math notranslate nohighlight">\(P\)</span>, background <span class="math notranslate nohighlight">\(B\)</span> and the shell variance model, the intensity and its uncertainty are <span class="math notranslate nohighlight">\( I = \frac{\sum P\,(c-B)/v}{\sum P^2/v},\qquad \sigma = \sqrt{\frac{1}{\sum P^2/v}},\qquad v = \max\!\left(B + I\,P,\ \tfrac{1}{2}B\right), \)</span> where <span class="math notranslate nohighlight">\(c\)</span> is the pixel value and the de-biased variance <span class="math notranslate nohighlight">\(v\)</span> (background plus model signal, rather than the down-fluctuating observed count) is iterated (a few passes). The plug-in <span class="math notranslate nohighlight">\(I\)</span> enters <strong>as it is</strong>: half-wave rectifying it, <span class="math notranslate nohighlight">\(v=B+\max(I,0)P\)</span>, lets <span class="math notranslate nohighlight">\(v\)</span> — and with it the reported <span class="math notranslate nohighlight">\(1/\sum P^2/v\)</span> — respond only to <em>upward</em> fluctuations of a noisy estimate, which adds <span class="math notranslate nohighlight">\(\approx0.4\,\sigma\sum P^3/(\sum P^2)^2\)</span> to every <span class="math notranslate nohighlight">\(\sigma\)</span> whatever the count rate. That offset is invisible on strong reflections and a large fractional inflation on weak ones; the <span class="math notranslate nohighlight">\(\tfrac12 B\)</span> clamp keeps <span class="math notranslate nohighlight">\(v\)</span> positive without reintroducing it. As a guard, if the profile intensity runs away from the box-sum seed (by more than ~10 box-sum <span class="math notranslate nohighlight">\(\sigma\)</span>) it falls back to the seed, and the background term is floored at <span class="math notranslate nohighlight">\(0.01\)</span> ct/px — enough to keep <span class="math notranslate nohighlight">\(P^2/v\)</span> finite when the ring mean reads exactly zero, which a ring of <span class="math notranslate nohighlight">\(n_B\)</span> pixels cannot distinguish from any background below <span class="math notranslate nohighlight">\(\approx1/n_B\)</span>. The rotation/excitation partiality is carried exactly as in the box-sum path.</p> </ol> <p><strong>Pixels the fit cannot use (MINPK).</strong> A profile fit is the amplitude of a <em>normalised</em> profile, so a pixel left out of the sum renormalises the estimator by construction: it costs information — <span class="math notranslate nohighlight">\(\sum P^2/v\)</span> shrinks and <span class="math notranslate nohighlight">\(\sigma\)</span> grows — but biases nothing. That is what keeps a reflection whose signal disk is cut by a mask, an untrusted region, a detector gap or an overload: those pixels are simply not read, and the fit is taken over the rest, exactly as the shared pixels of a crowded reflection are (<code class="docutils literal notranslate"><span class=pre >--overlap</span> <span class=pre >exclude</span></code>). The reflection is kept only while enough of the expected profile survives — at least <code class="docutils literal notranslate"><span class=pre >--overlap-minpk</span></code> of the profile mass that falls on the detector at all, default 0.75, which is XDS’s <code class="docutils literal notranslate"><span class=pre >MINPK</span></code> and dials’ <code class="docutils literal notranslate"><span class=pre >valid_foreground_threshold</span></code>. The complete reflections alone teach the profile, its resolution shells and their widths. <code class="docutils literal notranslate"><span class=pre >--integrator</span> <span class=pre >boxsum</span></code> has no profile to renormalise with and keeps the all-or-nothing rule of §9.2.</p> <p>“Biases nothing” holds only while the profile <em>model</em> is exact. Lose the peak and the amplitude is set by the wings alone, so the result stops being a measurement of the reflection and becomes a measurement of how well the fitted shape describes it. The worst case is a pixel invalidated <em>by the flux it saw</em> — a detector’s per-frame overload marker: that pixel goes missing <strong>because</strong> the reflection was bright, so the loss concentrates on the strong low-resolution reflections that are the largest terms of <span class="math notranslate nohighlight">\(R_\mathrm{meas}\)</span>, where the fit reads <span class="math notranslate nohighlight">\(-50\%\)</span> against the symmetry mates. MINPK cannot catch it, because it cuts on profile <em>mass</em> and the peak of a broad spot is a few percent of the mass. So a second condition applies alongside it, on any unreadable pixel whatever made it unreadable: <strong>no unreadable pixel may carry more than 0.9 of the profile’s own peak value</strong>. As a fraction of the peak rather than a radius in pixels, that scales with the spot — for a Gaussian it is a cut at <span class="math notranslate nohighlight">\(\sqrt{-2\ln f}\,\sigma = 0.46\sigma\)</span>, the peak pixel alone where <span class="math notranslate nohighlight">\(\sigma\)</span> is 0.8 px and the crest of the ridge where the profile is a bandwidth streak — and it costs well under 0.1 % of the recovered observations.</p> <p>The integrator is selected by <code class="docutils literal notranslate"><span class=pre >--integrator</span> <span class=pre >boxsum|gaussian|empirical</span></code> (default <code class="docutils literal notranslate"><span class=pre >gaussian</span></code>).</p> </section> <section id=lorentzpolarization-factor-handling > <h3 id=lorentzpolarization-factor-handling ><a class=toc-backref href="#id44" role=doc-backlink >9.4 Lorentz–polarization factor handling</a><a class=headerlink href="#lorentzpolarization-factor-handling" title="Link to this heading">¶</a></h3> <p>For integrated reflections, polarization correction can be applied as a multiplicative correction to the reflection scale via the geometry-based polarization term (§2.2). A Lorentz-like factor is carried as <code class="docutils literal notranslate"><span class=pre >rlp</span></code> in predictions, and used during scaling/merging (§10).</p> </section> <hr class=docutils /> <section id=choosing-the-signal-radius-from-the-crystals-own-spots-rotation > <h3 id=choosing-the-signal-radius-from-the-crystals-own-spots-rotation ><a class=toc-backref href="#id45" role=doc-backlink >9.5 Choosing the signal radius from the crystal’s own spots (rotation)</a><a class=headerlink href="#choosing-the-signal-radius-from-the-crystals-own-spots-rotation" title="Link to this heading">¶</a></h3> <p>The three radii are one triple for the whole run, but on rotation data they are no longer a fixed constant: <span class="math notranslate nohighlight">\(r_1\)</span> is measured from how wide <em>this</em> crystal’s spots actually are (<code class="docutils literal notranslate"><span class=pre >--adaptive-integration-radius</span></code>, on by default for rotation, off for stills, ignored when <code class="docutils literal notranslate"><span class=pre >--integration-radius</span></code> is given).</p> <p><strong>Why <span class="math notranslate nohighlight">\(r_1\)</span> matters even though it does not set the intensity.</strong> In the default <code class="docutils literal notranslate"><span class=pre >gaussian</span></code> mode the intensity is a profile-fit amplitude over the grid <span class="math notranslate nohighlight">\(\lceil r_2\rceil\)</span> (§9.3), so <span class="math notranslate nohighlight">\(r_1\)</span> is not the integration domain. It <em>is</em> the aperture the profile <strong>width</strong> is learned over, and a second moment taken over a disk of radius <span class="math notranslate nohighlight">\(a\)</span> saturates at <span class="math notranslate nohighlight">\(a^2/4\)</span>. At <span class="math notranslate nohighlight">\(r_1 = 4\)</span> the learned <span class="math notranslate nohighlight">\(\sigma\)</span> can therefore never exceed 2 px, and a crystal whose spots are broader than that is fitted with a profile the model cannot represent.</p> <p><strong>The measurement is independent of the integrator.</strong> It is made in the pre-scan, on the frames the beam-stop projection already reads, so it costs no extra frame reads and there is no feedback loop. On the spots the spot finder has already found, a spot is used only if it is clear of the detector edge and of the direct beam, has no neighbouring spot within 28 px, is one of the 40 strongest in its resolution band, sits on a fully readable disk, and reaches a signal-to-noise of 15 with its centroid within 2 px of the found position. For each surviving spot the background-subtracted <strong>encircled-flux curve</strong> is accumulated in 1-px annuli out to a fixed 14 px aperture and normalised at 8 px — an aperture that owes nothing to <span class="math notranslate nohighlight">\(r_1\)</span>, <span class="math notranslate nohighlight">\(r_2\)</span> or <span class="math notranslate nohighlight">\(r_3\)</span>.</p> <p><strong>Pooling.</strong> Spots are stratified into five resolution bands (2–3, 3–4.5, 4.5–7, 7–12, 12–30 Å), because a weak crystal’s strongest spots sit at high angle and a strong one’s at low angle. Each band with enough members contributes the radius at which its <strong>median</strong> curve reaches 0.80 of its normalised flux — <span class="math notranslate nohighlight">\(r_{80}\)</span> — at the band’s median <span class="math notranslate nohighlight">\(d\)</span>. Those points are fitted by weighted least squares against <span class="math notranslate nohighlight">\(1/d\)</span> (the mosaic contribution to the detector footprint grows as <span class="math notranslate nohighlight">\(1/d\)</span>) and evaluated at a common 5 Å, then clamped to the range the bands actually measured so the fit never extrapolates.</p> <p><strong>The radius.</strong></p> <div class="math notranslate nohighlight"> \[r_1 = \mathrm{clamp}\!\left(\mathrm{round}(2\,r_{80}),\ 4,\ 6\right),\qquad r_2 = r_1 + 2,\qquad r_3 = \sqrt{r_2^2 + 133}\]</div> <p>The factor 2 is not fitted: for a Gaussian <span class="math notranslate nohighlight">\(r_{80} = 1.794\,\sigma\)</span>, so <span class="math notranslate nohighlight">\(r_1 = 2r_{80} = 3.59\,\sigma\)</span>, where the truncated second moment recovers 0.990 of <span class="math notranslate nohighlight">\(\sigma^2\)</span>. The expression for <span class="math notranslate nohighlight">\(r_3\)</span> holds the <strong>background-ring area constant</strong> at its value for the shipped <span class="math notranslate nohighlight">\(4,6,13\)</span> (<span class="math notranslate nohighlight">\(13^2 - 6^2 = 133\)</span>) — a ring that shrank with the disk is what makes a bare <code class="docutils literal notranslate"><span class=pre >--integration-radius</span></code> worse than the default it replaces. The floor of 4 is that shipped default; the ceiling of 6 is pattern density, since <span class="math notranslate nohighlight">\(r_2\)</span> also drives the neighbour-ownership radius and the ring’s inner edge. At <span class="math notranslate nohighlight">\(r_1 = 4\)</span> the triple is bit-for-bit the shipped default, so a crystal with ordinary spots is left exactly where it was.</p> <p><strong>The sample grows until the answer settles.</strong> The frames are measured in tiers of stride 8, 4, 2, 1, each tier’s sample strictly containing the previous one, and the pooling is redone after each. Measuring stops when the new <span class="math notranslate nohighlight">\(r_{80}\)</span> is within 0.40 px of what the smaller sample said <strong>and</strong> is at least 0.25 px clear of both radii at which the rounding in <span class="math notranslate nohighlight">\(r_1\)</span> changes answer. Both conditions are load-bearing: clearance alone lets a small sample settle across a switch, and the step test alone lets it settle <em>on</em> one. Every frame of the sample is still read — the beam-stop mask and the beam centre are unchanged; what the tiers save is the decompression, preprocessing and spot finding the width measurement adds on top of the read.</p> <p><strong>It applies to the final pass only.</strong> A rotation run integrates twice (§7.5), and the widened radius is handed to the canonical second pass, not to the geometry pre-pass. The reason is that post-refinement takes its observed positions from the integrator, and an observed position is a first moment over the signal disk with the background still in it: a flat background adds nothing to the numerator but adds <span class="math notranslate nohighlight">\(n\,b\)</span> to the denominator, so every measured offset is pulled toward its prediction by <span class="math notranslate nohighlight">\(I/(I + n b)\)</span>, and <span class="math notranslate nohighlight">\(n\)</span> nearly doubles between <span class="math notranslate nohighlight">\(r_1 = 4\)</span> and <span class="math notranslate nohighlight">\(r_1 = 6\)</span>. A wider disk therefore <em>under</em>-corrects the geometry — enough, on a crystal whose metric is half a degree off orthorhombic, to flip the second pass’s de-novo Bravais choice.</p> <p><strong>The density guard.</strong> Widening <span class="math notranslate nohighlight">\(r_1\)</span> pushes <span class="math notranslate nohighlight">\(r_2\)</span>, the ring’s inner edge, into the neighbours; a reflection whose ring is left with five or fewer clean pixels has no background and is dropped whole. The integrator counts these, and separates the ones lost to <strong>neighbouring reflections</strong> from the ones lost to the detector itself (module gaps, the beam stop, the resolution mask) — a floor that reaches a couple of percent on some geometries and does not move with <span class="math notranslate nohighlight">\(r_1\)</span>. Where the neighbour-driven loss exceeds 1.13 % of the predicted reflections, the pattern is too dense for the widened radius and the final pass is integrated again at the fixed <span class="math notranslate nohighlight">\(4,6,13\)</span>, reported as pass 3 of 3 with the reason in <code class="docutils literal notranslate"><span class=pre >PASS_DECISION</span></code>.</p> <p>Every integration pass, adaptive or not, now logs the radii it used together with the fraction of predicted reflections that lost their background ring, split into the neighbour and detector parts, and the profile-fit fallback rate.</p> </section> </section> <hr class=docutils /> <section id=scaling-and-merging > <h2 id=scaling-and-merging ><a class=toc-backref href="#id46" role=doc-backlink >10. Scaling and merging</a><a class=headerlink href="#scaling-and-merging" title="Link to this heading">¶</a></h2> <p>After per-image integration, Jungfraujoch scales observations and merges them into unique reflections. The design is intentionally compatible with XDS/XSCALE concepts, and handles both still and rotation data.</p> <section id=observation-model > <h3 id=observation-model ><a class=toc-backref href="#id47" role=doc-backlink >10.1 Observation model</a><a class=headerlink href="#observation-model" title="Link to this heading">¶</a></h3> <p>For an observation <span class="math notranslate nohighlight">\(j\)</span> of a unique reflection <span class="math notranslate nohighlight">\(h\)</span> on image (or image group) <span class="math notranslate nohighlight">\(i\)</span>, the predicted measured intensity is modeled as: <span class="math notranslate nohighlight">\( I_{ij} \approx G_i \, L_{ij}\, P_{ij}\, I_h, \)</span> where:</p> <ul class=simple > <li><p><span class="math notranslate nohighlight">\(G_i\)</span> is the image scale factor,</p> <li><p><span class="math notranslate nohighlight">\(L_{ij}\)</span> is a Lorentz-like / geometry factor; predictions carry its <strong>reciprocal</strong> as <code class="docutils literal notranslate"><span class=pre >rlp</span></code>, so <span class="math notranslate nohighlight">\(L = 1/\texttt{rlp}\)</span> and the correction below is applied as a multiplication by <code class="docutils literal notranslate"><span class=pre >rlp</span></code>,</p> <li><p><span class="math notranslate nohighlight">\(P_{ij}\)</span> is a partiality term (model-dependent),</p> <li><p><span class="math notranslate nohighlight">\(I_h\)</span> is the merged (true) intensity parameter for that unique reflection.</p> </ul> <p>A least-squares objective is minimized: <span class="math notranslate nohighlight">\( \sum_{ij} \left(\frac{I_{ij}^{\mathrm{pred}} - I_{ij}^{\mathrm{obs}}}{\sigma_{ij}}\right)^2 \)</span> solved by robust (Cauchy) weighted least squares, with optional post-fit smoothing of the per-frame scales for rotation series (§10.3).</p> </section> <section id=partiality-models > <h3 id=partiality-models ><a class=toc-backref href="#id48" role=doc-backlink >10.2 Partiality models</a><a class=headerlink href="#partiality-models" title="Link to this heading">¶</a></h3> <p>The partiality applied is fixed by the data type and scaling stage, not chosen from a user menu:</p> <ol class="arabic simple"> <li><p><strong>Rotation partiality</strong> (XDS-like; see §8.3), used for the per-frame scaling of rotation partials: <span class="math notranslate nohighlight">\( P_{ij} = \frac{1}{2}\left[ \mathrm{erf}\!\left(\frac{\Delta\phi_{ij}+\Delta\phi/2}{\sqrt{2}\,\sigma_{M,i}/\zeta_{ij}}\right) - \mathrm{erf}\!\left(\frac{\Delta\phi_{ij}-\Delta\phi/2}{\sqrt{2}\,\sigma_{M,i}/\zeta_{ij}}\right) \right]. \)</span> The mosaicity <span class="math notranslate nohighlight">\(\sigma_{M,i}\)</span> is <strong>measured once per image at indexing</strong> (MLE, §11.2) and held fixed during scaling — only smoothed in frame order (§10.3), never re-refined (it is degenerate with the scale <span class="math notranslate nohighlight">\(G\)</span>; §11.2).</p> <li><p><strong>Unity</strong> (<span class="math notranslate nohighlight">\(P_{ij}=1\)</span>): used for the scale-on-fulls refit (§10.6), where each observation is already a complete reflection.</p> <li><p><strong>Fixed</strong>: use the per-reflection partiality carried from prediction. Still/serial images are predicted with <span class="math notranslate nohighlight">\(P=1\)</span>, so a single-pass stills scale is effectively unity/fixed — which is exactly what <code class="docutils literal notranslate"><span class=pre >--simple-stills</span></code> keeps. By default the stills path instead <strong>post-refines a physical partiality</strong>: a small per-crystal orientation tilt <span class="math notranslate nohighlight">\((\delta\psi_x,\delta\psi_y)\)</span> about the two axes perpendicular to the beam is refined against the running merge, and every reflection’s partiality is then recomputed analytically from the refined lattice through its excitation error <span class="math notranslate nohighlight">\(\Delta_\mathrm{Ewald}=\big|\,|\mathbf{q}+\mathbf{S}_0|-1/\lambda\,\big|\)</span> and a Gaussian width <span class="math notranslate nohighlight">\(\sigma^2=\gamma_0^2+(\gamma_e d^*)^2+(\mathrm{bw}\,|q_z|)^2\)</span> — the reciprocal-lattice point’s own radius (resolution-independent), the mosaic/divergence spread, and the bandwidth smear along the beam, in quadrature. The fit typically drives <span class="math notranslate nohighlight">\(\gamma_e\to0\)</span>, leaving the resolution-independent <span class="math notranslate nohighlight">\(\gamma_0\)</span> as the effective width. A tilt moves reflections on opposite sides of the Ewald sphere in opposite directions, so it reshapes the <em>spatial</em> pattern of partialities — a degree of freedom the per-image scale <span class="math notranslate nohighlight">\(G\)</span> does not have, and the reason the tilt is refined rather than a scalar partiality width, which would be degenerate with <span class="math notranslate nohighlight">\(G\)</span>. Nothing is re-integrated (the integrated intensities are fixed); the tilt is hard-bounded at about 1° and held by a soft prior, so it stays inert on sparse or weak crystals. The cycle is merge → per-crystal tilt refinement (with <span class="math notranslate nohighlight">\(G\)</span> profiled out by the same robust Cauchy IRLS used for the per-frame scales, §10.3) → recompute <span class="math notranslate nohighlight">\(P\)</span> → re-merge, repeated a few times.</p> </ol> <p>Reflections below a minimum partiality can be rejected from merging to avoid unstable corrections.</p> </section> <section id=smoothing-of-per-frame-scales > <h3 id=smoothing-of-per-frame-scales ><a class=toc-backref href="#id49" role=doc-backlink >10.3 Smoothing of per-frame scales</a><a class=headerlink href="#smoothing-of-per-frame-scales" title="Link to this heading">¶</a></h3> <p>The per-frame scales <span class="math notranslate nohighlight">\(G_i\)</span> are fit by robust (Cauchy) inverse-variance-weighted ratios; there is no explicit <span class="math notranslate nohighlight">\(G\approx1\)</span> prior. For rotation datasets, optional smoothing enforces the expectation that scale and mosaicity vary slowly across a sweep: <strong>after</strong> the per-frame fit, <span class="math notranslate nohighlight">\(\log G_i\)</span> (and the mosaicity) are replaced by a centred <strong>moving average</strong> over a window spanning a configurable rotation range (XDS DELPHI-like; <code class="docutils literal notranslate"><span class=pre >--smooth-g</span></code>, default 5° for rot3d, off otherwise). It is a post-fit smoothing pass, not a curvature penalty inside the least-squares objective.</p> <p>The <strong>crystal orientation</strong> is smoothed the same way, and for the same reason. Geometry is re-refined independently on every frame against that frame’s spots alone — as few as a dozen on a sparse crystal — so the per-frame orientation carries a real slow drift (crystal slippage, up to ~1.3° across a sweep) on top of fit noise that scales with spots per frame. Before scaling, the per-frame lattices are de-rotated to a common reference, averaged in frame order, rotated back, and every partial’s <span class="math notranslate nohighlight">\(\Delta\phi\)</span> — hence its partiality — is recomputed from the smoothed lattice. The window is chosen per dataset by leave-one-out cross-validation (does a frame’s neighbours predict its orientation?) rather than fixed, because drift and noise both vary by two orders of magnitude between crystals; it is capped, because the per-frame fit also absorbs a real per-frame systematic that smoothing too wide destroys. Only frames that actually indexed take part: a frame that did not carries an all-zero lattice, which is <em>finite</em> and so passes a validity check written as a finite test, and would otherwise be both averaged into its neighbours’ orientation and scored in the cross-validation that picks the window. Refining <em>less</em> is not an alternative: with per-image refinement off the space group is lost on several crystals.</p> <p>A per-frame scale enters every intensity as <span class="math notranslate nohighlight">\(1/G\)</span>, so a frame whose fit is not determined by its data can amplify it without bound — and <span class="math notranslate nohighlight">\(\sigma\)</span> is amplified by the same factor, which makes it invisible to any <span class="math notranslate nohighlight">\(\sigma\)</span>-based outlier test. A fitted <span class="math notranslate nohighlight">\(G\)</span> far below the run’s median is therefore treated as <em>undetermined</em> rather than as a successful fit, both here and in the separate refit on the combined fulls (§10.6). The bound is a ratio to the run’s own median because <span class="math notranslate nohighlight">\(G\)</span> is not gauge-fixed: it and the merged means have an exact global multiplicative degeneracy, so no absolute value is meaningful.</p> </section> <section id=merging-estimator > <h3 id=merging-estimator ><a class=toc-backref href="#id50" role=doc-backlink >10.4 Merging estimator</a><a class=headerlink href="#merging-estimator" title="Link to this heading">¶</a></h3> <p>After refinement, corrected observations are formed: <span class="math notranslate nohighlight">\( I^{\mathrm{corr}}_{ij} = \frac{I^{\mathrm{obs}}_{ij}}{G_i L_{ij} P_{ij}},\qquad \sigma^{\mathrm{corr}}_{ij} = \frac{\sigma^{\mathrm{obs}}_{ij}}{G_i L_{ij} P_{ij}}. \)</span></p> <p>Unique intensities are merged by inverse-variance weighted mean: <span class="math notranslate nohighlight">\( I_h = \frac{\sum_j w_j I^{\mathrm{corr}}_{ij}}{\sum_j w_j},\qquad w_j = \frac{1}{(\sigma^{\mathrm{corr}}_{ij})^2}. \)</span></p> <p>The weights use an <strong>expected</strong> variance: the Poisson signal part of each <span class="math notranslate nohighlight">\(\sigma^{\mathrm{corr}}_{ij}\)</span> is rebuilt at the reflection’s merged <span class="math notranslate nohighlight">\(\langle I\rangle\)</span> rather than at that observation’s own intensity. Weighting by an observation’s own <span class="math notranslate nohighlight">\(\sigma^2\)</span> biases the inverse-variance mean low below about one photon, because an up-fluctuated observation gets a larger sigma and is then down-weighted too hard. The rotation combine already does this; for stills it is on by default, and <code class="docutils literal notranslate"><span class=pre >--no-expected-variance-merge</span></code> restores the observed-sigma weighting.</p> <p>An internal-consistency term can inflate uncertainties when multiple observations are present, in the spirit of XSCALE.</p> </section> <section id=merging-statistics > <h3 id=merging-statistics ><a class=toc-backref href="#id51" role=doc-backlink >10.5 Merging statistics</a><a class=headerlink href="#merging-statistics" title="Link to this heading">¶</a></h3> <p>The shells are <strong>nine bins of equal width in <span class="math notranslate nohighlight">\(1/d^2\)</span></strong>, laid between the lowest- and the highest-resolution reflection the merge actually kept — XDS’s rule and XDS’s count, so at the same resolution limits the two programs’ tables have the same shell boundaries and can be read row for row. <code class="docutils literal notranslate"><span class=pre >--resolution-shells</span></code> changes the count; the binning rule does not change with it.</p> <p>Per-shell and overall merging statistics are computed on corrected intensities, including:</p> <ul class=simple > <li><p>number of observations and of unique reflections, and multiplicity,</p> <li><p>mean <span class="math notranslate nohighlight">\(I/\sigma(I)\)</span>,</p> <li><p><span class="math notranslate nohighlight">\(R_\mathrm{meas}\)</span> (the redundancy-independent Diederichs–Karplus form) from within‑HKL deviations,</p> <li><p><span class="math notranslate nohighlight">\(\mathrm{CC}_{1/2}\)</span> (half-set correlation) and, when a reference dataset is supplied, <span class="math notranslate nohighlight">\(\mathrm{CC}_\mathrm{ref}\)</span>,</p> <li><p>completeness against the enumerated reflections for the cell and symmetry,</p> <li><p>the anomalous signal-to-noise <span class="math notranslate nohighlight">\(\mathrm{SigAno}\)</span> (below).</p> </ul> <p>The error model is refined as <span class="math notranslate nohighlight">\(\sigma_\mathrm{corr}^2 = a\,\sigma^2 + (b\,\langle I\rangle)^2\)</span>, with <span class="math notranslate nohighlight">\(a\)</span> set by the scatter of weak (counting-limited) reflections and <span class="math notranslate nohighlight">\(b\)</span> the intensity-proportional systematic scatter of the strong ones. On the <strong>rotation</strong> path, <strong>ISa</strong> is the asymptotic (<span class="math notranslate nohighlight">\(I\to\infty\)</span>) signal-to-noise — by definition the reproducibility limit of the strongest reflections (Diederichs, <em>Acta Cryst.</em> <strong>D66</strong> (2010) 733) — and is read directly from the strong symmetry equivalents as the counting-subtracted fractional scatter of well-measured reflection groups (a robust median over strong groups; the <span class="math notranslate nohighlight">\(I/\sigma\)</span> threshold is relaxed on weak or radiation-damaged data that has few strong reflections), rather than as <span class="math notranslate nohighlight">\(1/b\)</span> of the whole-range fit, whose <span class="math notranslate nohighlight">\(b\)</span> is raised slightly by an intermediate-intensity excess and so understates the limit. The asymptotic value is <strong>report-only</strong> — nothing downstream reads it, and the merged <span class="math notranslate nohighlight">\(\sigma\)</span> is not floored at <span class="math notranslate nohighlight">\(b|I|\)</span> (that floor was removed). The per-observation <span class="math notranslate nohighlight">\(\sigma_\mathrm{corr}\)</span> (the merge weights) uses the whole-range <span class="math notranslate nohighlight">\(a,b\)</span>. The <strong>stills</strong> path has no asymptotic estimate and reports <span class="math notranslate nohighlight">\(\mathrm{ISa}=1/b\)</span> directly.</p> <p><span class="math notranslate nohighlight">\(a\)</span> and <span class="math notranslate nohighlight">\(b\)</span> are <strong>reported in XDS’s convention</strong>, which is <span class="math notranslate nohighlight">\(\sigma^2 = a(\sigma_0^2 + b I^2)\)</span> with <span class="math notranslate nohighlight">\(\mathrm{ISa}=1/\sqrt{ab}\)</span>, so the printed pair can be read straight against a <code class="docutils literal notranslate"><span class=pre >CORRECT.LP</span></code>. The internal fit keeps the form above; only the report converts, as <span class="math notranslate nohighlight">\(b_\mathrm{XDS} = b^2/a\)</span>. Note that <span class="math notranslate nohighlight">\(a\)</span> is the same in both conventions and that the two ISa expressions are the same number, <span class="math notranslate nohighlight">\(1/\sqrt{a\cdot b^2/a} = 1/b\)</span> — so the rotation log prints <strong>two</strong> ISa, the whole-range <span class="math notranslate nohighlight">\(1/b\)</span> (XDS’s meaning) and the strong-reflection asymptote beside it, which can only ever be the more optimistic of the two. The mmCIF follows the same split: <code class="docutils literal notranslate"><span class=pre >_reflns.jfjoch_diffrn_ISa</span></code> is the whole-range value, directly comparable with a <code class="docutils literal notranslate"><span class=pre >CORRECT.LP</span></code>, and the asymptote is written separately as <code class="docutils literal notranslate"><span class=pre >_reflns.jfjoch_diffrn_ISa_asymptotic</span></code>, with <code class="docutils literal notranslate"><span class=pre >_reflns.jfjoch_error_model_a</span></code> and <code class="docutils literal notranslate"><span class=pre >_b</span></code> alongside so the number can be re-derived. Note that a file written before this change carries the <em>asymptote</em> under the plain <code class="docutils literal notranslate"><span class=pre >ISa</span></code> name. A third, unrelated <span class="math notranslate nohighlight">\(b\)</span> appears in the space-group search (§13.1); it is fitted with the <span class="math notranslate nohighlight">\(\sigma^2\)</span> coefficient held at 1 and its gate constants are calibrated in that convention.</p> <p><strong>Anomalous signal-to-noise (SigAno).</strong> The strength of the anomalous signal is reported per shell and overall as <span class="math notranslate nohighlight">\(\mathrm{SigAno}=\langle|\Delta I|\rangle / \langle\sigma(\Delta I)\rangle\)</span>, where <span class="math notranslate nohighlight">\(\Delta I = I(+)-I(-)\)</span> over acentric reflections measured in both Bijvoet hands and <span class="math notranslate nohighlight">\(\sigma(\Delta I)=\sqrt{\sigma_+^2+\sigma_-^2}\)</span>. It is computed from the <strong>full-multiplicity</strong> inverse-variance <span class="math notranslate nohighlight">\(I(+)/I(-)\)</span> split (the same one written to the output), i.e. from all observations rather than a half-set. For pure noise <span class="math notranslate nohighlight">\(\mathrm{SigAno}\)</span> approaches the half-normal value <span class="math notranslate nohighlight">\(\sqrt{2/\pi}\approx0.8\)</span>, and it rises above <span class="math notranslate nohighlight">\(1\)</span> once a real anomalous difference is present. A half-set anomalous correlation (“<span class="math notranslate nohighlight">\(\mathrm{CC}_\mathrm{anom}\)</span>”) is <strong>not</strong> reported. Its two half estimates <span class="math notranslate nohighlight">\(\Delta I_0,\Delta I_1\)</span> are complementary partitions of one observation pool (<span class="math notranslate nohighlight">\(\Delta I_0+\Delta I_1=2\,\Delta I_\mathrm{full}\)</span>), and subtracting the two Bijvoet hands cancels the large common intensity that keeps <span class="math notranslate nohighlight">\(\mathrm{CC}_{1/2}\)</span> non-negative, leaving the small anomalous signal against the per-half split noise; below an anomalous signal-to-noise of <span class="math notranslate nohighlight">\(1\)</span> per half that correlation tends towards <span class="math notranslate nohighlight">\(-1\)</span> rather than <span class="math notranslate nohighlight">\(0\)</span>. <span class="math notranslate nohighlight">\(\mathrm{SigAno}\)</span> has no such floor. It is emitted only when an anomalous split was made, using the standard PDBx items <code class="docutils literal notranslate"><span class=pre >_reflns.pdbx_absDiff_over_sigma_anomalous</span></code> (overall) and <code class="docutils literal notranslate"><span class=pre >_reflns_shell.pdbx_absDiff_over_sigma_anomalous</span></code> (per shell), and appears as the <code class="docutils literal notranslate"><span class=pre >SigAno</span></code> column of the printed merge-statistics table.</p> </section> <section id=rotation-datasets-combining-partials-into-fulls-3d-integration > <h3 id=rotation-datasets-combining-partials-into-fulls-3d-integration ><a class=toc-backref href="#id52" role=doc-backlink >10.6 Rotation datasets: combining partials into fulls (3D integration)</a><a class=headerlink href="#rotation-datasets-combining-partials-into-fulls-3d-integration" title="Link to this heading">¶</a></h3> <p>In a rotation scan a reflection is recorded as a series of <em>partials</em> spread across the frames its rocking curve crosses. Merging those partials directly would force the merge error model to absorb the rocking-curve slicing as if it were measurement noise, capping the achievable <span class="math notranslate nohighlight">\(I/\sigma\)</span>. For rotation data Jungfraujoch instead <strong>combines</strong> each reflection’s partials into a single <em>full</em> intensity first, then scales and merges the fulls — a 3D integration over the rocking curve.</p> <p>The combine groups each reflection’s partials into rocking events (contiguous runs of frames) and reduces each event to one full:</p> <ul class=simple > <li><p><strong>De-biased weighted sum.</strong> Partials are combined by inverse-variance weighting, where each partial’s variance is its background-noise component plus the <em>model</em> signal shared across the event (Kabsch profile-fit form). Using the shared model signal rather than the individual down-fluctuating intensity stops weak partials from being over-weighted, which would otherwise inflate the merged error model. The weights depend on the full, so the estimate is iterated.</p> <li><p><strong>Captured fraction.</strong> The partiality summed over the event, <span class="math notranslate nohighlight">\(f=\min(1,\sum_j p_j)\)</span>, measures how completely the rocking curve was sampled. A full whose curve was captured below a threshold (<code class="docutils literal notranslate"><span class=pre >--min-captured-fraction</span></code>, default 0.7 for rotation) is dropped — an event seen over only a small fraction of its curve is unreliable however many frames it spans. (The per-partial minimum-partiality cut of §10.2 still applies upstream, in the per-frame scaling.)</p> <li><p><strong>Per-image rejection (opt-in).</strong> A frame whose observations correlate poorly with the merged reference is not measuring the crystal being merged — it may be off-crystal, or on a <em>different</em> crystal where two lattices occupy separate regions of the sample. <code class="docutils literal notranslate"><span class=pre >--min-image-cc</span></code> drops such frames. It has no default: the per-frame correlation measures data quality as much as frame validity, and its typical level varies widely between datasets, so no single absolute bound is generally valid.</p> <li><p><strong>Capture-aware uncertainty.</strong> A full captured incompletely (<span class="math notranslate nohighlight">\(f<1\)</span>) is extrapolated and biased high. The unobserved fraction is charged as an extra systematic uncertainty, <span class="math notranslate nohighlight">\(\sigma^2 \leftarrow \sigma^2 + \big(c\,(1-f)\,I\big)^2\)</span>, so the merge down-weights these extrapolated fulls and the error model treats their scatter as expected. It is enabled by default for the rotation path.</p> </ul> <p>The fulls are then re-scaled in the XDS sense — a per-image scale refit directly on the complete reflections under the unity partiality model — and merged (§10.4). Because every merged observation is now a counting-statistics-limited full rather than a partiality-divided slice, the error model reaches a far higher asymptotic <span class="math notranslate nohighlight">\(I/\sigma\)</span>.</p> <p>After scale-fulls, four <strong>correction surfaces</strong> are fitted on the combined fulls (rotation path, <strong>on by default</strong>; disable all with <code class="docutils literal notranslate"><span class=pre >--no-scaling-corrections</span></code>), each an alternating multiplicative refinement of the per-full scale against the merged reference:</p> <ul class=simple > <li><p><strong>Decay.</strong> Radiation damage weakens later frames more at higher resolution — a resolution×time (Debye–Waller) systematic the resolution-flat per-image scale cannot capture. A single global relative-<span class="math notranslate nohighlight">\(B\)</span> rate is fitted, <span class="math notranslate nohighlight">\(\ln(I_\mathrm{ref}/I_\mathrm{obs}) = 2\,(\mathrm{d}B/\mathrm{d}n)\,(n-\bar n)\,s^2\)</span> (frame <span class="math notranslate nohighlight">\(n\)</span>, <span class="math notranslate nohighlight">\(s^2 = 1/4d^2\)</span>), and folded into the scale. It engages only when the total relative-<span class="math notranslate nohighlight">\(B\)</span> over the run exceeds a physical floor (2 Ų); below that the decay is negligible and “correcting” it only spreads symmetry equivalents (same <span class="math notranslate nohighlight">\(s^2\)</span>, different frames). An optional <strong>per-batch relative-<span class="math notranslate nohighlight">\(B\)</span></strong> (<code class="docutils literal notranslate"><span class=pre >--relative-b[=deg]</span></code>, off unless requested; 10°-of-rotation batches by default) extends the single global rate to a smooth <span class="math notranslate nohighlight">\(B(n)\)</span> curve — the same <span class="math notranslate nohighlight">\(s^2\)</span>-weighted decay fit solved independently over short frame batches, curvature-penalized so it cannot over-fit and cross-validated like the surfaces below — for crystals whose decay is non-linear in dose. Its cross-validation splits on <strong>ASU-group parity</strong>, not the frame parity the surfaces below use: a per-batch parameter owns whole frames and so cannot be scored on a held-out frame, whereas splitting the symmetry equivalents tests whether a batch’s <span class="math notranslate nohighlight">\(B\)</span> generalises to reflections it was not fitted on.</p> <li><p><strong>Absorption.</strong> A smooth multiplicative factor over the diffracted-beam direction expressed in the goniometer (crystal) frame: each full’s predicted detector position gives the lab diffracted direction, de-rotated by the spindle so a fixed crystal-frame direction is sampled at many rotation angles and its grid cell is well-determined. Negligible at hard X-rays / thin crystals; it matters at low photon energy.</p> <li><p><strong>Modulation</strong> (detector-plane flat-field). A smooth multiplicative factor over where each reflection lands on the detector (predicted <span class="math notranslate nohighlight">\(x,y\)</span>): symmetry-equivalents land at different positions as the crystal rotates, over-determining the surface. It absorbs detector-response and geometric systematics that inflate <span class="math notranslate nohighlight">\(R_\mathrm{meas}\)</span>.</p> <li><p><strong>Time-dependent absorption.</strong> The same surface as <em>Absorption</em>, but over (rotation angle × detector position) instead of the crystal-frame direction alone. The two agree while the illuminated volume stays put — the incident path then depends only on the spindle angle, which the per-image scale already takes, and the exit path is fixed in the crystal frame. Once the diffracting volume drifts through the beam the exit path becomes a function of the spindle angle as well, and nothing time-independent describes it. Fitted last, on 12 rotation bins × a 10×10 equal-occupancy detector grid, so the two time-independent surfaces get first claim on what they can explain.</p> </ul> <p>Each surface is <strong>cross-validated</strong>: fitted on even-numbered frames and kept only if it improves the held-out odd-frame agreement by a clear margin (and vice versa), scored by a <strong>σ-independent, <span class="math notranslate nohighlight">\(R_\mathrm{meas}\)</span>-like</strong> fractional agreement <span class="math notranslate nohighlight">\(\sum|I_s-I_\mathrm{ref}|/\sum|I_\mathrm{ref}|\)</span> rather than a studentized <span class="math notranslate nohighlight">\(\chi^2\)</span> — so a surface cannot “pass” by reshaping the sigmas instead of tightening the intensities. A surface fitted to noise where its systematic is absent does not generalize and is discarded — a correction never adds scatter.</p> <p><strong>Radiation-damage report (rotation, report-only).</strong> Independently of whether any decay correction is applied, rugnux measures and reports the relative Debye–Waller <span class="math notranslate nohighlight">\(B\)</span> across the sweep: the per-image scale’s correlation to the merge and the per-image mosaicity versus frame (dose), together with a per-batch relative-<span class="math notranslate nohighlight">\(B\)</span> curve whose first→last change is a single headline number (measured before any decay correction, against the least-damaged early wedge). It is written to the log and to the merged mmCIF as a data-quality-vs-dose diagnostic and <strong>never</strong> alters the merged intensities — distinct from the decay correction above, which does fold into the scale.</p> <p>Each batch’s <span class="math notranslate nohighlight">\(B\)</span> is fitted on <strong>resolution-shell means</strong>, not on single observations: <span class="math notranslate nohighlight">\(\ln(I_\mathrm{ref}/I_\mathrm{obs})\)</span> of one weak observation is unbounded and biased downwards — the observation appears in the response and in its own weight, and the logarithm needs <span class="math notranslate nohighlight">\(I_\mathrm{obs} > 0\)</span>, which keeps only the upward half of the noise — and on decayed data that bias grows with dose until it reverses the sign of the answer. The shells are laid inside the range the run actually diffracted to, and the fit carries an intercept as well as a slope, so a batch that is merely <em>dimmer</em> than the run (an attenuated beam, a mis-fitted frame scale) is not reported as damage. A batch whose shells are too weak to fit, or whose solved value reaches the bound the smoothing solve clamps to, is reported as <strong>absent</strong> rather than as a number. The first→last headline is reported only where a straight line describes the curve: radiation damage is progressive, so a curve that dips and recovers is a disturbance rather than dose, and is left to the sweep-quality report (<code class="docutils literal notranslate"><span class=pre >docs/RUGNUX.md</span></code>) to name.</p> </section> <section id=r-free-test-set-flags > <h3 id=r-free-test-set-flags ><a class=toc-backref href="#id53" role=doc-backlink >10.7 R-free test-set flags</a><a class=headerlink href="#r-free-test-set-flags" title="Link to this heading">¶</a></h3> <p>A fraction of the unique reflections (<code class="docutils literal notranslate"><span class=pre >rfree_fraction</span></code>, default 0.05) is flagged as a <strong>free (test) set</strong>, written to the output (MTZ <code class="docutils literal notranslate"><span class=pre >FreeR_flag</span></code>, mmCIF <code class="docutils literal notranslate"><span class=pre >_refln.status_free</span></code>, a text-HKL column) for model validation (§14) and for downstream refinement. The flag is a pure function of the reflection’s <strong>Friedel-merged (Laue) ASU index</strong>, which gives three properties:</p> <ul class=simple > <li><p>all symmetry- and Friedel-equivalent reflections share one flag — in particular a Bijvoet pair <span class="math notranslate nohighlight">\(I(+)/I(-)\)</span>, kept as two separate merged rows in anomalous mode, is <strong>never split</strong> across the work and free sets (which would bias R-free);</p> <li><p>the free/work decision is a deterministic hash of that key, so the same reflection always lands in the same set — reproducible run-to-run and independent of the order in which observations were merged;</p> <li><p>the hash depends only on the reflection index, <strong>not</strong> on this dataset’s resolution range or which reflections it happens to contain, so a uniform draw takes ~<code class="docutils literal notranslate"><span class=pre >rfree_fraction</span></code> of the distinct reflections free and — crucially — <strong>every dataset of one crystal form gets the same free set</strong>. That cross-dataset consistency is what a multi-dataset campaign (ensemble refinement, PanDDA) requires; a per-shell stratification tied to each dataset’s own <span class="math notranslate nohighlight">\(d_\mathrm{min}\)</span> would break it.</p> </ul> <p>On small data, where <code class="docutils literal notranslate"><span class=pre >rfree_fraction</span></code> (default 0.05) would give too few test reflections for a statistically stable R-free (Brünger’s ~500–2000 rule), the fraction is <strong>floored</strong> so at least ~500 distinct reflections are free — capped at 10 % so a large test set never steals working data. For ordinary data this floor is inactive and the fraction stays flat at <code class="docutils literal notranslate"><span class=pre >rfree_fraction</span></code>, preserving the cross-dataset-identical property above; it only lifts the fraction on genuinely small datasets, where per-dataset R-free stability outweighs cross-dataset identity (and a shared reference set is the way to keep exact identity there).</p> <p>When a reference MTZ (<code class="docutils literal notranslate"><span class=pre >--reference-mtz</span></code>) carries a <code class="docutils literal notranslate"><span class=pre >FreeR_flag</span></code> column, its test set is <strong>imported</strong> instead: every merged reflection whose Laue-ASU index matches the reference takes the reference’s flag (reflections absent from the reference keep the hash flag). This lets a whole fragment-screening campaign inherit one shared free set from the apo/reference dataset. The CCP4/refmac convention (test set = flag 0, including the historical 0–19 form) is assumed, with the complement taken automatically if flag 0 would be the majority (a phenix-style file where 1 marks free).</p> </section> <section id=frenchwilson-amplitudes > <h3 id=frenchwilson-amplitudes ><a class=toc-backref href="#id54" role=doc-backlink >10.8 French–Wilson amplitudes</a><a class=headerlink href="#frenchwilson-amplitudes" title="Link to this heading">¶</a></h3> <p>The last step of the merge estimates a Bayesian structure-factor amplitude <span class="math notranslate nohighlight">\(|F|\)</span> for each unique reflection from its intensity <span class="math notranslate nohighlight">\(I\)</span> and error <span class="math notranslate nohighlight">\(\sigma\)</span>, so the output carries amplitudes alongside intensities (a naïve <span class="math notranslate nohighlight">\(\sqrt{\max(I,0)}\)</span> turns every weak or negative measurement into a biased — or zero — amplitude). With the Wilson prior for the true intensity <span class="math notranslate nohighlight">\(J\ge 0\)</span> at that resolution,</p> <p><span class="math notranslate nohighlight">\( P_\mathrm{acentric}(J) \propto e^{-J/\Sigma},\qquad P_\mathrm{centric}(J) \propto J^{-1/2}\,e^{-J/2\Sigma}, \)</span></p> <p>and a Gaussian likelihood <span class="math notranslate nohighlight">\(\mathcal{N}(I;J,\sigma^2)\)</span>, the posterior mean amplitude and its uncertainty are</p> <p><span class="math notranslate nohighlight">\( \langle |F|\rangle = \frac{\int_0^\infty \sqrt{J}\,\mathcal{N}(I;J,\sigma^2)\,P(J)\,\mathrm{d}J}{\int_0^\infty \mathcal{N}(I;J,\sigma^2)\,P(J)\,\mathrm{d}J},\qquad \sigma_F = \sqrt{\langle J\rangle - \langle|F|\rangle^2}. \)</span></p> <p>The prior mean is <span class="math notranslate nohighlight">\(\Sigma = \varepsilon\,\langle I/\varepsilon\rangle_\mathrm{shell}\)</span>, where <span class="math notranslate nohighlight">\(\varepsilon\)</span> is the reflection’s epsilon (symmetry-enhancement) multiplicity and <span class="math notranslate nohighlight">\(\langle I/\varepsilon\rangle\)</span> is the Wilson mean in its resolution shell (so reflections on symmetry elements, and each shell, are treated correctly). Strong reflections (<span class="math notranslate nohighlight">\(I>4\sigma\)</span>) short-circuit to <span class="math notranslate nohighlight">\(|F|=\sqrt{I}\)</span>, where the French–Wilson bias is negligible; a reflection with an unusable <span class="math notranslate nohighlight">\(I/\sigma\)</span> falls back to <span class="math notranslate nohighlight">\(\sqrt{\max(I,0)}\)</span>. The integral is evaluated numerically with a log-shift for stability.</p> <p>Amplitudes are written as MTZ <code class="docutils literal notranslate"><span class=pre >F</span></code>/<code class="docutils literal notranslate"><span class=pre >SIGF</span></code>, mmCIF <code class="docutils literal notranslate"><span class=pre >_refln.F_meas_au</span></code>/<code class="docutils literal notranslate"><span class=pre >F_meas_sigma_au</span></code>, and appended to the text HKL, alongside the intensity columns. The <strong>same</strong> <span class="math notranslate nohighlight">\(|F|\)</span> feed the model-validation step (§14), so the reflection file and the maps use one consistent set of amplitudes.</p> </section> <section id=reference-data-fixing-the-space-group-and-resolving-the-indexing-ambiguity > <h3 id=reference-data-fixing-the-space-group-and-resolving-the-indexing-ambiguity ><a class=toc-backref href="#id55" role=doc-backlink >10.9 Reference data: fixing the space group and resolving the indexing ambiguity</a><a class=headerlink href="#reference-data-fixing-the-space-group-and-resolving-the-indexing-ambiguity" title="Link to this heading">¶</a></h3> <p>A reference dataset (<code class="docutils literal notranslate"><span class=pre >--reference-mtz</span></code>) supplies known intensities for the same crystal form, and is used in two ways.</p> <p><strong>Fix the space group and cell.</strong> Unless overridden on the command line (<code class="docutils literal notranslate"><span class=pre >-S</span></code> for the space group, <code class="docutils literal notranslate"><span class=pre >-C</span></code> for the cell), the reference’s space group is adopted and its cell is used as the soft reference cell — indexing may still drift the cell within tolerance, so a small mismatch between reference and data is absorbed rather than rejected. This applies to both stills and rotation data.</p> <p><strong>Resolve the indexing (merohedral) ambiguity.</strong> When the lattice symmetry is higher than the crystal’s Laue symmetry (e.g. <span class="math notranslate nohighlight">\(P3\)</span>, <span class="math notranslate nohighlight">\(P4\)</span>, <span class="math notranslate nohighlight">\(P6\)</span>, <span class="math notranslate nohighlight">\(C2\)</span>), more than one indexing of the same lattice is geometrically valid, and the two solutions produce <em>different</em> merged intensities that a self-consistent scale cannot tell apart — only an external reference can. The candidate reindexings are the identity together with the twin-law cosets of the metric symmetry (from the unit-cell metric and the Laue group); each is scored by the intensity correlation <span class="math notranslate nohighlight">\(\mathrm{CC}_\mathrm{ref}\)</span> of the reindexed merge against the reference, and the data are re-merged in the best-correlating indexing. The reindex is <strong>metric-preserving</strong> — only the <span class="math notranslate nohighlight">\(hkl\)</span> labels change, the cell is unchanged — and it is a no-op for a holohedral crystal, which has no twin laws (the lattice and Laue symmetry coincide). For rotation data this is done once, after the space group is determined, and the whole merge is then repeated in the chosen indexing. For stills it has to be done <strong>per image</strong>, at integration time: each crystal is indexed independently, so a run resolves the ambiguity image by image (the image’s partiality/Lorentz-corrected intensities are correlated with the reference under each candidate operator, which is scale-invariant, and the best-correlating one is adopted once and for good) — otherwise the merge would average reflections that are not symmetry mates. In neither workflow is the reference a <strong>scale</strong> target: both scale against their own data (§10.2), so <span class="math notranslate nohighlight">\(\mathrm{ISa}\)</span> and the merging statistics come from the data alone and no cross-dataset systematic is imported. Because the stills choice is made at integration time, a later re-merge of stored reflections cannot repair a dataset integrated without a reference. Where there is no reference dataset but there is a <strong>model</strong> (<code class="docutils literal notranslate"><span class=pre >--model</span></code>), the reference intensities are computed from it instead - <span class="math notranslate nohighlight">\(|F_\mathrm{model}|^2\)</span> from the atomic structure factors with a flat bulk-solvent contribution at the standard constants (<span class="math notranslate nohighlight">\(k_\mathrm{sol}=0.35\)</span>, <span class="math notranslate nohighlight">\(B_\mathrm{sol}=46\)</span> Å<span class="math notranslate nohighlight">\(^2\)</span>), which are not fitted because there are no observations yet. Nothing is scaled against them; they serve only to rank the candidate indexings, and the correlation that does the ranking is scale-invariant. This needs the cell and space group up front (<code class="docutils literal notranslate"><span class=pre >-C</span></code> / <code class="docutils literal notranslate"><span class=pre >-S</span></code>, as serial indexing wants anyway); on rotation data the same job is done after the merge, in §14.5, where a merge exists to fit the model to properly.</p> </section> <section id=ice-rings-at-the-scale-and-merge-stages > <h3 id=ice-rings-at-the-scale-and-merge-stages ><a class=toc-backref href="#id56" role=doc-backlink >10.10 Ice rings at the scale and merge stages</a><a class=headerlink href="#ice-rings-at-the-scale-and-merge-stages" title="Link to this heading">¶</a></h3> <p>Where the gate of §3.3 has found ice, reflections falling within <span class="math notranslate nohighlight">\(\pm w\)</span> in <span class="math notranslate nohighlight">\(q\)</span> of a hexagonal-ice band (<span class="math notranslate nohighlight">\(w=0.03\)</span> Å<span class="math notranslate nohighlight">\(^{-1}\)</span> offline, about the measured ring half-width) are marked. Marked reflections are <strong>excluded where a model is fitted</strong> — the per-frame scale <span class="math notranslate nohighlight">\(G\)</span>, the per-image correlation, and the <span class="math notranslate nohighlight">\(P1\)</span> merge the space-group search runs on — because ice contamination is a <em>positive bias</em>, not extra scatter, and a least-squares scale absorbs it into <span class="math notranslate nohighlight">\(G\)</span> and into the error-model <span class="math notranslate nohighlight">\(b\)</span>, where it damages every other reflection on the same frame. They are <strong>kept in the final merge</strong>, which is also what the established scaling programs do by default, so the affected shells keep their completeness.</p> <p>Nothing on an ice band is deleted from the merged output. Deleting the bands was implemented, measured against an external arbiter rather than against the merge’s own statistics, and removed: on the one rotation-battery crystal where a band was both dead by its own merged <span class="math notranslate nohighlight">\(\mathrm{CC}_{1/2}\)</span> and scorable by anomalous peak height, dropping it changed the mean anomalous density at the known sites by <span class="math notranslate nohighlight">\(-0.001\pm0.018\,\sigma\)</span> — about 2 % of the site height — while removing 1149 unique reflections whose mean <span class="math notranslate nohighlight">\(I/\sigma\)</span> was 3.62 against the dataset’s own 3.05, i.e. better-than-average data, and costing 6 to 8 points of completeness in the affected shell.</p> </section> </section> <hr class=docutils /> <section id=mosaicity-and-profile-radius-monitoring > <h2 id=mosaicity-and-profile-radius-monitoring ><a class=toc-backref href="#id57" role=doc-backlink >11. Mosaicity and “profile radius” monitoring</a><a class=headerlink href="#mosaicity-and-profile-radius-monitoring" title="Link to this heading">¶</a></h2> <section id=profile-radius-intrinsic-excitation-error-width > <h3 id=profile-radius-intrinsic-excitation-error-width ><a class=toc-backref href="#id58" role=doc-backlink >11.1 Profile radius (intrinsic excitation-error width)</a><a class=headerlink href="#profile-radius-intrinsic-excitation-error-width" title="Link to this heading">¶</a></h3> <p>The “profile radius” is the intrinsic angular width of a reflection — crystal mosaicity plus beam divergence — estimated from the spread of <span class="math notranslate nohighlight">\(\Delta_\mathrm{Ewald}\)</span> over indexed spots, <span class="math notranslate nohighlight">\( R \approx \sqrt{\tfrac{1}{N}\sum_i \Delta_{\mathrm{Ewald},i}^2}. \)</span> When the beam has a finite energy bandwidth, that bandwidth smears each reflection radially by <span class="math notranslate nohighlight">\(\sigma_\mathrm{bw}\approx \mathrm{bandwidth}\cdot\lambda/2d^2\)</span> (largest at high resolution), which also broadens the measured <span class="math notranslate nohighlight">\(\Delta_\mathrm{Ewald}\)</span> spread. Since prediction re-applies the bandwidth term per reflection (§8.2), this contribution is deconvolved from the estimate — <span class="math notranslate nohighlight">\(R^2 = \langle\Delta_\mathrm{Ewald}^2\rangle - \langle\sigma_\mathrm{bw}^2\rangle\)</span> — so that <span class="math notranslate nohighlight">\(R\)</span> is the intrinsic width and bandwidth is not double-counted. Still predictions use an excitation-error cutoff proportional to <span class="math notranslate nohighlight">\(R\)</span>.</p> </section> <section id=mosaicity-from-rotation-data > <h3 id=mosaicity-from-rotation-data ><a class=toc-backref href="#id59" role=doc-backlink >11.2 Mosaicity from rotation data</a><a class=headerlink href="#mosaicity-from-rotation-data" title="Link to this heading">¶</a></h3> <p>For rotation data the mosaicity <span class="math notranslate nohighlight">\(\sigma_M\)</span> is estimated by maximum likelihood from the rocking offsets <span class="math notranslate nohighlight">\(\tau\)</span> of indexed spots, using the XDS reflection-fraction model <span class="math notranslate nohighlight">\(R(\tau;\sigma_M/\zeta)\)</span> (Kabsch 2010): each spot’s exact Bragg angle is located near its frame, <span class="math notranslate nohighlight">\(\zeta\)</span> (the rotation-axis Lorentz component) is computed, and <span class="math notranslate nohighlight">\(\sigma_M\)</span> is chosen to maximize <span class="math notranslate nohighlight">\(\sum_i \log R(\tau_i;\sigma_M/\zeta_i)\)</span>.</p> <p>The <span class="math notranslate nohighlight">\(\phi\)</span> search window for the Bragg angle is set <strong>wider than the oscillation</strong>, so that reflections recorded at large rocking offset are included. These tail reflections carry most of the information about the mosaic width; a window limited to the oscillation range would truncate the <span class="math notranslate nohighlight">\(\tau\)</span> distribution and bias <span class="math notranslate nohighlight">\(\sigma_M\)</span> low.</p> <p>The fit uses only the <strong>strongest 250 spots</strong> of an image, whatever the indexing spot budget (<code class="docutils literal notranslate"><span class=pre >--max-spots</span></code>) is. A spot is detected when <span class="math notranslate nohighlight">\(I_\mathrm{full}R(\tau)\)</span> clears the finder threshold, so selecting spots by intensity censors on <span class="math notranslate nohighlight">\(R(\tau)\)</span>: a deeper list holds proportionally more large-<span class="math notranslate nohighlight">\(\tau\)</span> partially recorded spots and the fit widens with it. Left uncapped, <span class="math notranslate nohighlight">\(\sigma_M\)</span> therefore tracks the spot budget rather than the crystal — and since an over-wide mosaicity mis-states every partiality, the merge degrades sharply with it.</p> <p>The estimated mosaicity feeds the rotation prediction (how many frames each reflection spans, §8.3) and the rotation partiality (§10.2). It is <strong>held fixed during scaling</strong>: in the per-image scale fit the mosaicity is degenerate with the scale <span class="math notranslate nohighlight">\(G\)</span> (both rescale the predicted intensity), so refining it there is unstable. A correct mosaicity matters because it controls both how much of each rocking curve is captured and the partiality used to form fulls (§10.6); too small a value truncates the captured curve and over-peaks the partiality, degrading the combined fulls.</p> </section> </section> <hr class=docutils /> <section id=auxiliary-statistics-i-i-and-wilson-plot > <h2 id=auxiliary-statistics-i-i-and-wilson-plot ><a class=toc-backref href="#id60" role=doc-backlink >12. Auxiliary statistics: ⟨I/σ(I)⟩ and Wilson plot</a><a class=headerlink href="#auxiliary-statistics-i-i-and-wilson-plot" title="Link to this heading">¶</a></h2> <section id=per-shell-i-i > <h3 id=per-shell-i-i ><a class=toc-backref href="#id61" role=doc-backlink >12.1 Per-shell ⟨I/σ(I)⟩</a><a class=headerlink href="#per-shell-i-i" title="Link to this heading">¶</a></h3> <p>For monitoring integration quality, Jungfraujoch reports mean <span class="math notranslate nohighlight">\(\langle I/\sigma(I)\rangle\)</span> in a fixed number of resolution shells. Shelling is performed in <span class="math notranslate nohighlight">\(1/d^2\)</span> space (typical of crystallographic practice).</p> </section> <section id=wilson-plot-b-factor-proxy > <h3 id=wilson-plot-b-factor-proxy ><a class=toc-backref href="#id62" role=doc-backlink >12.2 Wilson plot (B-factor proxy)</a><a class=headerlink href="#wilson-plot-b-factor-proxy" title="Link to this heading">¶</a></h3> <p>A Wilson-type analysis is computed by binning intensities by resolution and fitting: <span class="math notranslate nohighlight">\( \langle I\rangle \propto \exp\!\left(-\frac{B}{2}\frac{1}{d^2}\right), \)</span> i.e. <span class="math notranslate nohighlight">\( \log \langle I\rangle = \mathrm{const} - \frac{B}{2}\left(\frac{1}{d^2}\right). \)</span> A linear regression of <span class="math notranslate nohighlight">\(\log\langle I\rangle\)</span> vs <span class="math notranslate nohighlight">\(1/d^2\)</span> provides an estimate of <span class="math notranslate nohighlight">\(B\)</span>, subject to basic quality checks (e.g. <span class="math notranslate nohighlight">\(R^2\)</span> threshold).</p> <p>A <strong>dataset-wide</strong> Wilson <span class="math notranslate nohighlight">\(B\)</span> is also estimated over the merged reflections — restricted to the meaningful resolution range (skipping the low-resolution non-linear region below ~4 Å and shells past the signal limit <span class="math notranslate nohighlight">\(\langle I/\sigma\rangle < 1\)</span>, so it is insensitive to how far the merged data extend) — and written to the merged mmCIF as <code class="docutils literal notranslate"><span class=pre >_reflns.B_iso_Wilson_estimate</span></code> (and reported as <code class="docutils literal notranslate"><span class=pre >WILSON_B=</span></code>), the analogue of XDS’s Wilson-line <span class="math notranslate nohighlight">\(B\)</span>. It is diagnostic only and is not fed back into scaling. It is also where the flux the fixed integration disk clips lands: that loss is degenerate with an overall <span class="math notranslate nohighlight">\(B\)</span>, so the reported number carries an <span class="math notranslate nohighlight">\(r_1\)</span>-dependent contribution and is not a property of the crystal alone (§9.1). The <strong>per-image</strong> estimate (used for the live radiation-damage plot) is accepted only when the fit is well-correlated and physically plausible (<span class="math notranslate nohighlight">\(0 < B < 200\)</span> Ų); on a bad frame (an indexing glitch, too few reflections) the Wilson line runs wildly steep, so an implausible <span class="math notranslate nohighlight">\(B\)</span> is reported as NaN rather than a spurious hundreds-of-Ų value.</p> </section> </section> <hr class=docutils /> <section id=space-group-determination-and-merge-level-decisions > <h2 id=space-group-determination-and-merge-level-decisions ><a class=toc-backref href="#id63" role=doc-backlink >13. Space-group determination and merge-level decisions</a><a class=headerlink href="#space-group-determination-and-merge-level-decisions" title="Link to this heading">¶</a></h2> <section id=space-group-determination > <h3 id=space-group-determination ><a class=toc-backref href="#id64" role=doc-backlink >13.1 Space-group determination</a><a class=headerlink href="#space-group-determination" title="Link to this heading">¶</a></h3> <p>When no space group is supplied, a POINTLESS-like search scores Laue-group symmetry (CC of <span class="math notranslate nohighlight">\(E^2(h)\)</span> vs <span class="math notranslate nohighlight">\(E^2(Rh)\)</span> — the intensities normalised by the mean of their own resolution shell — plus merge self-consistency) and detects screw/centering absences from the <span class="math notranslate nohighlight">\(P1\)</span>-merged intensities. Three tests gate a promotion to higher symmetry, all aimed at the merohedral twin, whose twin law forces non-equivalent reflections together and so mimics symmetry:</p> <ol class="arabic simple"> <li><p><strong>Merge self-consistency</strong> (<span class="math notranslate nohighlight">\(\chi^2\)</span> under the candidate group, relative to the confirmed subgroup). On its own this is not sufficient: it is a ratio to an error model that moves with the <em>amount</em> of data — the parent’s systematic term grows as <span class="math notranslate nohighlight">\(\sigma\)</span> shrinks with <span class="math notranslate nohighlight">\(1/\sqrt{N}\)</span>, while a twin’s is already saturated — so its verdict depends on how much data the search saw.</p> <li><p><strong>Error-model <span class="math notranslate nohighlight">\(b\)</span></strong> (the intensity-proportional systematic). A genuine symmetry step gains multiplicity without inflating <span class="math notranslate nohighlight">\(b\)</span>; merging a twin law’s extra operator inflates it. A <span class="math notranslate nohighlight">\(\chi^2\)</span>-passing promotion is vetoed when <span class="math notranslate nohighlight">\(b\)</span> rises past a bound relative to the confirmed subgroup.</p> <li><p><strong>Operator disagreement</strong>, a sigma-free statistic <span class="math notranslate nohighlight">\(H=\mathrm{median}\,|I_1-I_2|/(I_1+I_2)\)</span>, formed as the ratio of the operators a promotion <em>adds</em> to the parent’s own, measured on the same reflections. Normalising against the parent divides out the systematic floor that symmetry mates carry on real data, which varies by crystal and by operator; a median is used because a twin perturbs every pair whereas a badly-measured minority perturbs only the tail. Where a candidate has several parents of the same order, it is judged against the worst of them, since a rival subgroup can itself contain the twin laws.</p> </ol> <p>The correlation is on <strong>resolution-normalised</strong> intensity <span class="math notranslate nohighlight">\(E^2 = I/\langle I\rangle(\text{shell})\)</span>, normalised over exactly the reflections the correlation pairs. Both members of a symmetry pair lie at the same <span class="math notranslate nohighlight">\(|s|\)</span>, so on raw <span class="math notranslate nohighlight">\(I\)</span> the resolution fall-off is variance shared perfectly between the two arms and appears as a positive correlation for <em>any</em> pairing at all: a shell-matched random pairing — the exact null for a metrically-allowed false operator — scores a median 0.31 across the rotation battery, and on one crystal 0.53 — above the bound the correlation is tested against. That floor varies more from crystal to crystal (spread 0.46) than the whole true/false gap is wide (0.38), so an absolute bound on the raw statistic is a different test on every crystal; and it moves with the search resolution cut, which is what made that cut a symmetry-deciding parameter. Normalised, the floor has a median of 0.015, never exceeds 0.06, and barely moves with the cut.</p> <p>Both the correlation stage and the absence tests need to know whether a reflection is <strong>genuinely present</strong>, and that question is asked of its <em>counting</em> significance, not of the merged <span class="math notranslate nohighlight">\(I/\sigma\)</span>. A merged <span class="math notranslate nohighlight">\(\sigma\)</span> carries the error model’s intensity-proportional term, <span class="math notranslate nohighlight">\(\sigma^2 = a\,\sigma_0^2 + (b\,I)^2\)</span>, so merged <span class="math notranslate nohighlight">\(I/\sigma\)</span> saturates — at <span class="math notranslate nohighlight">\(\mathrm{ISa}\sqrt{n}\)</span> for a reflection observed <span class="math notranslate nohighlight">\(n\)</span> times, and at <span class="math notranslate nohighlight">\(\mathrm{ISa}\)</span> exactly for one observed once. Above that knee it stops rising with the intensity: on the weakest search merge of the rotation battery the <span class="math notranslate nohighlight">\(I/\sigma\)</span> of every decile of <span class="math notranslate nohighlight">\(E^2\)</span> reads 1.58–1.65 against an <span class="math notranslate nohighlight">\(\mathrm{ISa}\)</span> of 1.70, so reflections an order of magnitude apart in real intensity report the same number. A single constant applied there demands anywhere between 2.2 and 11.9 in counting significance depending on the crystal.</p> <p>The nominal cut <span class="math notranslate nohighlight">\(T\)</span> (default 3.0) is therefore converted once, using the ISa of the merge being searched. For a reflection observed once <span class="math notranslate nohighlight">\(\sigma_\text{counting}^2 = \sigma^2 - (b\,I)^2\)</span>, so <span class="math notranslate nohighlight">\(I/\sigma_\text{counting} \ge T\)</span> is exactly</p> <div class="math notranslate nohighlight"> \[\frac{I}{\sigma} \;\ge\; \frac{T}{\sqrt{1 + (T/\mathrm{ISa})^2}}\]</div> <p>The converted cut lies strictly below <span class="math notranslate nohighlight">\(\mathrm{ISa}\)</span> for every <span class="math notranslate nohighlight">\(\mathrm{ISa}\)</span>, so it is always reachable by the reflection the ceiling binds hardest, and it is within 1 % of <span class="math notranslate nohighlight">\(T\)</span> on any merge with <span class="math notranslate nohighlight">\(\mathrm{ISa} \ge 21\)</span> — a healthy merge is left exactly where it was. Multiplicity is taken as 1 deliberately rather than estimated, for the same reason. Where the merge reports no ISa (or <span class="math notranslate nohighlight">\(b = 0\)</span>) the cut is used as it stands.</p> <p>Several space groups may share an absence pattern exactly. Where they do, the search scores them identically and <strong>all of them are named</strong> in the result rather than one being reported as the answer: some are enantiomorph pairs, which merged intensities cannot distinguish in principle, and others differ only by a screw condition that the centering condition already implies, so the screw has no observable signature at all. The representative reported first is the lowest space-group number, which is a convention and not a measurement.</p> <p>The Lorentz factor <span class="math notranslate nohighlight">\(\zeta\)</span> (§8.3) governs how well a reflection can be measured, so when the spindle lies in a plane of the lattice, an operator permuting the two in-plane axes samples a different mixture of measurement qualities than one that only flips signs. The search is therefore run a second time on a merge of only the well-measured observations (<code class="docutils literal notranslate"><span class=pre >--search-min-zeta</span></code>, rotation default 0.85), both answers are reported, and <strong>where they disagree the merge of all the observations decides</strong>. The filter discards 40–80 % of the observations, which can starve an operator correlation the full merge confirms and can equally leave an operator confirmed that the full merge refuses, so the decision — the point group as well as the absences, which live in the weak reflections the filter removes — rests on the arm with every observation behind it. A tie (same order, different symmetry) is reported with both candidates named, for trying in molecular replacement.</p> <p><strong>Centering</strong> is accepted when the systematically-absent class is weak relative to the present one by <em>either</em> of two floor-independent tests: its mean signed <span class="math notranslate nohighlight">\(I/\sigma\)</span> well below the present mean, <em>or</em> its rate of individually-significant reflections well below the present class’s own significant rate. The second test covers weak and low-energy data, where a positive intensity floor (background and profile leakage) lifts the absent class’s mean <span class="math notranslate nohighlight">\(I/\sigma\)</span> well above zero and, when the present class is itself weak, carries the plain mean ratio past its bound; a false centering fails both tests, its absent class being as strong as the present one. When several centerings pass, they are ranked by their <strong>net</strong> systematic absences (absent minus violating), not the gross absent count, so a super-centering (e.g. <span class="math notranslate nohighlight">\(F\)</span> over a true <span class="math notranslate nohighlight">\(C\)</span>) whose extra, only-half-populated absent class dilutes the strength ratio does not out-rank the correct lower centering.</p> </section> <section id=twinning-check > <h3 id=twinning-check ><a class=toc-backref href="#id65" role=doc-backlink >13.2 Twinning check</a><a class=headerlink href="#twinning-check" title="Link to this heading">¶</a></h3> <p>A Padilla–Yeates <span class="math notranslate nohighlight">\(L\)</span>-test (<span class="math notranslate nohighlight">\(\langle|L|\rangle\)</span>, <span class="math notranslate nohighlight">\(\langle L^2\rangle\)</span>) and the second moment <span class="math notranslate nohighlight">\(\langle I^2\rangle/\langle I\rangle^2\)</span> (taken per resolution shell with noise-only shells skipped and Wilson outliers rejected, so a single strong reflection in a collapsed-mean shell cannot skew it) are written to the merged mmCIF as a twinning diagnostic. Twinning is only flagged in Laue classes where a merohedral twin law can exist; the holohedral high-symmetry classes (<span class="math notranslate nohighlight">\(4/mmm\)</span>, <span class="math notranslate nohighlight">\(6/mmm\)</span>, <span class="math notranslate nohighlight">\(m\bar{3}m\)</span>, and <span class="math notranslate nohighlight">\(\bar{3}m\)</span> on a rhombohedral lattice) are exempt, so a low <span class="math notranslate nohighlight">\(\langle|L|\rangle\)</span> there is reported as a statistical artefact rather than twinning.</p> </section> <section id=outlier-rejection > <h3 id=outlier-rejection ><a class=toc-backref href="#id66" role=doc-backlink >13.3 Outlier rejection</a><a class=headerlink href="#outlier-rejection" title="Link to this heading">¶</a></h3> <p>Merging applies an optional per-observation median-based <span class="math notranslate nohighlight">\(N\sigma\)</span> cut (<code class="docutils literal notranslate"><span class=pre >--reject-outliers</span></code>, default 6σ for <code class="docutils literal notranslate"><span class=pre >rot3d</span></code>, off otherwise). The same <span class="math notranslate nohighlight">\(N\sigma\)</span> cut is fed back into the error model: after an initial <span class="math notranslate nohighlight">\(a,b\)</span> fit the parameters are re-fit once on the reflections that survive rejection (dropping any whose squared deviation exceeds <span class="math notranslate nohighlight">\(N\sigma^2\,[a\,\sigma^2 + (b\,\langle I\rangle)^2]\)</span>), so the calibrated errors describe the reflections that actually enter the merge rather than the pre-rejection pool.</p> </section> <section id=automatic-resolution-cutoff > <h3 id=automatic-resolution-cutoff ><a class=toc-backref href="#id67" role=doc-backlink >13.4 Automatic resolution cutoff</a><a class=headerlink href="#automatic-resolution-cutoff" title="Link to this heading">¶</a></h3> <p>By default the reported/written high-resolution limit is trimmed where <span class="math notranslate nohighlight">\(\mathrm{CC}_{1/2}\)</span> falls off: a logistic is fitted to <span class="math notranslate nohighlight">\(\mathrm{CC}_{1/2}(s)\)</span>, and the limit is set <strong>one reported-shell width past</strong> the point where the fit crosses 0.30 — deliberately “one shell too far”, so weak-but-real data below the crossing are kept rather than discarded. The extension is measured over the range that is actually kept, not the full measured range, so a detector reaching far past where the crystal diffracts cannot inflate it. <code class="docutils literal notranslate"><span class=pre >--scaling-high-resolution</span></code> overrides the limit and <code class="docutils literal notranslate"><span class=pre >--resolution-cutoff</span> <span class=pre >off</span></code> disables it.</p> </section> <section id=diffraction-anisotropy > <h3 id=diffraction-anisotropy ><a class=toc-backref href="#id68" role=doc-backlink >13.5 Diffraction anisotropy</a><a class=headerlink href="#diffraction-anisotropy" title="Link to this heading">¶</a></h3> <p>How fast the intensity falls off with resolution can depend on direction. rugnux measures that, reports it, and does nothing else with it: no intensity is corrected, no reflection is removed on a directional criterion, and the merged data and the written files do not depend on direction at all.</p> <p><strong>The tensor.</strong> A deviatoric anisotropic displacement tensor is fitted to the merged intensities as</p> <div class="math notranslate nohighlight"> \[\ln \langle I(\mathbf{s})\rangle = c(\text{shell}) - \tfrac{1}{2}\,\mathbf{s}^\mathsf{T} B\, \mathbf{s},\qquad \mathbf{s} = \text{reciprocal-space vector},\ |\mathbf{s}| = 1/d\]</div> <p>with one free constant per resolution shell, so every isotropic feature — the Wilson curve, an ice ring, a noise floor, a scaling error — is absorbed exactly and only the <span class="math notranslate nohighlight">\(\ell = 2\)</span> angular part drives the tensor. For an isotropic <span class="math notranslate nohighlight">\(B\)</span> this reduces to the ordinary Wilson plot, so <span class="math notranslate nohighlight">\(B\)</span> here is the ordinary crystallographic (<span class="math notranslate nohighlight">\(B = 8\pi^2 U\)</span>) <span class="math notranslate nohighlight">\(B\)</span>, directly comparable with phenix.xtriage’s <code class="docutils literal notranslate"><span class=pre >B_cart</span></code>, ctruncate’s anisotropic <span class="math notranslate nohighlight">\(B\)</span> eigenvalues and AIMLESS’s anisotropic <span class="math notranslate nohighlight">\(\Delta B\)</span>. Only the deviatoric part is fitted: the isotropic part is degenerate with the overall scale. The tensor is constrained to the directions the Laue class allows — five free deviatoric parameters in triclinic, three in monoclinic, two in orthorhombic, one in tetragonal, trigonal and hexagonal, and <strong>none at all in cubic</strong>, where symmetry forces <span class="math notranslate nohighlight">\(\Delta B\)</span> to be exactly zero.</p> <p>The fit is on <strong>intensities, with no positivity cut</strong>. Fitting amplitudes, or dropping non-positive intensities as an amplitude-based tool must, loses roughly 40% of the signal: in a direction that has died half the merged intensities are negative, so a positivity cut keeps only the positive noise excursions and flattens the fall-off exactly where the anisotropy is largest.</p> <p><strong>Two different quantities are reported, and they are not interchangeable.</strong> <span class="math notranslate nohighlight">\(\Delta B\)</span> (the range of the principal components) is a <em>rate</em>; the diffraction limit along each principal direction — where <span class="math notranslate nohighlight">\(\langle I/\sigma(I)\rangle\)</span> in a 20° cone about that direction falls through 2, read by interpolation in <span class="math notranslate nohighlight">\(s^2\)</span> over equal-count shells — is where the signal actually runs out. A crystal can have a large <span class="math notranslate nohighlight">\(\Delta B\)</span> and almost no spread in directional limit, or the reverse. Where <span class="math notranslate nohighlight">\(\langle I/\sigma(I)\rangle\)</span> never falls through 2 in a direction, the limit returned is the <strong>edge of the measured data</strong> rather than the crystal’s own; such a direction is marked — with a <code class="docutils literal notranslate"><span class=pre ><</span></code> in the report, a 1 in <code class="docutils literal notranslate"><span class=pre >ANISOTROPY_D_MIN_CENSORED</span></code>, and a note in the mmCIF — so the spread is not read as a measurement when it is a lower bound. Both constants are AIMLESS’s (cone half-angle 20°, the <span class="math notranslate nohighlight">\(\langle I/\sigma\rangle\)</span> level this project states resolution at).</p> <p><strong>The resolution signature.</strong> A genuine Debye–Waller <span class="math notranslate nohighlight">\(B\)</span> makes the directional deficit a straight line through the origin in <span class="math notranslate nohighlight">\(s^2\)</span>. The per-shell <span class="math notranslate nohighlight">\(\ell = 2\)</span> amplitude is therefore fitted against <span class="math notranslate nohighlight">\(s^2\)</span> and the curve is classified: <em>linear</em> (a real <span class="math notranslate nohighlight">\(B\)</span>), <em>flat</em> (a deficit that does not follow <span class="math notranslate nohighlight">\(\exp(-\tfrac12 \mathbf{s}^\mathsf{T} B \mathbf{s})\)</span> at all, so the fitted <span class="math notranslate nohighlight">\(\Delta B\)</span> describes the data with the wrong functional form and may be an <strong>under</strong>-estimate), or <em>convex</em> (a deficit that grows faster than <span class="math notranslate nohighlight">\(s^2\)</span>, which a <span class="math notranslate nohighlight">\(B\)</span> cannot do). The verdict is re-derived at 8 and at 16 shells, and reported as undetermined if it moves.</p> <p><strong>The verdict, and what it is measured against.</strong> Whether an anisotropy is real is not decided against a counting-statistics error bar. Real data carry systematic error far larger than counting error, and gating on the latter reports anisotropy on datasets that have none. Instead the data set measures its own systematic error: in the tensor directions the Laue class <em>forbids</em>, the true tensor is exactly zero whatever the crystal is, so whatever is measured there is systematic. That measurement needs the unmerged observations — a merge has exact Laue symmetry by construction, and the forbidden directions are identically zero in it — so it is made on the scaled, rocking-curve-assembled observations. The counting part is subtracted, the counting error of the directions actually being tested is added back, and the result is the <strong>floor</strong>. What is tested against it is <span class="math notranslate nohighlight">\(\Delta B_\text{linear}\)</span> — the part of the fall-off that actually follows <span class="math notranslate nohighlight">\(\exp(-\tfrac12\mathbf{s}^\mathsf{T}B\mathbf{s})\)</span>, clamped at zero — and <strong>not</strong> the headline <span class="math notranslate nohighlight">\(\Delta B\)</span>; the report names which of the two it is quoting. The ratio is banded: below 2 not established, 2–3.5 marginal, above 3.5 established, above 5 strong. The bands are calibrated against known ground truth — merging cubic crystals in proper subgroups of their own Laue class, where the true anisotropy is exactly zero — which puts the false-positive rate at 24 % at 2.0, 10 % at 3.5 and 5 % at 5.0.</p> <p>The report says <strong>NOT DETECTED</strong>, <strong>DETECTED</strong>, or <strong>CANNOT DETERMINE</strong>, and the third is a real answer rather than an evasion. It is returned when the Laue class is triclinic (no forbidden direction exists, so there is no internal measurement of the systematic error and no substitute for it), when the observed rotation range is under about 90° (a lab-fixed systematic then reaches several tensor directions instead of one), when the merged data are at the noise floor, when the scale model carried no dose term (an uncorrected dose ramp manufactures anisotropy that no significance test can see through), or when no unmerged observations were available. The smallest <span class="math notranslate nohighlight">\(\Delta B\)</span> that could have been established on the data set is reported with the verdict; it is set by the systematic error rather than by counting, so it does <strong>not</strong> improve with more reflections or a longer exposure.</p> <p>A too-high space-group assignment is the one failure mode that is silent: real anisotropy is then pushed into the directions used to measure the systematic error, which inflates the floor and biases the answer towards reporting none. A caution says so, but only where that is actually indicated — a single free direction, no detection, <strong>and</strong> a forbidden-direction measurement far above its own counting noise — rather than on every tetragonal, trigonal and hexagonal data set.</p> <p>Everything lands in <code class="docutils literal notranslate"><span class=pre ><prefix>_report.txt</span></code> section 9 (<code class="docutils literal notranslate"><span class=pre >ANISOTROPY_*</span></code> keys), in the printed statistics, and in the merged mmCIF: the eigen-decomposition of the tensor as the standard <code class="docutils literal notranslate"><span class=pre >_reflns.pdbx_aniso_B_tensor_*</span></code> items (relative to the weakest direction, since only the deviatoric part is determined), and the directional limits, the shape and the verdict under the <code class="docutils literal notranslate"><span class=pre >_reflns.jfjoch_aniso_*</span></code> local prefix.</p> </section> <section id=practical-notes-and-limitations > <h3 id=practical-notes-and-limitations ><a class=toc-backref href="#id69" role=doc-backlink >13.6 Practical notes and limitations</a><a class=headerlink href="#practical-notes-and-limitations" title="Link to this heading">¶</a></h3> <ul class=simple > <li><p><strong>Bragg integration is profile-fitted by default</strong> (per-shell Gaussian profile, Kabsch extraction; §9.3), with plain box summation available as a fallback (<code class="docutils literal notranslate"><span class=pre >--integrator</span> <span class=pre >boxsum</span></code>). The profiles are built per frame from that frame’s strong spots, which suits fast-feedback and serial/streaming use; a profile shared across many frames (as in full offline workflows) is not currently formed.</p> <li><p><strong>Space-group symmetry</strong> beyond centering absences is not enforced during prediction/integration unless the space group is supplied and used downstream.</p> <li><p><strong>Resolution masking</strong> is controllable, and so is every stage of ice-ring handling (§3.3, §10.10). None of it runs unless the crystal is measured to have ice, because the fixed bands are a fixed cost in unique reflections whether it does or not.</p> <li><p><strong>Rotation vs still modes</strong> differ substantially in prediction and scaling: partiality is angle-driven in rotation data, while stills are predicted within an excitation-error window and get their partiality from the default-on per-crystal tilt post-refinement (§10.2) — or unit partiality with <code class="docutils literal notranslate"><span class=pre >--simple-stills</span></code>.</p> <li><p><strong>Amplitudes and intensities.</strong> The merged output carries both intensities (mmCIF <code class="docutils literal notranslate"><span class=pre >intensity_meas</span></code>, MTZ <code class="docutils literal notranslate"><span class=pre >IMEAN</span></code>/<code class="docutils literal notranslate"><span class=pre >SIGIMEAN</span></code>) and French–Wilson amplitudes (mmCIF <code class="docutils literal notranslate"><span class=pre >F_meas_au</span></code>, MTZ <code class="docutils literal notranslate"><span class=pre >F</span></code>/<code class="docutils literal notranslate"><span class=pre >SIGF</span></code>; §10.8), so a downstream program can refine against either.</p> </ul> </section> </section> <hr class=docutils /> <section id=model-based-validation-r-free-against-a-model-and-electron-density-maps > <h2 id=model-based-validation-r-free-against-a-model-and-electron-density-maps ><a class=toc-backref href="#id70" role=doc-backlink >14. Model-based validation: R-free against a model and electron-density maps</a><a class=headerlink href="#model-based-validation-r-free-against-a-model-and-electron-density-maps" title="Link to this heading">¶</a></h2> <p>Offline (<code class="docutils literal notranslate"><span class=pre >rugnux</span> <span class=pre >--model</span> <span class=pre >model.pdb</span></code>) the merged data can be scored against a supplied atomic model and <strong>initial</strong> electron-density maps computed — enough to confirm that a model fits the data and to inspect the density, not a substitute for refinement. <strong>The structure itself is not refined</strong>; the model is only re-fractionalized into the data unit cell (a rigid cell adjustment, so a deposited model with a slightly different cell still lines up), and the observed amplitudes are the French–Wilson <span class="math notranslate nohighlight">\(|F|\)</span> from §10.8, so the R-free and the maps use exactly the same amplitudes as the written reflection file. The model, structure-factor, bulk-solvent and FFT machinery is provided by GEMMI.</p> <section id=model-structure-factors > <h3 id=model-structure-factors ><a class=toc-backref href="#id71" role=doc-backlink >14.1 Model structure factors</a><a class=headerlink href="#model-structure-factors" title="Link to this heading">¶</a></h3> <p>The model electron density is sampled on a grid (IT92 X-ray form factors, with a Refmac-compatible Gaussian blur chosen for the grid spacing) and Fourier-transformed to structure factors <span class="math notranslate nohighlight">\(F_\mathrm{calc}(hkl)\)</span> up to the data resolution.</p> </section> <section id=bulk-solvent-and-scaling > <h3 id=bulk-solvent-and-scaling ><a class=toc-backref href="#id72" role=doc-backlink >14.2 Bulk solvent and scaling</a><a class=headerlink href="#bulk-solvent-and-scaling" title="Link to this heading">¶</a></h3> <p>A flat bulk-solvent mask around the model is transformed to <span class="math notranslate nohighlight">\(F_\mathrm{mask}\)</span>, and the model is scaled to the observed amplitudes by an overall least-squares fit of a scale <span class="math notranslate nohighlight">\(k\)</span>, an anisotropic <span class="math notranslate nohighlight">\(B\)</span>, and the flat-solvent parameters <span class="math notranslate nohighlight">\(k_\mathrm{sol}, B_\mathrm{sol}\)</span>:</p> <p><span class="math notranslate nohighlight">\( F_\mathrm{model} = k\,e^{-\mathbf{h}^\top \mathbf{B}\,\mathbf{h}/4}\left(F_\mathrm{calc} + k_\mathrm{sol}\,e^{-B_\mathrm{sol}\,s^2}\,F_\mathrm{mask}\right),\quad s^2 = 1/4d^2. \)</span></p> <p>This is the standard, few-parameter scaling model used by refinement programs. No free-form per-resolution-shell rescale is applied: such a rescale is dataset-specific and reshapes each map’s radial amplitude profile differently, which would make maps from a multi-dataset campaign no longer directly comparable.</p> </section> <section id=r-work-and-r-free > <h3 id=r-work-and-r-free ><a class=toc-backref href="#id73" role=doc-backlink >14.3 R-work and R-free</a><a class=headerlink href="#r-work-and-r-free" title="Link to this heading">¶</a></h3> <p>Crystallographic R-factors are reported over the work and free sets (the §10.7 flags):</p> <p><span class="math notranslate nohighlight">\( R = \frac{\sum \big|\,|F_o| - |F_\mathrm{model}|\,\big|}{\sum |F_o|}, \)</span></p> <p>with R-free the same sum restricted to the free set. Note that the scaling of §14.2 is fitted over <strong>all</strong> reflections, work and free alike — its few parameters (<span class="math notranslate nohighlight">\(k\)</span>, an anisotropic <span class="math notranslate nohighlight">\(B\)</span>, <span class="math notranslate nohighlight">\(k_\mathrm{sol}\)</span>, <span class="math notranslate nohighlight">\(B_\mathrm{sol}\)</span>) are far too few to absorb individual reflections, but R-free here is strictly “free of refinement”, not free of the scaling fit.</p> </section> <section id=electron-density-maps > <h3 id=electron-density-maps ><a class=toc-backref href="#id74" role=doc-backlink >14.4 Electron-density maps</a><a class=headerlink href="#electron-density-maps" title="Link to this heading">¶</a></h3> <p>Two maps are formed with the model phases <span class="math notranslate nohighlight">\(\varphi_\mathrm{model}\)</span>: a <span class="math notranslate nohighlight">\(2F_o-F_c\)</span> map, coefficients <span class="math notranslate nohighlight">\((2|F_o|-|F_\mathrm{model}|)\,e^{i\varphi_\mathrm{model}}\)</span>, and an <span class="math notranslate nohighlight">\(F_o-F_c\)</span> difference map, <span class="math notranslate nohighlight">\((|F_o|-|F_\mathrm{model}|)\,e^{i\varphi_\mathrm{model}}\)</span>, each inverse-Fourier-transformed to a real-space CCP4 map (<code class="docutils literal notranslate"><span class=pre ><prefix>_2fofc.ccp4</span></code>, <code class="docutils literal notranslate"><span class=pre ><prefix>_fofc.ccp4</span></code>). A map-coefficient MTZ (<code class="docutils literal notranslate"><span class=pre ><prefix>_maps.mtz</span></code>: <code class="docutils literal notranslate"><span class=pre >FP</span></code>, <code class="docutils literal notranslate"><span class=pre >FC</span></code>, <code class="docutils literal notranslate"><span class=pre >PHIC</span></code>, <code class="docutils literal notranslate"><span class=pre >FWT</span></code>/<code class="docutils literal notranslate"><span class=pre >PHWT</span></code>, <code class="docutils literal notranslate"><span class=pre >DELFWT</span></code>/<code class="docutils literal notranslate"><span class=pre >PHDELWT</span></code>, <code class="docutils literal notranslate"><span class=pre >FREE</span></code>) is written alongside so the maps can be reopened or rebuilt in Coot / PyMOL. These are unweighted difference coefficients (no <span class="math notranslate nohighlight">\(\sigma_A\)</span> / figure-of-merit weighting), which is why they are described as <em>initial</em> maps.</p> </section> <section id=aligning-the-data-to-the-model-enantiomorph-and-indexing-ambiguity > <h3 id=aligning-the-data-to-the-model-enantiomorph-and-indexing-ambiguity ><a class=toc-backref href="#id75" role=doc-backlink >14.5 Aligning the data to the model: enantiomorph and indexing ambiguity</a><a class=headerlink href="#aligning-the-data-to-the-model-enantiomorph-and-indexing-ambiguity" title="Link to this heading">¶</a></h3> <p>The model fixes a definite hand and indexing, but the merged data need not share them, so before comparison the observed reflections are brought into the model’s frame.</p> <ul class=simple > <li><p><strong>Enantiomorph / screw.</strong> When the data space group is the enantiomorph of the model’s (e.g. data <span class="math notranslate nohighlight">\(P4_12_12\)</span>, model <span class="math notranslate nohighlight">\(P4_32_12\)</span>; or <span class="math notranslate nohighlight">\(P3_1/P3_2\)</span>), the two are <strong>indistinguishable from merged intensities</strong> — <span class="math notranslate nohighlight">\(|F_\mathrm{calc}|\)</span> is invariant under the change of hand, so R-free cannot choose between them and probing would be meaningless. The model’s group is therefore adopted as the <strong>label</strong> the reflections are written under, and the reflections themselves are left untouched. The two groups of an enantiomorphic pair differ only in the translations of their operations: their rotations are identical, so they transform <span class="math notranslate nohighlight">\(hkl\)</span> identically, share a reciprocal ASU, and assign the Bijvoet hands identically. The label carries no handedness, and there is nothing about it to undo. Reindexing by the change-of-hand operator — which is the inversion — would instead <strong>swap <span class="math notranslate nohighlight">\(I(+)\)</span> with <span class="math notranslate nohighlight">\(I(-)\)</span></strong>, flipping every anomalous difference on the strength of a label the space-group search itself reports as undetermined; where the model is genuinely the wrong enantiomorph for the crystal, it would manufacture agreement rather than reveal the mismatch. What does carry the hand is the indexing the data already have, from the diffraction geometry, and the anomalous differences that come with it. §14.6 is what tests them against the model.</p> <li><p><strong>Indexing (merohedral) ambiguity.</strong> When the crystal has a merohedral ambiguity (§10.9), the observed intensities <em>do</em> differ between indexings, and the right one is chosen against the best available reference. <strong>If a reference MTZ was supplied, the data were already reindexed to agree with it</strong> (§10.9 — by the reference-intensity correlation, at the merge stage for rotation data or per image in stills scaling), and model validation keeps that authoritative choice. <strong>Only with a model and no reference</strong> does validation resolve the ambiguity itself, as a fallback: the scaled model is fit to each reindexing of the data (identity plus the twin-law cosets) and the one giving the <strong>lowest R-free</strong> is kept. This matters for a multi-dataset campaign — a single shared reference fixes one indexing convention for every dataset, whereas an independent per-dataset lowest-R-free choice could send borderline datasets to different conventions. A no-op either way for a holohedral crystal (no twin laws) Both reindexings — the change of hand and the ambiguity choice — are then applied to the merged reflections themselves, which are written after this step, so the reflection file, the R-factors and the maps describe one indexing. The change of hand also changes the space group the file is written in (it is the model’s enantiomorph), and since it is the inversion it exchanges the Bijvoet mates: on anomalous data adopting a model’s hand is what puts the anomalous differences the right way round. The ambiguity choice is reported with the R-free of the winner and of the runner-up, since the margin between them is what says whether the data decided or the two came out within noise of each other.</p> </ul> </section> <section id=anomalous-difference-map-and-the-sites-it-names > <h3 id=anomalous-difference-map-and-the-sites-it-names ><a class=toc-backref href="#id76" role=doc-backlink >14.6 Anomalous difference map and the sites it names</a><a class=headerlink href="#anomalous-difference-map-and-the-sites-it-names" title="Link to this heading">¶</a></h3> <p>Where the merge kept the Bijvoet split (§10.5) — which a rotation merge does by default, whether or not the mates were averaged — an <strong>anomalous difference map</strong> is computed as well, with coefficients</p> <p><span class="math notranslate nohighlight">\( \big(|F(+)| - |F(-)|\big)\, e^{i(\varphi_\mathrm{model} - \pi/2)}, \)</span></p> <p>over the acentric reflections that have both hands (a centric reflection has no anomalous difference, only noise). Turning the model phase back by 90° is what makes the anomalous scattering, which is 90° out of phase with the normal scattering, add up in the real part: the map’s peaks then sit on the anomalous scatterers. It is written as <code class="docutils literal notranslate"><span class=pre ><prefix>_anom.ccp4</span></code>. Its hand is the one §14.5 settled: with the mates the wrong way round every peak becomes a trough, so a map of clean peaks is itself a check that the frame is right.</p> <p>Rather than searching the map for blobs and leaving a list of coordinates, the map is read <strong>at the model’s own atom centres</strong> (hydrogens excluded — they scatter no anomalous signal), and the ten highest, in units of the map’s r.m.s., are reported in the log and as <code class="docutils literal notranslate"><span class=pre >ANOMALOUS_SITE_01</span></code>…<code class="docutils literal notranslate"><span class=pre >ANOMALOUS_SITE_10</span></code> in the results report. Each site is therefore named — the atom, residue and chain it belongs to — which is what says <em>what</em> carries the signal, not just where it is. The reading is cubic, not linear: the map is sampled every <span class="math notranslate nohighlight">\(d_\mathrm{min}/3\)</span>, and a peak that sharp read by trilinear interpolation comes out up to a quarter low — unevenly enough to reorder the sites. This reading is the one ANODE reports.</p> <p>Because the map is built on the model’s phases and the data’s own indexing, it is also the only test of whether the two agree about the <strong>hand</strong> (§14.5). A model and a dataset in opposite hands turn every anomalous peak into a trough, so a map whose deepest hole at an atom is both deeper than <span class="math notranslate nohighlight">\(5\sigma\)</span> and deeper than its highest peak says so, and the run reports it as a warning naming that atom. It is not repaired by reindexing: that would make the two agree by construction and destroy the evidence for which of the model and the data is in the wrong hand. Note that R-free cannot see this at all — a mirrored model gives R-free to four decimal places unchanged, and an exactly inverted anomalous map.</p> <p>The list is always ten entries long, so it is their height that carries the information: on a sulfur-SAD dataset the sulfurs fill the top of the list and are followed by a clear drop to the couple of sigma that is the map’s noise, while a dataset with no anomalous signal has no such separation and lists ten unrelated atoms at noise level. A scatterer the <strong>model does not contain</strong> — a bound ion, a soaked heavy atom — is by construction invisible in the list, and is what the map file is for.</p> </section> </section> </section> </article> </div> </div> </main> </div> <footer class=md-footer > <div class=md-footer-nav > <nav class="md-footer-nav__inner md-grid"> <a href=DETECTOR_GEOMETRY.html title="Detector geometry" class="md-flex md-footer-nav__link md-footer-nav__link--prev" rel=prev > <div class="md-flex__cell md-flex__cell--shrink"> <i class="md-icon md-icon--arrow-back md-footer-nav__button"></i> </div> <div class="md-flex__cell md-flex__cell--stretch md-footer-nav__title"> <span class=md-flex__ellipsis > <span class=md-footer-nav__direction > "Previous" </span> Detector geometry </span> </div> </a> <a href=OPENAPI.html title=OpenAPI class="md-flex md-footer-nav__link md-footer-nav__link--next" rel=next > <div class="md-flex__cell md-flex__cell--stretch md-footer-nav__title"><span class=md-flex__ellipsis > <span class=md-footer-nav__direction > "Next" </span> OpenAPI </span> </div> <div class="md-flex__cell md-flex__cell--shrink"><i class="md-icon md-icon--arrow-forward md-footer-nav__button"></i> </div> </a> </nav> </div> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class=md-footer-copyright > <div class=md-footer-copyright__highlight > © Copyright 2024, Paul Scherrer Institute. </div> Created using <a href="http://www.sphinx-doc.org/">Sphinx</a> 8.1.3. and <a href="https://github.com/bashtage/sphinx-material/">Material for Sphinx</a> </div> </div> </div> </footer> <script src="_static/javascripts/application.js"></script> <script>app.initialize({version: "1.0.4", url: {base: ".."}})</script> |