Files
leonarski_fandClaude Opus 5.5 9caad1487d Refinement on the GPU: the normal matrix in the engine's memory budget
The positions' normal matrix a trial holds on its card while it minimises (J and its index, the
CG vectors, the per-residue block inverses, the kept residuals of the reorthogonalisation) was
outside the GPU budget that places the trials: on a 16 GB card a large model placed three trials
by a 12.1 GB count and peaked at 14.2 GB (a run died there with an out-of-memory once the card
had less to spare). RestraintNormalMatrixGPU::DeviceBytes counts what its allocations come to,
from the geometry restraints as the engine's restraint count takes them and the residues'
parameter atoms; RefineOnce adds it to each engine's own bytes, so the slot placement and the CPU
fallback, both decided up front, see it.

Ribosome case (open arm, 16 GB card, -N 16): 0.91 GB per trial; two trials at once instead of
three; predicted peak 11.74 GB (11196 MiB), measured 11348 MiB. Without the count the measured
14198 MiB matches the old placement plus three normal matrices (14124 MiB). With the card
reported as 7 GB the run goes to the CPU before the first trial, with 12 GB it places one trial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
2026-11-17 16:24:38 +01:00

43 lines
3.1 KiB
C++

// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#pragma once
// Restrained refinement's targets on a GPU (CUDA builds only): the RefinementEngine of RefinementXray.h with
// everything per evaluation on the device - the atoms' density (ModelDensityGPU's gather), its transform, the
// symmetry composed in reciprocal space, the likelihood target, the derivative map by the inverse transform
// of the target's derivatives and every atom's gradient read off it, and the geometry restraints. The host
// passes positions in and gets one value and one gradient per atom back.
#include <memory>
#include "RefinementXray.h"
// What an engine holds on its card: its own part; the part the engines like it on the same card share (the
// reflections' tables and the bulk-solvent mask's buffers), which the first of them there makes; and a lane of the
// transforms' working buffers, which an engine has to itself or shares with others, one evaluation at a time.
struct RefinementGPUBytes {
size_t engine = 0, card = 0, lane = 0;
};
// The engine for refining `model` against `refl` on `device`, or null where the GPU cannot take this case (an
// atom's box wider than the cell, too many symmetry operators) or its memory is over `budget` bytes.
// `b_max`: the largest B an atom may reach, which the gather's reserve is sized for. Throws JFJochException on a
// CUDA failure. `nthreads`: the host threads that work out the atoms' densities. `restraint_records`,
// `restraint_slots`: the most records (plane atoms included) and gradient slots its two restraint sets hold together,
// which the memory is worked out for (0: about thirty records and sixty slots per atom). `normal_bytes`: what the
// minimiser's normal matrix holds beside it while a trial refines on the engine (RestraintNormalMatrixGPU::DeviceBytes),
// counted as the engine's own.
std::unique_ptr<RefinementEngine> MakeRefinementEngineGPU(int device, const gemmi::UnitCell &cell,
const gemmi::SpaceGroup &sg, double d_min,
const RefinementReflections &refl,
const gemmi::Model &model, double b_max, size_t budget,
RefinementGPUBytes *bytes = nullptr, size_t nthreads = 1,
size_t restraint_records = 0, size_t restraint_slots = 0,
size_t normal_bytes = 0);
// An engine like `like` (one the function above made) on `device`: the same case, and on a card where an engine
// like it is, that card's shared part with it - so it holds RefinementGPUBytes::engine there, and ::lane beside where
// `own_lane` (else it shares the card's least used lane). Throws JFJochException on a CUDA failure.
std::unique_ptr<RefinementEngine> MakeRefinementEngineGPU(const RefinementEngine &like, int device, size_t nthreads,
bool own_lane);