Both were one std::sort on one thread over tens of millions of elements, and
together they were a third of a crowded crystal's run.
Bucketing by h first makes them parallel. h is the comparator's leading key, so
the sorted array is exactly the buckets laid end to end, and each bucket sorts
on its own thread. In Ingest the keys are built straight into their bucket slot,
so this replaces the build pass rather than adding one and the packed-key array
is never duplicated; the extra memory is a few hundred kilobytes of histograms.
Buckets are taken largest first, because the tail of the phase is whichever
bucket finishes last.
The run split falls out of the same structure for free: a run of equal (h,k,l)
never crosses an h boundary, so each bucket counts its own runs, a scan over the
buckets gives the offsets, and the arrays are sized exactly - which also removes
the repeated growth the push_backs were paying for. The h range comes from the
finiteness pass, which already reads every observation.
The partials order became total in an earlier commit, when the observation index
was added as the last key. That is what makes this safe rather than merely fast:
the permutation is uniquely determined, so a bucket sort produces the same one a
single sort would.
Measured on a crystal with 66 million partial observations: Ingest 15.2 s and
14.3 s -> 8.3 s and 7.4 s, the post-refine event sort out of the top ten gaps
entirely, the whole crystal 2m22s -> 1m24s. Battery 15m32s -> 10m05s. Same space
group on all 24 crystals, none failed, and no crystal's R_meas moved by more
than 0.3 points.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
