Performance#

All figures below are on a stack of 8280×5520 sixteen-bit TIFFs, an RTX 3060 and a 128-thread host, with the same frames and alignment settings for every row. Each figure is the whole run, depth pass included.

Native#

The --no-align row is decode-bound — about a second per frame for sixteen-bit PNG with four decoder threads — which is why it is no faster than the aligning CPU row and why the GPU does not help it. Uncompressed TIFF input decodes an order of magnitude faster.

Browser#

Measured in headless Chrome on the same RTX 3060:

The browser's 0.65 s per frame breaks down as decode 0.4 s, align 0.2 s, fuse 0.1 s. Both paths stream, and the per-frame GPU work is the same; the difference is decode and the cost of round-tripping to the GPU process.

Before the search's per-level schedule and batched readbacks, the same browser rows were 30 s and 2.0 s.

Memory#

Memory is flat in the stack's length. Frames stream through alignment and fusion one at a time, so what is resident is a handful of frames plus the accumulator.

What does grow is the depth pass's focus slices — 45 MB per 45-megapixel frame natively at the default half grid, 5.7 MB in the browser where they are quantized to sixteen bits on a quarter grid. A hundred frames is therefore about 4.5 GB natively, or 570 MB in the browser.

Peak RSS swings between runs by how far the read-ahead decoders (four frames) get ahead of the fold.

In the browser, every full-resolution plane is one storage buffer needing 3 × w × h × 4 bytes (550 MB at 45 MP), within the adapter's maxBufferSize and maxStorageBufferBindingSize, and about 3.5 GB of GPU memory in total. See browser and hardware.

Where the time went#

The current numbers are the end of a chain of optimizations, and the history says what actually costs time in this kind of program.

The aligner#

Before the cost was fused into one pass and the search put on a per-level schedule, the four native rows read 118 s, 74 s, 449 s and 265 s. The CPU search alone was 1.2 s of every frame and the two registration pyramids another 0.5 s.

The search now follows the pyramid: at each level the simplex starts one pixel of that level wide — in every parameter, measured by how far it moves the frame's edge — and stops at a tenth of one. A coarse level only has to hand the next one a start within its own pixel, and the finest level searched settles sub-pixel. That is about 300 cost evaluations for a 45-megapixel pair at coarsen 2, where a fixed simplex and tolerance at every level took 450 to 700.

Each evaluation is one row-parallel pass that warps and reduces at once, with nothing allocated: 1.6 ms at level 2 of 45 MP on 128 threads, 20 ms at full resolution.

Frames on the device#

Until the frames moved onto the GPU with --gpu-align, the CUDA rows read 50 s and 161 s: the CPU-side warps, pyramids and focus measure were 0.5 s of every frame. Now a decoded frame is uploaded once and everything happens there.

The depth pass#

It used to decode and warp every frame a second time to take its focus measure — 1.2 s per frame on either path, 42 s of a 25-frame run. Now the fold takes each frame's focus slice as it passes, at 0.15 s per frame.

The remaining pass was 29 s of a 25-frame run and 83 s of a 100-frame one. The box filters were allocating four fresh 45 MB planes per slice; keeping the planes from one slice to the next took the aggregation from 0.75 s to 0.045 s per slice. The WLS went from 9 s to 2.8 s for both solves with the multigrid preconditioner — and converged, which it had not always done before.

The pooling allocator#

The CLI runs under a pooling global allocator. A block of 4 MB or more is not returned to the system on free but kept — up to 64 blocks and 4 GB — and the next request of its size class gets it back with its pages still mapped.

This matters more than it should. Every stage of the fold makes fresh planes: the decoder's buffers, the warp, the luma, the pyramid's levels, the energies, the focus slice. glibc unmaps a block above its mmap threshold on free, so each frame took the page faults again — and with 128 threads first-touching a plane at once that is about 150 ms per 45 MB plane.

It was more than half of the fold:

Output is bit-identical either way; the pool costs 1.8 GB more peak memory in idle blocks. A 2 GB cap loses most of the gain — a frame's working set of planes is larger than that — and 6 GB gains nothing over 4. Zeroed requests on a recycled block are cleared with memset, about 4 ms per 45 MB.

The CUDA rows gain too, because the decode and the slices are host planes, and the depth pass's host side with them (1.7 s to 1.0 s for 25 frames).

The browser build does not use the pool.

Reproducibility#

The browser's aligner produces the same kind of transforms as the native one — scale 1.000 to 1.064 over a 25-frame sweep, with sub-pixel shifts — but not bit-identical ones, since the objective is evaluated in FP32 on a slightly different pyramid.

What to do about it#