Depth from focus#

Every run produces a dense, sub-frame depth map and a confidence map. This page is the algorithm; the depth map is what you see and what it is for.

The pipeline is the modern non-learned depth-from-focus recipe, written from the papers. Each frame's focus measure is taken as the frame is folded into the fusion, so the pass costs no measurable wall time on a 25-frame run, and the fused image itself becomes the guide.

1 · Focus measure#

--depth-focus F, default rdf:1:3.

The ring difference filter of Jeon, Surh, Im and Kweon (IEEE TIP 2019):

| mean(disk r ≤ 1) − mean(ring 1 < r ≤ 3) |

on luma, at full resolution. The inner and outer radii are rdf:RIN:ROUT.

sml[:STEP] selects Nayar and Nakagawa's sum-modified Laplacian instead.

2 · Cost aggregation#

--depth-scale S (default 1 = half resolution), --depth-agg R:EPS (default 3:1e-4).

The measure is block-averaged down to a working grid, and then every slice of the focus volume is filtered by a guided filter (He, Sun and Tang, 2013) with the fused all-in-focus luma as the guide.

This is the edge-aware aggregation of fast cost-volume filtering: a pixel's focus score is pooled with its neighbors', but only across neighbors that belong to the same thing in the image.

--depth-agg 0 turns aggregation off.

3 · Peak search#

Streamed over the frame axis with O(1) memory in the stack size. For each pixel, tracking as the slices go by:

• The global peak with its two neighbors — Gaussian interpolation (Nayar and Nakagawa,
  1994. gives a fractional frame index, which is where the sub-frame precision comes from
• The second-best local maximum, which gives the peak-ratio confidence
• The profile mean, for prominence
• A noise gate against the median profile minimum (--depth-gate G, default 1: full confidence needs a peak at least (1+G) × the noise floor)

Confidence is normalized so that its 90th percentile is 1.

4 · Regularization#

--depth-lambda L (default 3), --depth-sigma S (default 0.04), --depth-cg N (default 200), --depth-robust T (default 1), --depth-no-median.

Edge-aware weighted least squares (Farbman et al., 2008) with the confidence as the data weight. Flat, noisy or ambiguous pixels take their depth from confident neighbors, without crossing image edges. --depth-sigma is how sensitive the guide is to an edge, in luma units.

The separable fast global smoother (Min et al., 2014) gives the initial guess, and a conjugate gradient solves the two-dimensional system to a relative residual of 1e-5.

The multigrid preconditioner#

The CG is preconditioned by one multigrid V-cycle: the system is aggregated 2×2 down to a 16-pixel grid — the data weights summed over each block, the edges a block boundary cuts summed into the coarse edge, which is the Galerkin operator of a piecewise-constant prolongation, so the preconditioner stays symmetric — with one damped-Jacobi sweep before and after each coarse correction.

This matters more than it sounds. A Jacobi-preconditioned CG left the low modes to the CG itself: on the 4140×2760 half grid of a 45-megapixel stack it needed 300 iterations to reach 1e-5, so the --depth-cg 200 cap stopped it at 7e-5 — 0.4 frames off the converged map in places — and where confidence is low over a large area (a flat wall, or a synthetic stack with no peaks) it did not converge at all.

The V-cycle takes the same solve to 1e-5 in about 20 iterations, 50 on the pathological case: 3.5 s down to 1 s on the CPU.

Robust reweighting#

One Huber reweighting pass (--depth-robust, default 1 frame) then removes outliers a least-squares fit would otherwise average in. The second solve reuses the hierarchy.

A 3×3 median runs before the WLS unless --depth-no-median says otherwise.

5 · Upsampling#

--depth-upsample U, default guided:2:1e-2.

Guided-filter upsampling on the full-resolution luma, so depth edges land on image edges. bilinear is the alternative.

Outputs#

The winner map#

--depth winner reports something else entirely: the raw winning frame index read from pyramid level --depth-level (default 2, quarter resolution) — a free by-product of the fusion, not a depth estimate.

It is noisy wherever the scene is flat, and the finest level's winner map is noise nearly everywhere. It is useful for debugging the fusion, not for measuring a subject. In the browser it is not shown as a layer; it drives the Ctrl + G pixel lookup and has its own save button.

On the GPU#

With --gpu, the whole pass runs on the device — the same stages as the browser's, multigrid included. Slices are uploaded one at a time, only the two medians (the noise floor and the confidence scale) are taken on the host, and the maps come back: 1.7 s for 25 frames, 2.3 s for 100.

The maps are within 4 × 10⁻⁴ of full scale of the CPU's. A few pixels in a million of the confidence differ, where a near-tie in the peak search falls the other way in float order.

The browser's map matches the native one to 4 × 10⁻⁴ frames — the limit of the sixteen-bit quantization the slices are stored in — and the confidence to 3 × 10⁻³ on average, with about 0.1 % of pixels differing by more.

What it costs#

On a 25 × 45-megapixel stack the pass is not measurable against the run. The history is instructive about why:

• Originally the depth pass decoded and warped every frame a second time to take its focus measure: 1.2 s per frame, 42 s of a 25-frame run.
• Taking each frame's focus slice as it passes through the fold brought that to 0.15 s per frame.
• The remaining pass — aggregation, WLS, upsampling — was 29 s of a 25-frame run, with the aggregation at 0.75 s per slice. The box filters were allocating four fresh 45 MB planes per slice, and faulting a fresh plane in from 128 threads at once costs about 150 ms while the sums themselves cost 20 ms. Keeping the planes from one slice to the next: 0.045 s per slice, sixteen times faster.
• The WLS went from 9 s to 2.8 s for both solves with the multigrid preconditioner — and actually converged.

What the pass keeps is one slice per frame: 45 MB per 45-megapixel frame natively at the default half grid, or 5.7 MB in the browser where slices are quantized to sixteen bits on a quarter grid.