How lapstack works#

lapstack is a from-scratch implementation written from two papers:

• Adelson, Anderson, Bergen, Burt and Ogden, Pyramid methods in image processing (RCA Engineer, 1984) — REDUCE and EXPAND, the band-pass decomposition, and the "multifocus composite": pick, node by node, the pyramid coefficient with the larger magnitude, then expand and add. The blending between frames happens in the reconstruction itself.
• Wang and Chang, A Multi-focus Image Fusion Method Based on Laplacian Pyramid (Journal of Computers 6(12), 2011) — the 5×5 binomial generating kernel, maximum region energy selection for the band-pass levels, and a local deviation plus entropy rule for the residual.

The problem with picking sharp pixels#

The obvious algorithm — for each pixel, take the frame where it looks sharpest — does not work. Sharpness is not a property a single pixel has; it depends on the neighborhood, and on the scale you look at. A fine texture and a soft broad gradient are sharp in different senses, and a per-pixel decision produces speckle.

The 1984 insight is to take the picture apart by scale first.

The pyramid#

A Gaussian pyramid is a sequence of progressively smaller, blurrier copies of an image. Each level is made from the one above by REDUCE: blur with a small kernel, then take every second pixel.

A Laplacian pyramid is the difference between consecutive Gaussian levels. Each of its levels holds the detail that exists at one scale and not the next — a band-pass image — and at the bottom sits a small residual, the low-pass remainder that carries the overall tone and color.

Two properties make this useful:

• It is complete. Collapse the pyramid (expand and add, bottom to top) and you get the original image back exactly.
• It is linear. The pyramid of a sum is the sum of the pyramids.

lapstack's transform#

A separable kernel [1 4 6 4 1]/16 — Burt's generating kernel with a = 0.375 — with reflect-101 borders. That makes REDUCE and EXPAND plain linear operators at any image size, odd or even at every level, so collapse(build(x)) == x to float precision regardless of the dimensions.

The number of band-pass levels defaults to as many as keep the residual's short side at 32 pixels or more: seven levels on an 8280×5520 frame, whose residual is then 65×44. --levels N overrides.

The fusion rule#

The paper's rule is for two frames; lapstack generalizes it to N.

Band-pass levels: maximum region energy#

For each coefficient, compute a region energy

RE = Σ ω · L²

over a binomial-weighted window (--energy-radius, default 1 = 3×3) of the luminance coefficient — Y = 0.299R + 0.587G + 0.114B of the RGB coefficients. The pyramid is linear, so that is the luma pyramid.

The frame with the largest region energy wins the coefficient, ties going to the earlier frame. The selection is then applied to all three channels, so color never splits at a selection edge.

Radius 0 reduces this to Adelson's original per-node |L| maximum.

The residual: a Pareto rule#

Winner-take-all is wrong at the residual, because every frame has essentially the same low-pass content. The paper uses local deviation D and local entropy E (--top-radius, default 2 = 5×5; --entropy-bins, default 256): take A when A wins both measures, B when B wins both, average otherwise.

Generalized to N frames, lapstack keeps that as a Pareto rule. A frame is dominated if another is at least as good on both measures and strictly better on one; the fused value is the mean of the frames that are not dominated.

On a deep stack this barely matters — as Burt and Adelson note, the low-pass content is shared between frames, so the residual rule is nearly moot.

Halo control#

A pyramid's characteristic artifact comes from each level picking its own winner independently.

Beside a bright object, the frames focused behind it carry the object's defocused copy spread over the background — strong coarse energy exactly where the frame that has the object sharp has none. So the coarse levels collect that glow from one frame after another while the fine levels correctly take the sharp background, and the result has a soft rim.

With --halo-control P, the levels coarser than --depth-level (the guide, default 2) do not select. Each of them — the residual included — is the mean of the frames weighed by

((RE_guide + ε) / ρ) ^ P

that is, the guide level's region energy raised to hardness P and REDUCEd to the level's size. That is the Gaussian pyramid of a weight mask, exactly as a multiresolution spline blends with one. The coarse structure then follows the frames the guide found sharp, and where none is — a flat area — the frames average.

Checked on the halo in isolation: a sharp fine texture against the same texture blurred away with a wide soft bump added, which is coarse structure only, exactly as a defocused copy of a bright object is. Letting every level pick its own winner admits the bump at an RMS error above 0.05 of full scale; with the guide finding the sharp frame everywhere, the error falls by more than 85 % at every hardness from 1 to 8.

On a real stack — dark grapes on a bright table — the change is about 1 % of full scale along the silhouettes and invisible elsewhere. The halo is a coarse-level effect, and a deep stack of small focus steps leaves little of it to remove.

The streaming accumulator#

The fusion folds frames in one at a time. What it holds is:

• The running fused pyramid
• One best-energy plane per level (plus a weight sum at the levels halo control guides)
• The tiny residuals

Nothing grows with the number of frames. Frames are decoded on demand with a bounded read-ahead, aligned or not, folded, and dropped.

This is why stack length costs time but not memory. Before streaming, a 25-frame 45-megapixel CPU run took 56 s to align and 21 s to fuse at 31 GB peak, and 100 frames did not fit at all.

Where the depth map comes from#

Each frame's focus measure is taken as the frame is folded — the depth pass takes no second look at the frames. Its focus slice (45 MB per 45-megapixel frame on the default half grid) is kept, and that is the one thing that grows with the stack's length.

After the fusion, whose result becomes the guide image, the slices are turned into a dense, sub-frame depth map. See depth from focus.

What the parameters actually change#

A parameter sweep on the same aligned frames, against the default (3×3 window, 7 levels, --top de):

The energy window is the only knob that matters. Everything else is there for completeness and for pathological stacks.