FRAPPE v3 — image

@inproceedings{jacobellis2026frappe,
  title={FRAPPE: Full Input, Residual Output Autoencoding with Projection Pursuit Encoder},
  author={Jacobellis, Dan and Yadwadkar, Neeraja J.},
  note={under review},
  year={2026},
  url={https://ut-sysml.github.io/FRAPPE}
}

Project webpage · GitHub

What changes from v2 to v3:

• Entropy coding is the v3 two-stream lawcoder — per channel the persistent model is 6 scalars (bias μ; scale law a, b; shape law u, v); the quantile edges, canonical GGD-Huffman tables, and the zero-run mode’s trigger/order/exclusion tables are all generated from those scalars at load — no fitted tables persist. One global stream (first scale group, whole plane, no reset) plus one detail stream per macroregion (block-reset, run mode). Every macroregion is independently decodable given the global plane, and every operating point’s stream is a literal bit-prefix.

• SC6 companding (softsign, bits=6, affine='bounded') — the coder alphabet [-31, 31] is a structural guarantee (v2 used SC8 with ±127).

• Per-group decoder architecture (the multidec feature) — each operating point can have its own decoder depth/dim/kernel.

The codec is loaded from the Hugging Face hub through the compressors.frappe_v3_image public API — there is no local checkpoint or training code in this notebook.

import json, glob
import torch, numpy as np, matplotlib.pyplot as plt
import PIL.Image, datasets
from torchvision.transforms.v2.functional import pil_to_tensor, to_pil_image
from huggingface_hub import hf_hub_download
from compressors.frappe_v3_image import load_all_codecs, encode_to_latents
from compressors._frappev3.entropy_coding import rom_facts
from compressors._frappev3.ops import srgb_to_linear

Load the codec and its entropy coder

One MergedAutoencoder per operating point, all from the Hugging Face hub. Each operating point has its own merged decoder (the v3 multidec feature); the encoders freeze at merge, so every operating point’s encoder is a channel prefix of the full one. The entropy.json blob holds the lawcoder’s per-channel scalars; tables, edges, and run-mode configuration are generated from them when EntropyCoder loads the blob.

device = 'cpu'
config, models, coder = load_all_codecs(device=device)
ops = list(config.cumulative_channels)
model = models[ops[-1]]
rom = rom_facts(coder.blob)

print(f'modality      = {config.modality}  (n_groups={config.dim})')
print(f'channels      = {config.input_channels}')
print(f'ps ladder     = {config.ps} x {config.group_sizes} channels')
print(f'boundaries    = {ops}  (cumulative channels = operating points)')
print(f'decoder ps    = {config.decoder_ps}   decoder_dim = {config.decoder_dim}')
print(f'decoder depth = {config.decoder_depth}   layerscale = {config.decoder_layerscale}')
print(f'affine        = {config.affine}   linear_input = {config.linear_input}')
for s, (ps_s, start, end) in enumerate(model.scale_groups):
    print(f'  scale {s+1}: ps={ps_s:>2}  channels {start+1}-{end}  '
          f'(latent rate 1/{ps_s}^2 per pixel per channel)')
print(f'\nentropy coder: stream_mode={coder.mode}, macroregion={coder.macroregion} px')
print(f'  alphabet [-31, 31]   persistent model = {rom["model_scalars"]} scalars '
      f'({rom["blob_bytes_f32"]} B as f32) for {rom["n_channels"]} channels')
print(f'  generated at load: {rom["generated_tables_per_channel"]} tables/channel, '
      f'run mode on {rom["run_mode_channels"]} detail channels')

modality      = image  (n_groups=5)
channels      = 3
ps ladder     = [32, 16, 8, 4, 2] x [3, 6, 6, 6, 6] channels
boundaries    = [3, 9, 15, 21, 27]  (cumulative channels = operating points)
decoder ps    = [8, 8, 8, 8, 8]   decoder_dim = [768, 768, 768, 768, 768]
decoder depth = [12, 12, 24, 32, 32]   layerscale = [True, True, True, True, True]
affine        = True   linear_input = False
  scale 1: ps=32  channels 1-3  (latent rate 1/32^2 per pixel per channel)
  scale 2: ps=16  channels 4-9  (latent rate 1/16^2 per pixel per channel)
  scale 3: ps= 8  channels 10-15  (latent rate 1/8^2 per pixel per channel)
  scale 4: ps= 4  channels 16-21  (latent rate 1/4^2 per pixel per channel)
  scale 5: ps= 2  channels 22-27  (latent rate 1/2^2 per pixel per channel)

entropy coder: stream_mode=two_stream, macroregion=64 px
  alphabet [-31, 31]   persistent model = 162 scalars (648 B as f32) for 27 channels
  generated at load: 41 tables/channel, run mode on 24 detail channels

Analysis filterbank

One learned ps×ps projection per latent channel. The top row of each panel is the impulse response (the RGB filter taps, normalized to ±4σ per scale); the bottom row is the magnitude response (2-D DFT magnitude, averaged over the three input channels, zero frequency at the center).

N = 65
for s, (ps_s, start, end) in enumerate(model.scale_groups):
    W = model.encoders[s][0].weight.data.cpu()
    n_g = end - start
    sigma = W.std().item()
    fig, axes = plt.subplots(2, n_g, figsize=(1.4 * n_g, 3.0), dpi=120)
    axes = axes.reshape(2, n_g)
    for j in range(n_g):
        f = W[j]
        axes[0, j].imshow((f.permute(1, 2, 0) / (4 * sigma) + 0.5).clamp(0, 1).numpy())
        H = np.fft.fftshift(np.abs(np.fft.fft2(f.mean(0).numpy(), s=(N, N))))
        axes[1, j].imshow(H, cmap='inferno')
        axes[0, j].set_title(f'ch{start + j + 1}', fontsize=8)
        for ax in (axes[0, j], axes[1, j]):
            ax.axis('off')
    axes[0, 0].set_title(f'ch{start + 1}\nimpulse', fontsize=8)
    axes[1, 0].text(-0.25, 0.5, '|H(f)|', transform=axes[1, 0].transAxes,
                    rotation=90, va='center', fontsize=8)
    fig.suptitle(f'scale ps={ps_s}  (channels {start + 1}-{end})', fontsize=10)
    plt.tight_layout()
    plt.show()

Load an example image

A Kodak image, mapped to the codec convention [-1, 1], cropped to a multiple of the macroregion size (a no-op at Kodak’s native sizes). linear_input=false, so pixels are fed in sRGB. Kodak is eval-only — nothing in the codec or coder was trained or fit on it.

dataset = datasets.load_dataset('danjacobellis/kodak', split='validation')
img = dataset[22]['image'].convert('RGB')
x = pil_to_tensor(img).to(torch.float).to(device).unsqueeze(0) / 127.5 - 1.0
unit = coder.macroregion
x = x[..., :unit * (x.shape[2] // unit), :unit * (x.shape[3] // unit)]
x_in = srgb_to_linear(x) if config.linear_input else x
n_pixels = x.shape[2] * x.shape[3]
x_01 = x / 2 + 0.5
print(f'input {tuple(x.shape)}  ({n_pixels} pixels = '
      f'{x.shape[2] // unit} x {x.shape[3] // unit} macroregions of {unit}x{unit}),  '
      f'range [{x.min():.2f}, {x.max():.2f}]')
display(to_pil_image(x_01[0].cpu().clamp(0, 1)))

input (1, 3, 512, 768)  (393216 pixels = 8 x 12 macroregions of 64x64),  range [-1.00, 1.00]

Analysis transform

with torch.no_grad():
    latents = model.encode(x_in)
n_latent_values = sum(z.numel() for z in latents)
print(f'{x_in.numel()} input values -> {n_latent_values} latent values '
      f'({n_latent_values / x_in.numel():.2f}x)')
for s, z in enumerate(latents):
    print(f'  scale {s+1} (ps={model.scale_groups[s][0]:>2}): latent {tuple(z.shape)}  '
          f'range [{z.min():.1f}, {z.max():.1f}]')

1179648 input values -> 784512 latent values (0.67x)
  scale 1 (ps=32): latent (1, 3, 16, 24)  range [-15.1, 13.1]
  scale 2 (ps=16): latent (1, 6, 32, 48)  range [-8.2, 8.1]
  scale 3 (ps= 8): latent (1, 6, 64, 96)  range [-10.9, 10.0]
  scale 4 (ps= 4): latent (1, 6, 128, 192)  range [-11.4, 10.5]
  scale 5 (ps= 2): latent (1, 6, 256, 384)  range [-18.4, 13.5]

Companding

Softsign compander curves per latent channel (SC6: bits=6, affine='bounded' — a learnable per-channel scale, itself softsign-bounded to (−1, 1)). The softsign asymptote makes the coder alphabet bound ±31 (dotted lines) structural: companded values cannot leave [-31, 31] no matter what training does.

ramp = torch.linspace(-600, 600, 2401, device=device)
plt.figure(figsize=(5.5, 3.5), dpi=130)
colors = plt.cm.viridis(np.linspace(0, 1, len(model.scale_groups)))
for s, (ps_s, start, end) in enumerate(model.scale_groups):
    Cg = end - start
    with torch.no_grad():
        curves = model.encoders[s][1](ramp.view(1, 1, -1, 1).expand(1, Cg, -1, 1))
    plt.plot(ramp.cpu(), curves[0, :, :, 0].cpu().T, lw=0.5, color=colors[s])
    plt.plot([], [], color=colors[s], label=f'ps={ps_s}')
plt.axhline(31, ls=':', c='k', lw=0.6); plt.axhline(-31, ls=':', c='k', lw=0.6)
plt.xlabel('latent value'); plt.ylabel('companded value')
plt.title('softsign compander curves (dotted: coder alphabet bound +/-31)')
plt.legend(fontsize=7); plt.tight_layout(); plt.show()

Rounding (to the coder’s [-31, 31] grid)

z_c = torch.cat([z.flatten() for z in latents])
n_clip = int((z_c.round().abs() > 31).sum())
latents_q = [z.round().clamp(-31, 31) for z in latents]
z_q = torch.cat([z.flatten() for z in latents_q])
qsnr = 10 * (z_c.pow(2).mean() / (z_c - z_q).pow(2).mean()).log10().item()
print(f'integer latent range:    [{int(z_q.min())}, {int(z_q.max())}]   (clipped: {n_clip}/{z_c.numel()})')
print(f'latent quantization SNR: {qsnr:.2f} dB')

plt.figure(figsize=(5, 2), dpi=120)
plt.hist(z_q.cpu().numpy(), range=(-31.5, 31.5), bins=63, width=0.85)
plt.xlim([-15, 15]); plt.title('histogram of integer latents (zoomed)')
plt.xlabel('value'); plt.ylabel('count'); plt.tight_layout(); plt.show()

latents_np = [z[0].to(torch.int8).cpu().numpy() for z in latents_q]

integer latent range:    [-18, 14]   (clipped: 0/784512)
latent quantization SNR: 24.09 dB

Quantized latents (prior to entropy coding)

Every channel group is a multiple of 3 channels, so consecutive channel triples display directly as RGB images at the latent’s native resolution — the coder alphabet [-31, 31] mapped linearly to [0, 255] (0 → mid-gray). The image sizes convey the dimensionality per scale: a ps-32 channel holds 1/32² values per pixel, ps-16 1/16², ps-8 1/8².

for s, (ps_s, start, end) in enumerate(model.scale_groups):
    z = latents_np[s]                                     # (C_scale, H_s, W_s) int8 in [-31, 31]
    for j in range(0, z.shape[0], 3):
        rgb = ((z[j:j+3].astype(np.int16) + 31) * 255 // 62).astype(np.uint8)
        print(f'ps={ps_s}  channels {start + j + 1}-{start + j + 3}  ({rgb.shape[2]}x{rgb.shape[1]} native)')
        display(PIL.Image.fromarray(rgb.transpose(1, 2, 0)))

ps=32  channels 1-3  (24x16 native)

ps=16  channels 4-6  (48x32 native)

ps=16  channels 7-9  (48x32 native)

ps=8  channels 10-12  (96x64 native)

ps=8  channels 13-15  (96x64 native)

ps=4  channels 16-18  (192x128 native)

ps=4  channels 19-21  (192x128 native)

ps=2  channels 22-24  (384x256 native)

ps=2  channels 25-27  (384x256 native)

Two-stream lawcoder entropy coding

coder.encode produces one global stream (the first scale group over the whole plane, no reset, one byte-pad) plus one detail stream per macroregion (block-reset, zero-run mode, one byte-pad each). Per sample, the two laws map the causal statistic s to a scale estimate and a shape; the generated grid picks a canonical GGD-Huffman table. Zero signaling: the decoder recomputes every table choice from already-decoded values. The stream for operating point n is the literal bit-prefix covering channels ≤ n.

enc = coder.encode(latents_np)
print(f'global stream:  {len(enc.gblob):,} bytes  (channels 1-{coder.n_coarse}, whole plane, one byte-pad)')
if enc.streams:
    region_bytes = (enc.prefix[:, -1] + 7) // 8
    print(f'detail streams: {len(enc.streams)} macroregions, one byte-pad each; at max detail '
          f'{int(region_bytes.sum()):,} bytes total, per region min/median/max = '
          f'{int(region_bytes.min())}/{int(np.median(region_bytes))}/{int(region_bytes.max())}')
else:
    region_bytes = None
print(f'\n{"op":>4} {"stream bits":>12} {"bpp":>9} {"CR":>8} {"bits/latent":>12}')
for n in ops:
    bits = coder.op_bits(enc, n)
    n_lat_n = sum(min(max(n - start, 0), end - start) * z.shape[1] * z.shape[2]
                  for z, (ps_s, start, end) in zip(latents_np, model.scale_groups))
    print(f'{n:>4} {bits:>12,} {bits / n_pixels:>9.4f} {24 * n_pixels / bits:>7.1f}x {bits / n_lat_n:>12.3f}')

global stream:  554 bytes  (channels 1-3, whole plane, one byte-pad)
detail streams: 96 macroregions, one byte-pad each; at max detail 125,667 bytes total, per region min/median/max = 595/1213/3077

  op  stream bits       bpp       CR  bits/latent
   3        4,432    0.0113  2129.3x        3.847
   9       23,704    0.0603   398.1x        2.286
  15       81,792    0.2080   115.4x        1.732
  21      251,056    0.6385    37.6x        1.290
  27    1,009,768    2.5680     9.3x        1.287

Round-trip verification

coder.verify decodes the global stream from its bytes alone, then every macroregion from (decoded global plane slice, its own bytes) only — structural isolation — at every operating point’s literal bit-prefix, and asserts the latents bit-exact.

coder.verify(enc, latents_np, ops)
print(f'round trip verified at ops {ops}: global stream decoded from its bytes alone; every macroregion')
print('decoded from (decoded global plane slice, own bytes) only; op streams are literal bit-prefixes')

round trip verified at ops [3, 9, 15, 21, 27]: global stream decoded from its bytes alone; every macroregion
decoded from (decoded global plane slice, own bytes) only; op streams are literal bit-prefixes

Decoding

The full-rate reconstruction from the integer latents. The decoder takes the per-scale int8 latents directly (no byte-stream round-trip needed — the latents are the codec’s internal representation).

with torch.no_grad():
    xhat = model.decode([torch.from_numpy(z).unsqueeze(0).to(device) for z in latents_np]).clamp(-1, 1)
xhat_01 = xhat / 2 + 0.5
psnr = -10 * torch.nn.functional.mse_loss(x_01, xhat_01).log10().item()
print(f'bpp  = {coder.op_bits(enc, ops[-1]) / n_pixels:.4f}')
print(f'PSNR = {psnr:.2f} dB   ({ops[-1]} ch)')
display(to_pil_image(xhat_01[0].cpu().clamp(0, 1)))

bpp  = 2.5680
PSNR = 43.15 dB   (27 ch)

Per-macroregion rate map

Each macroregion carries its own byte-padded stream, so the spatial rate allocation is directly visible — and any macroregion could be transmitted or upgraded independently.

if region_bytes is not None:
    nuh, nuw = x.shape[2] // unit, x.shape[3] // unit
    fig, axes = plt.subplots(1, 2, figsize=(9, 3.6), dpi=120)
    axes[0].imshow(x_01[0].permute(1, 2, 0).cpu().clamp(0, 1).numpy())
    axes[0].set_title('input'); axes[0].axis('off')
    im = axes[1].imshow(region_bytes.reshape(nuh, nuw), cmap='viridis')
    axes[1].set_title(f'detail-stream bytes per {unit}x{unit} macroregion')
    axes[1].axis('off')
    fig.colorbar(im, ax=axes[1], fraction=0.046)
    plt.tight_layout(); plt.show()
else:
    print('no detail streams (single channel group) — nothing to map yet')

Reconstructions at each operating point

One encode serves every operating point: the latents are channel-prefix-truncated at each group boundary and decoded with that point’s own merged decoder; the rate is the same encode’s literal bit-prefix. A fixed 256×256 crop of the original (left) and the reconstruction (right) per point.

ch_, cw_ = 256, 256
ct = (x.shape[2] - ch_) // 2
cl = (x.shape[3] - cw_) // 2
orig_crop = to_pil_image(x_01[0, :, ct:ct+ch_, cl:cl+cw_].cpu().clamp(0, 1))

rd_img = []
for n in ops:
    partial = models[n]
    lts = [torch.from_numpy(latents_np[s][:end - start]).unsqueeze(0).to(device)
           for s, (ps_s, start, end) in enumerate(partial.scale_groups)]
    with torch.no_grad():
        xh = partial.decode(lts).clamp(-1, 1)
    bpp_n = coder.op_bits(enc, n) / n_pixels
    psnr_n = -10 * torch.nn.functional.mse_loss(x_01, xh / 2 + 0.5).log10().item()
    rd_img.append((bpp_n, psnr_n, n))
    print(f'{n:>2} ch:  bpp={bpp_n:.4f}   CR={24 / bpp_n:7.1f}x   PSNR={psnr_n:.2f} dB')
    recon_crop = to_pil_image((xh[0, :, ct:ct+ch_, cl:cl+cw_] / 2 + 0.5).cpu().clamp(0, 1))
    pair = PIL.Image.new('RGB', (2 * cw_ + 4, ch_), 'white')
    pair.paste(orig_crop, (0, 0)); pair.paste(recon_crop, (cw_ + 4, 0))
    display(pair)

 3 ch:  bpp=0.0113   CR= 2129.3x   PSNR=23.79 dB

 9 ch:  bpp=0.0603   CR=  398.1x   PSNR=27.60 dB

15 ch:  bpp=0.2080   CR=  115.4x   PSNR=32.30 dB

21 ch:  bpp=0.6385   CR=   37.6x   PSNR=37.19 dB

27 ch:  bpp=2.5680   CR=    9.3x   PSNR=43.15 dB

Rate–distortion

The curve below overlays this image’s operating points on the published Kodak validation average from rd.json (rates round-trip verified by the trainer’s final pass), against the FRAPPE v1 and v2 baselines from the results/ directory (with their JPEG-LS accounting).

rd_path = hf_hub_download(repo_id='danjacobellis/FRAPPEv3', filename='image/rd.json')
rd3 = json.load(open(rd_path))
val_bpp  = [p['bpp'] for p in rd3['rd_curve']]
val_psnr = [p['psnr'] for p in rd3['rd_curve']]

def load_baseline(pattern):
    path = sorted(glob.glob(pattern))[-1]
    d = json.load(open(path))
    key = 'channel_counts' if 'channel_counts' in d else 'quality_values'
    pts = [(d['results'][str(q)]['mean']['bpp'], d['results'][str(q)]['mean']['PSNR_dB'])
           for q in d[key]]
    return [p[0] for p in pts], [p[1] for p in pts]

v1_bpp, v1_psnr = load_baseline('results/frappe/rate_distortion_*.json')
v2_bpp, v2_psnr = load_baseline('results/frappe_v2_image/rate_distortion_*.json')

plt.figure(figsize=(6.5, 4), dpi=120)
plt.semilogx(v1_bpp, v1_psnr, '-', color='0.75', lw=1.0,
             label='FRAPPE v1 + JPEG-LS')
plt.semilogx(v2_bpp, v2_psnr, '^-', color='black', ms=5, lw=1.2, alpha=0.8,
             label='FRAPPE v2 + JPEG-LS')
plt.semilogx(val_bpp, val_psnr, 'o-', color='tab:blue',
             label='FRAPPE v3 — Kodak average (verified streams)')
plt.semilogx([p[0] for p in rd_img], [p[1] for p in rd_img], 's--', color='tab:blue',
             alpha=0.5, label='FRAPPE v3 — this image')
for n, bv, psv in zip(ops, val_bpp, val_psnr):
    plt.annotate(f'{n}ch', (bv, psv), fontsize=7, color='tab:blue',
                 xytext=(3, -9), textcoords='offset points')
lo = min(min(val_bpp), v1_bpp[0]) / 1.5
hi = max(max(val_bpp), max(v2_bpp)) * 2
plt.xlim(lo, hi)
plt.xlabel('bpp'); plt.ylabel('PSNR (dB)')
plt.title('FRAPPE v3 vs v1/v2 — Kodak')
plt.grid(True, alpha=0.3); plt.legend(fontsize=8); plt.tight_layout(); plt.show()

print('v3 Kodak average (rd.json, final verification pass; stream bits only):')
for n, bv, psv in zip(ops, val_bpp, val_psnr):
    print(f'  {n:>2} ch:  bpp={bv:.4f}   CR={24 / bv:7.1f}x   PSNR={psv:.2f} dB')

v3 Kodak average (rd.json, final verification pass; stream bits only):
   3 ch:  bpp=0.0107   CR= 2239.7x   PSNR=21.58 dB
   9 ch:  bpp=0.0643   CR=  373.4x   PSNR=25.09 dB
  15 ch:  bpp=0.2635   CR=   91.1x   PSNR=28.75 dB
  21 ch:  bpp=0.9433   CR=   25.4x   PSNR=33.73 dB
  27 ch:  bpp=3.6646   CR=    6.5x   PSNR=41.92 dB