Speech compressors on LibriTTS test-clean#
Side-by-side rate-distortion plots for the speech codecs in this repo, sourced from the JSONs hardcoded below.
Evaluation set. danjacobellis/libritts, split validation = LibriTTS test-clean (4,837 utterances, 39 speakers, 24 kHz mono, CC BY 4.0 — the same rows as the upstream split, re-hosted). A fixed subset of N = 200 utterances is selected deterministically by compressors.speech_eval (header duration ≥ 4 s, random.Random(20260829).sample, ascending row order), each cropped to its first 10 s (shorter utterances keep their length), resampled once to 16 kHz and cropped to a 256-sample multiple (the E2A2 speech macroregion, so every codec sees identical samples). The selected record ids are stored in every JSON’s config.clip_selection.
Reference domain. The 16 kHz mono signal, zero-mean and peak-normalized to the harness’s [-0.5, 0.5] convention, is the reference for every codec. Codecs that run at another rate resample at their own boundary and are charged only their own bits: Mimi 16 → 24 → 16 kHz, Opus 16 → 48 → 16 kHz (libopus through torchcodec / FFmpeg); E2A2 runs natively at 16 kHz. Constant delays were checked on the first clip at the top operating point of each codec (all zero).
Metrics. Rate in kbps of the 16 kHz stream (bits / duration). PSNR on the [0, 1]-mapped waveforms (-10·log10(MSE), the same convention as the other audio harnesses). ViSQOL = Google’s ViSQOL v3 in speech mode (MOS-LQO, via the speequal package: scaled MOS mapping with the TFLite lattice NSIM→MOS mapper, whose ceiling for an identical pair is ≈ 4.5–4.65, not 5.0). Rate accounting: E2A2 = the literal GGDLPC stream prefix per operating point (every op decoded from the truncated bytes); Mimi = 11 bits × codes actually returned; Opus = the whole Ogg container.
Codecs. e2a2:clean_speech (ours; ops 4 / 10 / 16 / 28 / 36 channels), e2a2:denoising_speech fed clean speech (ops 4 / 12 / 20 / 32), transformers:mimi (kyutai/mimi, 1–32 RVQ quantizers), torchcodec:opus (target 6–64 kbps, VBR — the reported rate is the measured bytes).
import json
from pathlib import Path
import matplotlib.pyplot as plt
plt.rcParams["font.family"] = "serif"
RD_PATHS = {
"e2a2_clean": "results/speech_e2a2_clean/rate_distortion_1788277406.json",
"e2a2_denoising": "results/speech_e2a2_denoising/rate_distortion_1788030218.json",
"mimi": "results/speech_mimi/rate_distortion_1788030223.json",
"opus": "results/speech_opus/rate_distortion_1788030219.json",
}
AXIS_LABEL = {
"kbps": "Rate [kbps]",
"PSNR_dB": "PSNR [dB]",
"ViSQOL": "ViSQOL [MOS-LQO]",
}
TITLE_LABEL = {
"kbps": "Rate",
"PSNR_dB": "PSNR",
"ViSQOL": "ViSQOL",
}
LOG_KEYS = ("kbps",)
def load_codec(name):
rd = json.loads(Path(RD_PATHS[name]).read_text())
fs = rd["sample_rate"]
points = []
for k in rd["quality_values"]:
rm = rd["results"][str(k)]["mean"]
points.append({
"sweep": k,
"kbps": rm["kbps"],
"bits_per_sample": rm["kbps"] * 1000.0 / fs,
"PSNR_dB": rm["PSNR_dB"],
"ViSQOL": rm["ViSQOL"],
})
return points
def col(pts, key):
return [p[key] for p in pts]
def plot_panel(group, y_key, x_key, legend=True):
"""One small panel: y vs x for every codec in `group`.
The rate axis is log-scaled wherever it appears.
"""
plt.figure(figsize=(4.5, 4), dpi=180)
ax = plt.gca()
if x_key in LOG_KEYS:
ax.set_xscale("log")
if y_key in LOG_KEYS:
ax.set_yscale("log")
for c in group:
ax.plot(col(c["data"], x_key), col(c["data"], y_key),
marker=c.get("marker", "."),
linestyle=c.get("linestyle", "-"),
color=c.get("color"), label=c["name"])
ax.set_xlabel(AXIS_LABEL[x_key])
ax.set_ylabel(AXIS_LABEL[y_key])
ax.set_title(f"{TITLE_LABEL[y_key]} vs {TITLE_LABEL[x_key]} (LibriTTS test-clean)")
if legend:
ax.legend(loc="best")
log_either = (x_key in LOG_KEYS) or (y_key in LOG_KEYS)
ax.grid(True, which="both" if log_either else "major", alpha=0.4)
plt.tight_layout()
plt.show()
# Comment / uncomment to control which codecs appear on the plots
CODECS = [
{"name": "Opus", "data": load_codec("opus"), "color": "black", "linestyle": "-", "marker": "."},
{"name": "Mimi", "data": load_codec("mimi"), "color": "orange", "linestyle": "-", "marker": "."},
{"name": "E2A2 clean speech", "data": load_codec("e2a2_clean"), "color": "tab:blue", "linestyle": "-", "marker": "."},
{"name": "E2A2 denoising (fed clean)", "data": load_codec("e2a2_denoising"), "color": "tab:cyan", "linestyle": "--", "marker": "."},
]
for c in CODECS:
kb = col(c["data"], "kbps")
print(f"{c['name']:>27}: {len(c['data'])} points,"
f" kbps [{min(kb):.3f}, {max(kb):.3f}],"
f" PSNR [{min(col(c['data'],'PSNR_dB')):.2f}, {max(col(c['data'],'PSNR_dB')):.2f}] dB,"
f" ViSQOL [{min(col(c['data'],'ViSQOL')):.2f}, {max(col(c['data'],'ViSQOL')):.2f}]")
Opus: 8 points, kbps [6.447, 85.466], PSNR [28.80, 48.46] dB, ViSQOL [1.76, 4.50]
Mimi: 6 points, kbps [0.138, 4.415], PSNR [22.36, 34.19] dB, ViSQOL [1.02, 3.46]
E2A2 clean speech: 5 points, kbps [0.859, 26.294], PSNR [28.91, 46.43] dB, ViSQOL [1.30, 2.25]
E2A2 denoising (fed clean): 4 points, kbps [0.740, 7.078], PSNR [27.21, 35.67] dB, ViSQOL [1.28, 1.73]
PSNR vs rate; every codec sweeps its rate knob (E2A2: channel-truncation operating points; Mimi: number of RVQ quantizers; Opus: target bit rate).
plot_panel(CODECS, "PSNR_dB", "kbps")
ViSQOL (speech mode, lattice mapper) vs rate on the same operating points.
plot_panel(CODECS, "ViSQOL", "kbps", legend=False)
Operating points#
One row per operating point, sorted by rate.
rows = []
for c in CODECS:
for p in c["data"]:
rows.append((p["kbps"], c["name"], p["sweep"], p["bits_per_sample"], p["PSNR_dB"], p["ViSQOL"]))
rows.sort()
print(f"{'codec':>27} | {'knob':>5} | {'kbps':>7} | {'bits/sample':>11} | {'PSNR dB':>7} | {'ViSQOL':>6}")
print("-" * 82)
for kbps, name, knob, bps, psnr, vq in rows:
print(f"{name:>27} | {str(knob):>5} | {kbps:7.3f} | {bps:11.4f} | {psnr:7.2f} | {vq:6.3f}")
codec | knob | kbps | bits/sample | PSNR dB | ViSQOL
----------------------------------------------------------------------------------
Mimi | 1 | 0.138 | 0.0086 | 22.36 | 1.016
Mimi | 2 | 0.276 | 0.0172 | 23.48 | 1.338
Mimi | 4 | 0.552 | 0.0345 | 26.10 | 1.668
E2A2 denoising (fed clean) | 4 | 0.740 | 0.0462 | 27.21 | 1.280
E2A2 clean speech | 4 | 0.859 | 0.0537 | 28.91 | 1.476
Mimi | 8 | 1.104 | 0.0690 | 29.36 | 2.416
Mimi | 16 | 2.208 | 0.1380 | 32.03 | 3.087
E2A2 denoising (fed clean) | 12 | 2.508 | 0.1567 | 32.35 | 1.549
E2A2 clean speech | 10 | 2.566 | 0.1604 | 33.72 | 1.303
E2A2 denoising (fed clean) | 20 | 3.903 | 0.2439 | 34.55 | 1.470
Mimi | 32 | 4.415 | 0.2760 | 34.19 | 3.461
E2A2 clean speech | 16 | 4.708 | 0.2942 | 36.44 | 1.889
Opus | 6 | 6.447 | 0.4029 | 28.80 | 1.764
E2A2 denoising (fed clean) | 32 | 7.078 | 0.4424 | 35.67 | 1.735
Opus | 8 | 8.281 | 0.5175 | 29.90 | 2.291
Opus | 12 | 12.403 | 0.7752 | 36.87 | 3.922
E2A2 clean speech | 28 | 17.150 | 1.0719 | 42.67 | 1.932
Opus | 16 | 17.932 | 1.1207 | 38.03 | 4.008
Opus | 24 | 25.826 | 1.6141 | 40.63 | 4.121
E2A2 clean speech | 36 | 26.294 | 1.6434 | 46.43 | 2.254
Opus | 32 | 33.787 | 2.1117 | 42.21 | 4.171
Opus | 48 | 49.712 | 3.1070 | 44.43 | 4.237
Opus | 64 | 85.466 | 5.3416 | 48.46 | 4.495
Listening#
One fixed clip — the first of the selected set, test-clean/1089/134686/1089_134686_000002_000003.wav — as the 16 kHz reference and as each codec’s reconstruction at its operating point nearest ~1.1 kbps (Mimi’s native 8-quantizer point) and nearest ~6 kbps, re-encoded here through the same harness functions. Each is shown as a log-mel spectrogram with an audio player. Opus’s lowest target (6 kbps) is its nearest point to both targets, so it appears once.
import torch, torchaudio, numpy as np
from IPython.display import Audio, display
from torchvision.transforms import ToPILImage
from compressors import speech_eval, e2a2
from compressors.audio_eval import psnr_db
from compressors.mimi.evaluate_rate_distortion import load_model as load_mimi, process_clip as mimi_clip
from compressors.torchcodec.evaluate_rate_distortion import encode_decode as opus_clip
device = "cuda:0" if torch.cuda.is_available() else "cpu"
fs = speech_eval.SPEECH_SAMPLE_RATE
CLIP_INDEX = 0
clips = speech_eval.load_libritts_clips(CLIP_INDEX + 1, verbose=False)
record = clips.provenance["clip_selection"]["records"][CLIP_INDEX]
x = speech_eval.normalize_speech(clips[CLIP_INDEX]) # (1, L) in [-0.5, 0.5]
n_samples = x.shape[-1]
print(record, f"{n_samples / fs:.2f} s")
mel = torchaudio.transforms.MelSpectrogram(sample_rate=fs, n_fft=1024, hop_length=256, n_mels=80)
def make_spectrogram(x_1ch):
S = (mel(x_1ch) + 1e-8).log()
S = ((S - S.mean()) / (3 * S.std()) + 0.5).clamp(0, 1)
return ToPILImage()(S.flip(0)).resize((600, 160))
def nearest(name, target_kbps):
pts = load_codec(name)
return min(pts, key=lambda p: abs(p["kbps"] - target_kbps))["sweep"]
TARGETS = (1.1, 6.0)
print({name: [nearest(name, t) for t in TARGETS] for name in RD_PATHS})
print("reference")
display(make_spectrogram(x[0]))
Audio((2 * x).numpy(), rate=fs)
test-clean/1089/134686/1089_134686_000002_000003.wav 10.00 s
{'e2a2_clean': [4, 16], 'e2a2_denoising': [4, 32], 'mimi': [8, 32], 'opus': [6, 6]}
reference
def show(label, x_hat, bits):
x_hat = x_hat[..., :n_samples].float()
kbps = bits / (n_samples / fs) / 1000
print(f"{label}: {kbps:.3f} kbps, PSNR {psnr_db(x, x_hat):.2f} dB, ViSQOL {speech_eval.visqol_speech(x, x_hat):.3f}")
display(make_spectrogram(x_hat[0]))
display(Audio((2 * x_hat).numpy(), rate=fs))
E2A2#
for name, key in (("clean_speech", "e2a2_clean"), ("denoising_speech", "e2a2_denoising")):
ops = sorted({nearest(key, t) for t in TARGETS})
config, weights, coder, all_ops = e2a2.load_from_hub(name, device=device)
top = e2a2.load_codec(name, all_ops[-1], device, config=config, weights=weights)
latents_q = e2a2.encode_to_latents(top, (2 * x).unsqueeze(0).to(device))
enc = e2a2.encode_latents(coder, latents_q)
for n in ops:
model = top if n == all_ops[-1] else e2a2.load_codec(name, n, device, config=config, weights=weights)
lts = e2a2.decode_latents(coder, enc, n)
with torch.inference_mode():
x_hat = model.decode([z.to(device) for z in lts]).clamp(-1, 1)[0].float().cpu() / 2
show(f"E2A2 {name} op {n}", x_hat, e2a2.op_bits(coder, enc, n))
del top, model, weights
torch.cuda.empty_cache()
[tflite_quality_mapper.cc : 32] RAW: Loading TF Lattice TFLite model at /home/dgj335/g/lib/python3.12/site-packages/speequal/visqol/model/lattice_tcditugenmeetpackhref_ls2_nl60_lr12_bs2048_learn.005_ep2400_train1_7_raw.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
E2A2 clean_speech op 4: 0.828 kbps, PSNR 30.57 dB, ViSQOL 1.322
E2A2 clean_speech op 16: 4.640 kbps, PSNR 39.38 dB, ViSQOL 1.739
[tflite_quality_mapper.cc : 32] RAW: Loading TF Lattice TFLite model at /home/dgj335/g/lib/python3.12/site-packages/speequal/visqol/model/lattice_tcditugenmeetpackhref_ls2_nl60_lr12_bs2048_learn.005_ep2400_train1_7_raw.tflite
E2A2 denoising_speech op 4: 0.714 kbps, PSNR 27.66 dB, ViSQOL 1.000
[tflite_quality_mapper.cc : 32] RAW: Loading TF Lattice TFLite model at /home/dgj335/g/lib/python3.12/site-packages/speequal/visqol/model/lattice_tcditugenmeetpackhref_ls2_nl60_lr12_bs2048_learn.005_ep2400_train1_7_raw.tflite
[tflite_quality_mapper.cc : 32] RAW: Loading TF Lattice TFLite model at /home/dgj335/g/lib/python3.12/site-packages/speequal/visqol/model/lattice_tcditugenmeetpackhref_ls2_nl60_lr12_bs2048_learn.005_ep2400_train1_7_raw.tflite
E2A2 denoising_speech op 32: 6.310 kbps, PSNR 37.58 dB, ViSQOL 1.550
Mimi#
mimi = load_mimi(device, torch.float32)
for nq in sorted({nearest("mimi", t) for t in TARGETS}):
x_hat, bits = mimi_clip(mimi, device, torch.float32, x, nq, sample_rate=fs)
show(f"Mimi num_quantizers={nq}", x_hat, bits)
del mimi
torch.cuda.empty_cache()
Skipping import of cpp extensions due to incompatible torch version 2.11.0+cu130 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
Mimi num_quantizers=8: 1.100 kbps, PSNR 31.41 dB, ViSQOL 2.638
[tflite_quality_mapper.cc : 32] RAW: Loading TF Lattice TFLite model at /home/dgj335/g/lib/python3.12/site-packages/speequal/visqol/model/lattice_tcditugenmeetpackhref_ls2_nl60_lr12_bs2048_learn.005_ep2400_train1_7_raw.tflite
Mimi num_quantizers=32: 4.400 kbps, PSNR 37.47 dB, ViSQOL 3.606
[tflite_quality_mapper.cc : 32] RAW: Loading TF Lattice TFLite model at /home/dgj335/g/lib/python3.12/site-packages/speequal/visqol/model/lattice_tcditugenmeetpackhref_ls2_nl60_lr12_bs2048_learn.005_ep2400_train1_7_raw.tflite
Opus#
for br in sorted({nearest("opus", t) for t in TARGETS}):
x_hat, bits = opus_clip(x, "opus", br, sample_rate=fs)
show(f"Opus target {br} kbps", x_hat, bits)
Opus target 6 kbps: 6.374 kbps, PSNR 29.78 dB, ViSQOL 1.869
[tflite_quality_mapper.cc : 32] RAW: Loading TF Lattice TFLite model at /home/dgj335/g/lib/python3.12/site-packages/speequal/visqol/model/lattice_tcditugenmeetpackhref_ls2_nl60_lr12_bs2048_learn.005_ep2400_train1_7_raw.tflite
