Lightweight, Versatile Codec Design

Lightweight, Versatile Codec Design#

Modern sensors generate rich, high-fidelity data, yet applications operating on wearable or remote sensing devices remain constrained by bandwidth and power budgets. Standardized codecs such as JPEG and MPEG achieve efficient trade-offs between bitrate and perceptual quality but are designed for human perception, limiting their applicability to machine-perception tasks and non-traditional modalities such as spatial audio arrays, hyperspectral images, and 3D medical images. General-purpose compression schemes based on scalar quantization or resolution reduction are broadly applicable but fail to exploit inherent signal redundancies, resulting in suboptimal rate-distortion performance. Recent generative neural codecs, or tokenizers, model complex signal dependencies but are often over-parameterized, data-hungry, and modality-specific, making them impractical for resource-constrained environments. We introduce a Lightweight, Versatile, and Asymmetric neural codec architecture (LiVeAction), that addresses these limitations through two key ideas. (1) To reduce the complexity of the encoder to meet the resource constraints of the execution environments, we impose an FFT-like structure and reduce the overall size and depth of the neural-network-based analysis transform. (2) To allow arbitrary signal modalities and simplify training, we replace adversarial and perceptual losses with a variance-based rate penalty. Our design produces codecs that deliver superior rate-distortion performance compared to state-of-the-art generative tokenizers, while remaining practical for deployment on low-power sensors. We release our code, experiments, and python library at UT-SysML/liveaction.

Introduction#

Modern sensors—from wearables and medical devices to satellites—generate rich streams of high-resolution data (Engel and others, 2023, Zhang et al., 2022). Efficient compression is critical for applications in health monitoring, remote sensing, and autonomous systems, as these deployments operate under strict power and bandwidth constraints. Standardized codecs (JPEG and MPEG) provide strong bitrate–quality trade-offs at low computational cost, but their human-centric design makes them unsuitable for machine-perception tasks and non-standard modalities where perceptual quality is not the target (Jacobellis and Yadwadkar, 2025).

General-purpose methods, such as scalar quantization (Davisson, 1968) and resolution reduction (Kortman, 1967) remain widely used for their simplicity and universality. They apply to arbitrary signals, provide analytical guarantees on information loss, and combine easily with domain-specific approaches (Chen et al., 2025, Khani et al., 2021). But, being agnostic to real-world data, they fail to exploit inherent redundancies, leading to poor rate–distortion performance (Davisson, 1972).

Recent advances in deep neural network (DNN)–based autoencoders (Ballé et al., 2017, He et al., 2022) and generative codecs (Agarwal et al., 2025, Ramesh et al., 2021) show that data-driven models can capture complex signal dependencies, greatly improving compression efficiency and realism. These tokenizer-style codecs use learned transforms and perceptual losses to reconstruct high-quality outputs at low bitrates but remain impractical for resource-constrained settings. Their deep, wide encoders dominate computational cost, and their architectures are often modality-specific. Additionally, generative codecs often depend on perceptual or adversarial losses tuned to human perception, making them ill-suited for scientific or machine-perception tasks. Such objectives are undefined for many signal types and can destabilize training, preventing these models from serving as general-purpose codecs, especially in low-power or embedded settings.

To address these limitations, we propose LiVeAction, a Lightweight, Versatile, and Asymmetric neural codec designed to achieve efficient, high-fidelity compression across diverse signal modalities. LiVeAction is built to meet three primary goals: (1) extreme computational encoding efficiency, (2) competitive rate–distortion performance, and (3) versatility across signal modalities.

../_images/rate_distortion_complexity.png

Fig. 6 Rate-distortion-complexity trade-off for RGB images measured on the kodak dataset. BD-rate is averaged between SSIM and DISTS (Ding et al., 2020). Throughput is measured on a low power mobile CPU (Intel Arrow Lake 255U).#

Extreme computational encoding efficiency. Real-time sensing on mobile or remote platforms demands encoders that are computationally efficient and power-conscious. Most neural autoencoders use symmetric architectures, where analysis and synthesis transforms share nearly identical DNN layers (Agarwal et al., 2025, Evans et al., 2025). However, increasing encoder depth or width yields diminishing returns (Hansen-Estruch et al., 2025). LiVeAction adopts an asymmetric design with a lightweight encoder that minimizes computation while preserving representational quality. LiVeAction improves efficiency using structured, FFT-inspired operations instead of dense projections. These impose a block-diagonal structure reminiscent of ShuffleNet (Zhang et al., 2018) and Monarch matrices (Dao et al., 2022, Fu et al., 2023), allowing multiple layers with alternating nonlinear activations at roughly the cost of one dense layer.

Competitive rate-distortion performance. To enable applications with severe bandwidth limitations, the rate-distortion performance must match or exceed conventional standards like JPEG or MPEG. Existing autoencoder designs (e.g. Stable diffusion (Rombach et al., 2022), Stable Audio (Evans et al., 2025), and Cosmos (Agarwal et al., 2025)) rely heavily on perceptual and adversarial losses, enabling the decoder to synthesize realistic, but hallucinated details. Prior work shows that removing these losses can improve compressed-domain learning by maximizing the dimension–distortion trade-off (Jacobellis and Yadwadkar, 2025). In LiVeAction, the training objective is purely to optimize the rate-distortion trade-off, similar to learned image compression systems (Ballé et al., 2017). To simplify the training process and increase accessibility for new modalities, we replace the continuously-relaxed probability model and auxiliary optimizer with a simplified rate penalty based on the sample variance. Compared to codecs with generative or adversarial losses, this formulation requires fewer hyperparameters and provides stable training for a wide range of signal types using thousands, rather than millions, of training examples.

Versatility for use with any modality. LiVeAction is designed for architectural and loss-function generality to support diverse sensing applications. Prior autoencoders are often tied to specific modalities through custom objectives such as LPIPS (Zhang et al., 2018), optical flow loss (Agarwal et al., 2025), or adversarial losses (Mentzer et al., 2020, Zeghidour et al., 2021). In contrast, LiVeAction shows that a simple mean-squared-error (MSE) based rate–distortion objective suffices across modalities, eliminating the need for perceptual losses. Existing DNN architecture designs also limit versatility. The convolutional and transformer-based architectures underlying previous autoencoders are meticulously engineered for specific modalities. LiVeAction’s analysis and synthesis transforms are modality-agnostic and apply to any uniformly grid-sampled signal. Additionally, simple heuristics are sufficient to choose hyperparameters, avoiding costly searches when adapting to new sensors. Together, these design choices reduce development cost while maintaining strong performance across various modalities.

Contributions. Using LiVeAction, we create codecs for a wide range of signal types—spatial audio arrays, hyperspectral images, and 3D medical CT—as well as standard audio, image and video signals. Even compared to state-of the art neural tokenizers using modality-specific designs and trained with orders of magnitude more data and compute, we show improvements in the rate-distortion-complexity trade-off. For example, compared to Cosmos (Agarwal et al., 2025), LiVeAction provides a 34% BD-rate improvement while encoding more than 10\(\times\) faster (see Fig. 6).

Encoding-Efficient Asymmetric Autoencoder Design#

In order to enable applications of machine perception using diverse signal modalities in resource-constrained environments, LiVeAction is designed around three key goals: (1) extreme runtime computational encoding efficiency (2) competitive rate-distortion performance, and (3) flexibility for use with arbitrary modalities.

../_images/liveaction.png

Fig. 7 Proposed design. The analysis transform uses a lightweight DNN with block-diagonal structured operations.#

Overview and codec workflow. LiVeAction inherits the overall architecture from WaLLoC (Jacobellis and Yadwadkar, 2025) and Cosmos (Agarwal et al., 2025), consisting of an FSQ (Mentzer et al., 2024) based autoencoder sandwiched between the WPT and IWPT. However, our asymmetric design introduces several changes to the DNN-based transforms and training procedures. Fig. 7 provides an overview of the codec workflow and structured convolution layers, which we describe next.

Let \(x\in\mathbb{R}^{C\times T_1\times\cdots\times T_D}\) signal with \(D\) spatio-temporal dimensions and \(C\) channels. The end-to-end codec is

(1)#\[ \hat x=\operatorname{IWPT}_{J}\circ \; C^{-1} \circ \mathcal G_{\!S}\circ \Phi^{-1}\circ \mathcal Q\circ \Phi\circ \mathcal G_{\!A}\circ C \circ \operatorname{WPT}_{J}(x). \]

\(\operatorname{WPT}_{J}\) and \(\operatorname{IWPT}_{J}\) apply \(J\) dyadic filter bank stages using the Cohen–Daubechies–Feauveau 9/7 filters to trade spatiotemporal resolution for frequency resolution. The analysis transform \(\mathcal G_{\!A}\). consists of \(d_{\text{enc}}\) factorized group-convolution residual blocks followed by a \(1{\times}1\) projection to latent width \(C_z\). A factorized convolution replaces a dense kernel by two grouped convolutions with groups \((g_1,g_2)\) chosen to minimize MACs (Monarch/ShuffleNet-style), yielding an FFT-like block-diagonal structure. GELU is used as the nonlinearity. The group normalization uses 8 groups. \(C\) is an Invertible power-law compander \(C(x)=\operatorname{sgn}(x)\bigl[(|x|+\varepsilon)^{\gamma}-\varepsilon^{\gamma}\bigr],\) where \(\gamma{=}0.4,\;\varepsilon{=}0.1\). \(\Phi\) is a Non-invertible per-channel Laplacian CDF \(\Phi(x)=127\,\operatorname{sgn}(x)\!\bigl(1-e^{-|x|/\sigma_c}\bigr),\) where \(\sigma_c>0\) is learned; \(\Phi\) ensures latents lie in \([-127,127]\) (strictly less than 8 bits). \(\mathcal Q\) is Finite scalar quantization trained using a soft-to-hard scheme: for the first 70% of training, \(\mathcal Q(x)=x+u,~u{\sim}\mathcal U[-\tfrac12,\tfrac12]\); afterwards the encoder is frozen and \(\mathcal Q(x)=\operatorname{round}(x)\). \(\mathcal G_{\!S}\) is the synthesis transform consisting of EfficientViT linear-attention blocks (generalized to 1/2/3-D), with depth \(d_{\text{dec}}\).

../_images/scaling.png

Fig. 8 Scaling behavior of linear projection (solid line) vs the proposed structured matrix pair (dotted line).#

Lightweight analysis transform for efficient encoding#

In WaLLoC, the encoder consists solely of a learnable linear projection, trading expressiveness for high efficiency. Yet, this projection can still be costly. As an example, consider a spatiotemporal autoencoder for RGB videos. The WPT maps a \(3\times8^3\) RGB video region to 1536 color-frequency bands; projecting these to a \(12\)-D latent requires a \(1536\times12\) matrix-vector product for each local video region. At 1080p, this results in \(> 1.7\) billion FLOPs per second of video for the projection alone. To significantly increase the computational efficiency of encoding, LiVeAction replaces this monolithic projection by several grouped convolutional layers, yielding a structured pair with substantially fewer parameters and lower computational requirements compared to a dense matrix, as shown in Fig. 8. This results in an FFT-like structure for the analysis transform, similar to Shufflenet (Zhang et al., 2018) and Monarch (Dao et al., 2022, Fu et al., 2023). Even using several of these layers with alternating nonlinear activations, added channel attention (Hu et al., 2018), and group normalization, we achieve encoding throughput competitive with the fully connected linear projection used in WaLLoC.

Linear attention synthesis transform for versatility across modalities#

The intended applications of LiVeAction—real time sensing on resource-constrained mobile and remote sensors—place extreme demands on the encoder. However, at runtime, the decoder can be run on powerful cloud GPUs, or even discarded entirely in the case of compressed domain processing. Still, increasing accessibility for new codecs requires high-resolution training to be possible with low or moderate compute resources—not datacenter-scale GPU clusters. Thus, we adopt an EfficientViT-based design (Cai et al., 2023), leading to uncompromised expressiveness while enabling high-resolution training on a single GPU. We make two modifications to EfficientViT: (1) replacement of batch normalization with group normalization to eliminate differences between train-test behavior (Wu and He, 2018), and (2) generalization to one and three dimensions to accommodate additional signal modalities other than 2D images.

Finite scalar quantization with simplified rate penalty#

To achieve a high compression rate, we use finite scalar quantization (FSQ) (Mentzer et al., 2024) a type of learned vector quantization. Unlike standard VQ-AEs (Van Den Oord et al., 2017), which require expensive codebook lookup operations, FSQ uses of a guaranteed dimension bottleneck (typically between 32\(\times\) and \(128\times\) reduction) combined with scalar quantization to achieve equally efficient coding. Existing FSQ designs typically aim for a small or moderate codebook size (typically \(\leq 16\) bits) to support standard cross-entropy losses and increase compression ratio at the cost of objective reconstruction metrics like PSNR. To meet our goal of maximum versatility, we instead opt for much larger codebook size, but include a rate penalty during optimization, similar to the standard approach used in learned image and codecs (Ballé et al., 2017, He et al., 2022, Jia et al., 2025). To reduce the design, implementation, and “operational” (Jia et al., 2025) complexity, we introduce an extremely simplified formulation for the rate loss. Assuming that latent activations follow a distribution in the exponential family (e.g. generalized Gaussian), minimizing the rate is equivalent to minimizing the log of the sample variance. Thus, our overall training objective is to minimize

(2)#\[ \mathcal L=\log_{10}\!\bigl\|x-\hat x\bigr\|_{2}^{2}\;+\;\lambda\, \log_{2}\!\bigl(\operatorname{Var}[\Phi\!\circ\!\mathcal G_{\!A}(x)]\bigr) \]

with a single global hyper-parameter \(\lambda\). The first term is the MSE distortion; the second approximates the latent rate under an exponential-family prior. We set \(\lambda=3\times 10^{-2}\) for all modalities. Finally, we adopt a soft-then-hard quantization scheme (Guo et al., 2021). During the main training phase, additive noise is used to encourage resilience to quantization (Ballé et al., 2017). Near the end of training (70 percent in our experiments), the encoder is frozen, and the additive noise is replaced with hard quantization (rounding) for the remainder of the decoder training. After quantization, any entropy coding method can be used, including lossless media codecs (e.g. FLAC, PNG, FFV1, etc) by reshaping the latents to the appropriate dimension. In our experiments, we find that WEBP lossless and JPEG-LS (Weinberger et al., 2000) provide the best trade-off between compression and computation efficiency for the entropy coding step, though the differences between methods are minor. We include the cost of entropy coding and file storage when measuring throughput.

Heuristics for choosing hyperparameter values#

Building a codec using LiVeAction requires choosing hyperparameters. The exact settings used to reproduce our results for each modality are available in the accompanying code repository. Here, we list several heuristics for choosing these hyperparameters for new modalities.

  1. Dimension. The codec can operate on 1D, 2D, or 3D signals with arbitrary channel count. For many modalities (e.g. single channel audio) the choice of dimension is unambiguous. However, for modalities with high channel count (e.g. the 224 band hyperspectral AVIRIS images), the channels may be treated as an additional dimension. As a rule of thumb, we recommend treating the channels as an additional dimension if both (1) the number of channels is similar to spatiotemporal resolution of the other dimensions and (2) all of the channels have consistent units/scale.

  2. Rate-distortion Lagrangian. In our experiments, all LiVeAction codecs are trained to minimize \(\log_{10} \| x-\hat{x}\| + \lambda \log_2(\hat{\sigma})\), with the parameter \(\lambda\) controlling the trade-off between rate and distortion. We find that \(\lambda=0.03\) provides stable training across all codecs while cutting the average bitrate by about half (about 4 bits per latent channel instead of 8).

  3. Latent dimension. In addition to \(\lambda\), the main hyperparameter affecting the compression ratio is the number of latent channels. For natural signals with significant redundancy, we recommend choosing a latent dimension to be \(64\times\) lower than the original dimension.

  4. Number of levels \(J\) in wavelet packet analysis. With the exception of the projection to and from the latent dimension, all hidden DNN layers operate with a hidden dimension of \(C 2^{JD}\), where \(C\) is the number of signal channels and \(D\) is the dimension. We recommend choosing \(J\) such that the hidden dimension is between 512 and 1536.

  5. Depth. In our experiments, we find that an encoder depth of 4 and a decoder depth of 8 leads to a good balance between runtime encoding efficiency, decoder training cost, and rate-distortion performance.

Evaluation#

Using LiVeAction, we train codecs across multiple signal modalities. We next describe the datasets, evaluation metrics, testbed, and baselines used.

Stereo audio. We train on the lossless MUSDB18-HQ dataset (Rafii et al., 2017), progressively raising clip length from 500k (11s) to 2M samples (48s). Training runs for 200k steps (batch size 2). For augmentation, stems (vocals, drums, bass, other) are randomly remixed; evaluation uses the original validation mixes.

Spatial audio. We train a spatial audio codec for the 7-channel Aria (Engel and others, 2023) microphone array, progressively increasing clip length from 3 to 7 seconds. Training runs for 288k steps with a batch size of 2. Evaluation uses the validation split. In addition to PSNR, we measure the signal to spatial distortion ratio (SSDR) and signal to residual distortion ratio (SRDR) to isolate spatial distortion from other impairments (Watcharasupat and Lerch, 2024).

Image. The codec is trained on LSDIR (Li et al., 2023), with resolution increasing from 128\(^2\) to 480\(^2\) over 500k steps (batch size 16). Evaluation follows (Agarwal et al., 2025) on the 50k-image validation split of ImageNet, resizing all images to height 1024. We also evaluate the rate-distortion performance and top-1 classification accuracy[1] on the ImageNet validation split at low resolution (\(224 \times 224\)) and on the Kodak dataset.

Hyperspectral. We extract 1,394 crops (1,300 training, 94 validation; \(\sim\)0.5 MP each) from 224-band AVIRIS images (Jet Propulsion Laboratory, California Institute of Technology, n.d.). The codec is trained for 130k steps with a maximum resolution of \(224\times288^2\). Evaluation is performed on full-size images.

3D medical images. We train a 3D codec on the MEDMNIST 3D dataset (Yang et al., 2023), with 6 categories of medical volumes: organ, adrenal, fracture, nodule, synapse, and vessel. Resolution increases from \(24^3\) to \(64^3\) voxels over 863.5k steps.

Video. We train on 6,000 Vimeo90k (Xue et al., 2019) clips using two 24-frame batches, with resolution increasing from 112\(\times\)64 to 640\(\times\)384 over 120k steps. The model is fine-tuned on 3,000 high-resolution Vimeo90k clips (batch size 1), with resolution increasing from 680\(\times\)384 to 1152\(\times\)648. Evaluation uses full-length DAVIS (Perazzi et al., 2016) videos at 1920\(\times\)1080.

Metrics and baselines. We evaluate the trade off between rate distortion, and complexity (Minnen and Johnston, 2023) using compression ratio (CR), PSNR, and per-sample throughput, and report dimensionality reduction (DR) as a proxy for downstream acceleration (Jacobellis and Yadwadkar, 2025). PSNR is computed on signals in \([0,1]\); some works (e.g., Cosmos) use \([-1,1]\), yielding values \(6.02\) dB higher (\(20\log_{10}(2)\)). We compare with conventional and neural compression systems, including JPEG2000, Stable Audio (Evans et al., 2025), EnCodec (Défossez et al., 2023), Cosmos (Agarwal et al., 2025), and WalloC (Jacobellis and Yadwadkar, 2025).

Results and Discussion#

Tables 2, 3, and 4 summarize LiVeAction’s performance across modalities. Fig. 9 shows downstream machine-perception quality for RGB images via ImageNet classification accuracy on decoded outputs. Overall, LiVeAction establishes a superior rate–distortion–complexity frontier, particularly in encoding efficiency on resource-constrained hardware. Despite using simpler training objectives, smaller datasets, and far fewer GPU hours than prior generative tokenizers, LiVeAction remains highly competitive in rate–distortion performance while enabling practical deployment on low-power sensor devices.

Table 2 Rate-distortion-complexity trade-off for each modality. DR is the degree of dimensionality reduction. CR is the compression ratio. Encoding throughput (Enc.) is measured in megasamples per second (audio), megapixels per second (images), megavoxels per second (hyperspectral) and frames per second (video). The analysis transform is run on GPU (RTX 4090) and the entropy coding is run on CPU (Intel i9 13900k) with the exception of JPEG 2000\(\dagger\), where no GPU acceleration is available. \(^\ast\) Encoding the entire video in one pass using Cosmos is not possible due to memory constraints. Instead, we encode chunks of 24 frames with 50 percent overlap, resulting in reduced compression ratio and throughput. If no memory constraints were imposed, the CR would be increased by \(4\times\) and the throughput would be increased by \(3\times\).#

Modality

Codec

Metrics

DR

CR

Enc.

PSNR

Stereo music

Stable Audio

64

64.0

12.31

28.42

LiVeAction

64

195

199.3

36.57

Spatial audio

EnCodec

5

455

10.23

27.96

LiVeAction

64

1013

363.2

33.12

RGB Image

Cosmos DI8

32

96.0

54.96

31.20

Cosmos DI16

128

384

116.5

25.08

LiVeAction f16c48

16

34.3

58.88

37.81

LiVeAction f16c12

64

140

143.3

31.09

Hyperspectral

JPEG 2000

1

575

12.47\(^{\dagger}\)

18.18

LiVeAction

64

575

600.1

18.52

3D Medical

JPEG 2000

1

95.62

13.60\(^{\dagger}\)

22.00

LiVeAction

64

209

54.08

24.74

Video

Cosmos DV\(4\times8\)

128

96.0\(^\ast\)

7.656\(^\ast\)

28.96

Cosmos DV\(8\times8\)

256

192\(^\ast\)

13.73\(^\ast\)

27.43

LiVeAction f8c48

32

79.6

33.61

30.24

LiVeAction f8c12

128

331

52.94

27.60
Table 3 BD-rate relative to JPEG 2000 and encoding throughput on low-power mobile CPU (Intel Arrow Lake 255U) for RGB images. All metrics are measured on the Kodak dataset except for Accuracy, which is measured on ImageNet. Lower BD-rate is better for all metrics.#

Codec

BD-rate (PSNR)

BD-rate (DISTS)

BD-rate (SSIM)

Throughput (MPix/s)

Cosmos

+49.61

−90.88

−29.94

N/A

WaLLoC

−27.61

−61.71

−57.52

6.12

LiVeAction

−36.55

−70.27

−70.30

9.95

AVIF

−64.03

−60.56

−71.10

3.01
Table 4 Encoding throughput on high-power CPU (Intel Raptor Lake i9-13900k). Cosmos models are not supported for CPU inference. Sizes for small and large inputs are \(2^{12}\) samples (85 ms) and \(2^{16}\) samples (1.3 s) [music]; 240p and 1080p [images]; \(224^3\) voxels and \(224\times1024^2\) voxels [hyperspectral]; 240p and 1080p [video].#

Modality

Codec

Throughput

Small Input

Large Input

Music (stereo)

Stable Audio

88.73 KSamp/s

229.4 KSamp/s

LiVeAction

323.76 KSamp/s

5012. KSamp/s

RGB Image

JPEG 2000

6.097 Mpix/s

6.333 Mpix/s

Ballé18 (Ballé et al., 2018)

3.440 Mpix/s

5.106 Mpix/s

LiVeAction

5.252 Mpix/s

12.28 Mpix/s

Hyperspectral

JPEG 2000

6.298 Mvox/s

6.448 Mvox/s

LiVeAction

13.76 Mvox/s

14.93 Mvox/s

Video

LiVeAction

107.6 fps

2.386 fps
../_images/imagenet_acc.png

Fig. 9 Machine perceptual quality of Image codecs measured via Imagenet Classification accuracy. Note: Cosmos is trained on ImageNet, while LiVeAction is not.#

Music (stereo). The VAE in Stable Audio generates Gaussian but high-entropy latents, requiring fp16 precision to avoid artifacts. In contrast, LiVeAction’s FSQ design with a rate penalty produces lower-entropy latents, achieving 3\(\times\) higher compression. Stable Audio’s reliance on perceptual and adversarial losses causes cross-channel inconsistencies, whereas LiVeAction’s MSE loss yields better stereo fidelity and +8 dB PSNR. Its structured encoder operations are also far cheaper than Stable Audio’s CNN layers, providing over 16\(\times\) higher throughput.

Spatial audio. LiVeAction outperforms EnCodec with 12.8\(\times\) greater dimensionality reduction (\(64\times\) vs \(5\times\)), 2.2\(\times\) higher compression, and 35.6\(\times\) faster encoding while improving all distortion metrics—achieving +6.09 dB SSDR and +13.55 dB SRDR.

RGB image. On low-power mobile CPU (Intel Arrow Lake 255U), LiVeAction achieves the highest encoding throughput (9.95 MPix/s) and strong BD-rate savings relative to JPEG 2000 (−36.55% PSNR, −70.30% SSIM, −70.27% DISTS); Cosmos is not supported on this platform. Compared to prior neural tokenizers, LiVeAction provides comparable reconstruction quality at similar or higher compression ratios while enabling far greater encoding speed. Notably, despite not being trained on ImageNet (unlike Cosmos), LiVeAction matches Cosmos’ downstream ImageNet top-1 classification accuracy on decoded images while using 48% lower bitrate (Fig. 9).

Hyperspectral. Compared to JPEG 2000, LiVeAction reduces latent dimensionality by \(64\times\) to accelerate downstream models while slightly improving rate–distortion performance. Its DNN-based design also benefits from GPU acceleration, delivering \(\sim\)70\(\times\) higher throughput than CPU-only JPEG 2000 and over 2\(\times\) faster encoding even on the same CPU.

3D medical images. On MEDMNIST 3D, LiVeAction surpasses JPEG 2000 across all metrics, achieving 64\(\times\) dimensionality reduction, 2.1\(\times\) higher compression, and 2.7dB higher PSNR for improved rate–distortion performance.

Video. LiVeAction’s lightweight encoder design enables single-pass encoding of full-length 1080p videos on a single RTX 4090, avoiding the memory-intensive chunking required by Cosmos. At comparable quality, LiVeAction achieves \(>1.7\times\) higher compression ratio (330.7\(\times\) vs. 192\(\times\)) and \(>3.8\times\) higher GPU throughput (52.94 fps vs. 13.73 fps). Real-time encoding (\(>60\)fps) is possible on CPU at low or moderate resolution.

Additional experiments#

Ablation of simplified rate loss. To isolate the effect of the simplified rate loss, we retrained the RGB codec using an explicit rate term. The implementation uses the EntropyBottleneck module from CompressAI (Bégaint et al., 2020) with an auxiliary optimizer.[2] Results are shown in Table 5. The approximate rate model provides a 22 percent reduction in bitrate with minor quality impact.

Table 5 Ablation of the simplified rate loss. The reported bitrate is the actual number of bits after entropy coding, not the rate estimated from the distribution.#

Objective

bpp

PSNR

LPIPS (dB)

Approximate rate: \(\log_2(\hat{\sigma})\)

0.6456

30.8464

6.7503

Exact rate + density model

0.8334

31.1914

6.8621

Perceptual quality enhancement using score-based generative model. Since LiVeAction omits adversarial and perceptual losses, its decoder does not resynthesize high-frequency details. We show that a separate score-based generative model can enhance perceptual quality post-decoding. Specifically, we use a FLUX ControlNet (Zhang et al., 2023) conditioned on the decoder output. Neither model was trained on our codec outputs; instead, a generic version trained on common image corruptions (blur, JPEG, noise) was used. This approach yields modest perceptual gains (+0.5 dB DISTS) but significantly improves realism by restoring textures and fine details (Fig. 10).

../_images/flux_enhanced.jpg

Fig. 10 Comparison of Cosmos, LiVeAction, and LiVeAction enhanced using a generative model. Best viewed zoomed in. The image was rescaled before compressing with cosmos to match the rate of two codecs (0.15 bpp).#

Table 6 Additional results and baselines for RGB images. For ImageNet, the reported accuracy is the top-1 accuracy for 1000-way classification. The accuracy without lossy compression is 0.8979.#

Model

Kodak (\(512\times768\) and \(768\times512\))

ImageNet-1k (\(224\times224\))

bpp

PSNR

LPIPS

DISTS

SSIM

bpp

PSNR

LPIPS

DISTS

SSIM

Acc

LiVeAction f16c12 (half res.)

0.0428

24.2336

2.7625

6.2970

0.7194

0.1112

23.647

3.5264

5.4999

0.6257

0.6277

Cosmos di16\(\times\)16

0.0625

21.7743

5.3784

10.619

0.6449

0.0625

19.710

5.3340

8.4825

0.4882

0.7938

AVIF (q=10)

0.0916

26.5260

3.1129

6.9885

0.8080

0.3373

24.830

3.7429

5.7411

0.6982

0.7299

JPEG 2000 (CR 250:1)

0.0957

24.3492

2.4231

5.7073

0.6842

0.0978

20.450

2.0800

4.0033

0.4794

0.2807

LiVeAction f16c12

0.1496

27.0384

4.2153

8.8654

0.8600

0.1943

25.402

4.4948

6.3726

0.7122

0.7981

Ballé et al. (2018)

0.2110

27.2377

3.9050

7.9973

0.8258

0.3338

28.188

5.3061

6.8009

0.8116

0.8360

AVIF (q=25)

0.2316

29.5176

4.4152

9.1244

0.9007

0.5281

27.789

5.3463

7.1298

0.8106

0.8262

JPEG 2000 (CR 100:1)

0.2393

26.4546

3.1854

7.1317

0.7829

0.2402

23.404

3.1483

5.0364

0.6014

0.6767

Cosmos di8\(\times\)8

0.2500

25.9193

7.7112

13.265

0.8558

0.2500

24.334

8.0240

10.808

0.7036

0.8708

LiVeAction f16c48 (\(\lambda=0.1\))

0.3992

29.8397

5.6911

12.073

0.9303

0.4868

27.978

6.1277

7.9186

0.8206

0.8463

WaLLoC f8c12

0.6171

30.5576

6.5138

13.244

0.9501

0.7720

29.102

7.0447

8.5500

0.8529

0.8569

LiVeAction f16c48

0.6606

31.1669

6.5692

13.670

0.9571

0.8072

29.501

7.1563

8.6339

0.8568

0.8560

AVIF (q=50)

0.6838

34.4449

7.1089

12.992

0.9657

1.1147

32.299

8.6288

9.9068

0.9144

0.8679

JPEG 2000 (CR 20:1)

1.1984

32.0019

5.4387

11.365

0.9262

1.1952

29.517

6.1171

7.7432

0.8206

0.8550

WaLLoC f8c48

2.5436

37.3370

11.674

18.294

0.9873

3.0067

35.107

12.120

13.063

0.9529

0.8838

LiVeAction f16c192

3.9126

40.2877

15.290

25.534

0.9962

4.2843

36.595

14.870

15.568

0.9640

0.8851

Conclusion and Future Work#

We introduced LiVeAction, a neural codec design that establishes a new performance frontier and increases accessibility of learned compression for new types of signals and sensors. By improving signal-ingestion efficiency, LiVeAction lowers power and bandwidth demands while maintaining quality, enabling new mobile and remote sensing applications. Future work will explore variable-rate training and joint optimization with downstream ML tasks to better align compression with inference accuracy.

References#

[AAB+25] (1,2,3,4,5,6,7,8,9,10,11)

Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, and others. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025.

[AMJ+20]

Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In CVPR. 2020.

[BalleLS17] (1,2,3,4,5,6)

Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. In ICLR. 2017.

[BalleMS+18] (1,2)

Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. ICML, 2018.

[BegaintRacapeFP20]

Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029, 2020.

[CLT+23] (1,2)

Han Cai, Junyan Li, Muyan Tian, Zhekai Hu, and Song Han. Efficientvit: lightweight multi-scale attention for high-resolution dense prediction. In CVPR. 2023.

[CBL+25]

Li-Heng Chen, Christos G Bampis, Zhi Li, Lukas Krasula, and Alan C Bovik. Estimating the resize parameter in end-to-end learned image compression. Signal Processing: Image Communication, 2025.

[CKG+24]

Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. NeurIPS, 2024.

[DCS+22] (1,2,3)

Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. Monarch: expressive structured matrices for efficient and accurate training. In ICML. 2022.

[Dav68]

Lee D Davisson. The theoretical analysis of data compression systems. P. IEEE, 1968. URL: https://ieeexplore.ieee.org/abstract/document/1448142.

[Dav72]

Lee D Davisson. Rate distortion theory: a mathematical basis for data compression. IEEE Trans. on Communications, 1972.

[DC16]

Sheng Di and Franck Cappello. Fast error-bounded lossy hpc data compression with sz. In IEEE international parallel and distributed processing symposium. 2016.

[DMWS20]

Keyan Ding, Keda Ma, Shiqi Wang, and Eero P Simoncelli. Image quality assessment: unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence, 2020.

[DefossezCSA23]

Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023.

[DefossezMoulinieC+24]

Alexandre Défossez, Laurent Moulinié, Jade Copet, and others. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037, 2024.

[E+23] (1,2)

J. Engel and others. Project aria. arXiv preprint arXiv:2308.13561, 2023.

[EPC+25] (1,2,3)

Zach Evans, Julian Parker, CJ Carr, Zack Zukowski, Jordan Taylor, and Jordi Pons. Stable audio open. In ICASSP. 2025.

[FDS+23] (1,2,3)

Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Monarch mixer: a simple sub-quadratic gemm-based architecture. NeurIPS, 2023.

[GZFC21]

Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Rethinking the quantization in neural image compression. In ICML. 2021.

[HEYC+25] (1,2)

Philippe Hansen-Estruch, David Yan, Ching-Yao Chuang, Orr Zohar, Jialiang Wang, Tingbo Hou, Tao Xu, Sriram Vishwanath, Peter Vajda, and Xinlei Chen. Learnings from scaling visual tokenizers for reconstruction and generation. arXiv preprint arXiv:2501.09755, 2025.

[HYP+22] (1,2)

Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Elic: efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In CVPR. 2022.

[HSS18] (1,2)

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR. 2018.

[H+24]

A. Hurst and others. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.

[JY25] (1,2,3,4,5,6)

Dan Jacobellis and Neeraja J. Yadwadkar. Learned compression for compressed learning. In Data Compression Conference. IEEE, 2025.

[JLL+25] (1,2,3,4)

missing journal in jia2025towards

[KRH+21]

Mehrdad Khani, Prabakkore Ramaniharan, Burhan Hamza, Mohammed Alzayat, Amin Haghani, Saurabh Singh, Sang Klein, Arash Vahdat, and Mohammad Alizadeh. Efficient video compression via content-adaptive sup. res. In CVPR. 2021.

[Kor67]

C Kortman. Redundancy reduction: a practical method of data compression. Proceedings of the IEEE, 1967.

[LCM+23]

Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: masked generative encoder to unify representation learning and image synthesis. In CVPR. 2023.

[LFY+23]

Yawei Li, Yulun Fan, Xiaoyu Yu, Joshua Batson, Kai Qian, Eirikur Agustsson, and Radu Timofte. Lsdir: a large scale dataset for image restoration. In CVPR. 2023.

[MMAT24] (1,2,3)

Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: vq-vae made simple. In ICLR. 2024.

[MTTA20] (1,2)

Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compression. NeurIPS, 2020.

[MJ23]

D. Minnen and N. Johnston. Advancing the rate-distortion-computation frontier for neural image compression. In ICIP. 2023.

[MHV25]

Matthew J Muckley, Marton Havasi, and Jakob Verbeek. Architecture optimizations for improving neural image compression compute complexity. In DCC). IEEE, 2025.

[PCH+23]

Song Park, Sanghyuk Chun, Byeongho Heo, Wonjae Kim, and Sangdoo Yun. Seit: storage-efficient vision training with tokens using 1% of pixel storage. In CVPR. 2023.

[PPTM+16]

Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR. 2016.

[P+24]

A. Polyak and others. Movie gen: a cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024.

[RLStoter+17]

missing journal in rafii2017musdb18

[RPG+21] (1,2)

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML. 2021.

[RBL+22] (1,2)

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022.

[VDOV+17] (1,2)

Aaron Van Den Oord, Oriol Vinyals, and others. Neural discrete representation learning. NeurIPS, 2017.

[WL24]

Karn N Watcharasupat and Alexander Lerch. Quantifying spatial audio quality impairment. In ICASSP. 2024.

[WSS00]

Marcelo J Weinberger, Gadiel Seroussi, and Guillermo Sapiro. The loco-i lossless image compression algorithm: principles and standardization into jpeg-ls. IEEE Transactions on Image processing, 2000.

[WH18]

Yuxin Wu and Kaiming He. Group normalization. In ECCV. 2018.

[XCW+19]

Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 2019.

[YSW+23]

Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 2023.

[ZLO+21] (1,2)

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: an end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.

[ZCC+22]

Bing Zhang, Y Chen, J Chanussot, and others. Progress and challenges in intelligent remote sensing satellite systems. IEEE Topics in Applied Earth Observations and Remote Sensing, 2022.

[ZRA23]

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to diffusion models. In CVPR. 2023.

[ZIE+18]

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR. 2018.

[ZZLS18] (1,2,3)

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: an extremely efficient convolutional neural network for mobile devices. In CVPR. 2018.

[ZDD+21]

Kai Zhao, Sheng Di, Maxim Dmitriev, Thierry-Laurent D Tonellot, Zizhong Chen, and Franck Cappello. Optimizing error-bounded lossy compression for scientific data by dynamic spline interpolation. In IEEE International Conference on Data Engineering. 2021.

[JetPLaboratoryCIoTechnology]

Jet Propulsion Laboratory, California Institute of Technology. Airborne Visible/Infrared Imaging Spectrometer (AVIRIS). https://aviris.jpl.nasa.gov.