# Machine Perceptual Quality

In the field of neural data compression, the prevailing focus has been on optimizing algorithms for either classical distortion metrics, such as PSNR or SSIM, or human perceptual quality. With increasing amounts of data consumed by machines rather than humans, a new paradigm of machine-oriented compression---which prioritizes the retention of features salient for machine perception over traditional human-centric criteria---has emerged, creating several new challenges to the development, evaluation, and deployment of systems utilizing lossy compression. In particular, it is unclear how different approaches to lossy compression will affect the performance of downstream machine perception tasks. To address this under-explored area, we evaluate various perception models---including image classification, image segmentation, speech recognition, and music source separation---under severe lossy compression. We utilize several popular codecs spanning conventional, neural, and generative compression architectures. Our results indicate three key findings: (1) using generative compression, it is feasible to leverage highly compressed data while incurring a negligible impact on machine perceptual quality; (2) machine perceptual quality correlates strongly with deep similarity metrics, indicating a crucial role of these metrics in the development of machine-oriented codecs; and (3) using lossy compressed datasets, (e.g. ImageNet) for pre-training can lead to counter-intuitive scenarios where lossy compression increases machine perceptual quality rather than degrading it. Our code and experiments are available at: <https://github.com/danjacobellis/MPQ>.

```{figure} ../figures/MPQ_image_comparison.png
:name: fig:image_compression_methods
:width: 100%

Visual comparison of image compression methods. HiFiC {cite}`mentzer2020high` is a generative neural codec. MBT2018 {cite}`minnen2018joint` is a neural codec optimized for pixel-wise mean squared error. WEBP and JPEG are transform codecs based on perceptual quantization of DCT coefficients.
```

## Background

Conventional media compression standards (called codecs) rely on simple but effective linear transforms that exploit the redundancies of natural signals. For example, the discrete cosine transform (DCT) used in JPEG and MP3 compresses signal energy into fewer coefficients. Carefully designed quantization matrices then assign more bits to perceptually important temporal or spatial sub-bands based on models of human sensitivity. These compression techniques have remained popular for decades since they offer a decent compression rate without excessively compromising signal quality.

Two key developments led to a greater focus on neural network based compression. First, {cite:t}`balle2017end` showed that autoencoders optimized end-to-end for both rate and distortion (a.k.a. rate distortion autoencoders or RD-AEs) compress images more effectively than traditional codecs. In parallel, {cite:t}`van2017neural` introduced the vector quantized variational autoencoder (VQ-VAE) as a method of representation learning. Variants of these architectures emerged specializing them for better human perceptual quality, both for audio {cite}`zeghidour2021soundstream,defossez2023high` and images {cite}`he2022po`. The advent of generative compression methods {cite}`mentzer2020high` led to observation of a rate-distortion-perception trade-off {cite}`wagner2022rate`.

In addition to better rate-distortion performance, ongoing codec development efforts also aim to optimize for machine perception, a paradigm referred to as "compression for machines {cite}`chamain2021end`" or "machine-oriented compression {cite}`kang2023super`." Most notably, The JPEG AI standard {cite}`ascenso2023jpeg` proposes a single stream image encoder supporting multiple decoders for both human and machine perception. {cite:t}`harell2025rate` proposed a taxonomy of three different machine-oriented compression approaches. Notable to our work is the method of full-input machine-oriented compression, where the signal is fully decoded before performing downstream tasks; this can either be achieved using an existing codec or by optimizing the compression system for the downstream task.

While the evaluation of human perceptual quality has been extensively studied, it is less clear how different types of lossy compression affect machine perception. {cite:t}`hendrycks2019benchmarking` study the impact of various corruptions, including JPEG compression, on image classification performance. {cite:t}`matsubara2023sc2` study the impact of various image compression methods on classification and segmentation. Despite these contributions, a dedicated analysis of severe lossy compression effects across a variety of applications, including generative compression methods and audio models, remains unexplored and is the focus of our investigation.

## Evaluation Framework for Machine Perceptual Quality

We investigate the impact of different audio and image compression techniques on machine perceptual quality under severe lossy compression---which we define as compression ratios between 20:1 and 1000:1---and compare against a baseline that does not have additional compression.
We employ six datasets, seven different lossy compression methods, and popular pre-trained models for various discriminative tasks, as detailed in {cite:t}`jacobellis2024machine`.
We use the performance on the validation split of each dataset as a measure of machine perceptual quality. We evaluate the compression performance based on bitrate, conventional distortion metrics, and deep similarity metrics.

**Models and datasets.** We employ the ImageNet-1k dataset for image classification, using a vision transformer (ViT) pre-trained on ImageNet-21k {cite}`dosovitskiy2021image`. The NIH ChestX-ray8 dataset {cite}`wang2017chestx` for pneumonia classification and the bean disease dataset {cite}`singh2023classification` are also used in conjunction with an ImageNet-21k pre-trained ViT. Semantic segmentation is performed on the ADE20k dataset {cite}`zhou2017scene` using the SegFormer model {cite}`xie2021segformer`. The Common Voice 11.0 dataset {cite}`ardila2020common` and the Whisper model {cite}`radford2023robust` are used for speech recognition. Finally, the MUSDB-HQ dataset, an uncompressed version of MUSDB18 {cite}`rafii2017musdb18`, and the Demucs v3 model {cite}`defossez2021hybrid` are used for music source separation. See {cite:t}`jacobellis2024machine` and the associated code repository[^1] for additional implementation details and a listing of specific model variants.

**Compression methods.** The image compression methods in our study include JPEG, WEBP, the distortion-optimized neural compression approach of {cite:t}`minnen2018joint`, and the generative compression method HiFiC {cite}`mentzer2020high`. For audio compression, we use MPEG Layer III (MP3), Opus, and the neural audio model EnCodec {cite}`defossez2023high`.

**Evaluation metrics.** We use conventional rate-distortion metrics as well as deep similarity metrics---quality metrics derived from deep neural networks and trained to predict human judgments of quality. Each metric is calculated on a per-sample basis.
Peak Signal-to-Noise Ratio (PSNR) is used as a conventional distortion metric for both images and audio.
Learned Perceptual Image Patch Similarity (LPIPS) {cite}`zhang2018unreasonable` is a deep similarity metric specifically designed for images. It captures complex perceptual differences that simpler metrics like PSNR or SSIM are insensitive to. In the table, we report $-10\log_{10}(\text{LPIPS similarity})$ to align it with the other quality metrics.
Contrastive Deep Perceptual Audio Similarity Metric (CDPAM) {cite}`manocha2021cdpam` is a deep similarity metric designed for audio. Like LPIPS for images, it captures perceptual differences more effectively than PSNR. Similar to LPIPS, we report $-10\log_{10}(\text{CDPAM similarity})$.

## Key Findings

Our evaluation across multiple datasets and machine perception tasks reveals key insights into the impact of lossy compression on machine perceptual quality. See {cite:t}`jacobellis2024machine` for a detailed listing of results.
For image-based tasks, LPIPS is a better predictor of downstream performance than PSNR, and generative compression (HiFiC) performs the best at all but one of the tasks (Chest X-ray) despite having the lowest average bitrate, and consistently achieves results close to the uncompressed baseline. In the audio domain, similar trends are observed; the audio quality measured by CDPAM is better predictor of downstream performance than PSNR, and, among the methods tested, EnCodec provides the best trade-off between rate and downstream performance for both datasets.

### Generative compression preserves machine perceptual quality

One area of concern is that generative compression methods like HiFiC and EnCodec, whose adversarial training objectives allow them to discard details at the encoder and re-synthesise them at the decoder, are ill-suited for use within machine perception pipelines. However, our results indicate the contrary; despite having the highest compression ratios among the methods tested, these methods performed well across all tasks, often outperforming methods with significantly higher bitrate. Unfortunately, current generative compression methods are far from being production-ready, and rely on architectures which are difficult to train, adapt, and deploy. However, recent advancements have shown remarkable inference speedup in score-based generative models {cite}`song2023consistency` and vastly simplified training procedures for VQVAEs {cite}`mentzer2024finite`. By incorporating such advancements and making these methods more accessible, generative compression could enable new applications, such as satellite, maritime and aerial remote sensing systems that require very high compression ratios.

### Correlation of machine perceptual quality with deep similarity metrics

Deep similarity metrics like LPIPS and CDPAM are known to be highly effective at predicting human perceptual quality as measured by mean opinion score (MOS). Across the six datasets tested, our results indicate that such metrics are also strongly correlated with machine perceptual quality, despite only being trained in a supervised fashion to predict human judgments of signal distortion pairs. A promising avenue for future research would be to extend the training objectives for these metrics to include machine judgments of distortion pairs, making them even more robust.

### Pretraining on lossy datasets

Our experiments reveal a surprising phenomenon: for models pre-trained on lossy datasets like ImageNet, additional lossy compression at test time may have negligible impact on performance, and can sometimes behave as an enhancement. For example, the top-1 classification accuracy on the bean disease dataset is higher when compressed using HiFiC (compression ratio of 839:1) than when using the original lossless images. Even more surprising is that severe JPEG compression (see {numref}`fig:image_compression_methods`) results in an increase in pneumonia classification performance on the Chest X-ray dataset, despite having the lowest quality measured by PSNR or LPIPS. Viewing lossy compression as a type of distribution shift provides one possible explanation for this phenomenon; subtle high-frequency details that only exist in lossless images never occur in pre-training datasets like the JPEG-compressed ImageNet. Exploring pre-training with lossless data may be feasible considering the moderate compression ratios (5:1) used such datasets. The development of lossless datasets at ImageNet or larger scale could be valuable for the development of neural compression systems---for both human and machine applications.

[^1]: [Github: danjacobellis/MPQ](https://github.com/danjacobellis/MPQ)

## References

```{bibliography}
:filter: docname in docnames
```