# Introduction

The staggering economic scale of the smartphone market has driven extraordinary advances in sensing: modern low-cost, low-power image sensors let small, battery-powered robots and wearables capture billions of pixels per second at a fidelity once reserved for earth-observation satellites, consuming on the order of $10^{-11}$ joules per pixel {cite}`chen201412,kim2022fully`. However, the systems we use to transmit, store, and analyze this superabundance of information have not kept up. Fully utilizing these information-dense signals on-device is prohibitive, because the most capable machine perception models require computation that scales super-linearly with resolution {cite}`beyer2024vitspeed`; offloading the computation to the cloud, where power is abundant, instead demands extreme compression to fit on-device power and bandwidth budgets. For example, a 1080p 30 fps stream over a 25 Mbps Wi-Fi channel requires a compression ratio of about 60:1, and a 480p stream over a 1 Mbps Bluetooth Low Energy channel about 288:1 {cite}`balasubramanian2009energy,carroll2010analysis,gupta20243,tosi2017performance`. It is common to instead use only low-resolution, heavily compressed feeds and discard the rest.

Media compression standards, or "codecs," such as JPEG and MPEG reduce storage and transmission requirements by multiple orders of magnitude by discarding imperceptible details and eliminating signal redundancies. Historically, standards adopt carefully engineered energy-compacting transforms, prediction mechanisms, and entropy coding methods, meticulously designed to provide the greatest compression efficiency while maintaining tolerable computational cost. Data-driven approaches, or end-to-end learned compression (E2ELC), using neural networks have been studied for decades {cite}`sonehara1989image`, and in recent years have come to dominate in terms of compression efficiency. Ideally, such approaches should increase the accessibility and practicality of designing application-specific compression systems that benefit from specialization to a specific signal, sensor type, or environment, without requiring years of laborious and expensive algorithm development, perceptual quality testing, and iteration. However, adoption of such methods, and therefore the realization of their benefits, has not occurred, even long after their superiority in terms of compression efficiency has been firmly established. The reasons for this are twofold: neural-network-based approaches are typically many times more expensive to encode, often differing by two or three orders of magnitude; and while they replace meticulously engineered transforms, quantization matrices, and prediction algorithms with a unified approach---gradient-based optimization of neural networks---they introduce a difficult-to-navigate landscape of loss functions, hyperparameters, and modality-specific neural architectures that must be meticulously engineered, and also require expensive GPU compute infrastructure. Thus, the vast majority of applications choose to adopt some existing standard even if the required bandwidth, target signal type, environment, or human-perception-oriented design are inappropriate.

However, these standards meet the compression ratios above only at severe perceptual cost {cite}`jacobellis2024machine`. More fundamentally, they are built on the zombie assumption that distortion should be optimized to reflect subjective quality to a human viewer or listener. In an earlier era, this objective accurately reflected the predominant use cases of multimedia compression. The validity of this assumption has long been put to rest by the proliferation of machines possessing superhuman perception, diffused throughout our society, and with unimaginable potential for benefit---more accurate medical diagnostics, safer transportation, and accelerated scientific discovery, to name a few.

Given irreducible power and bandwidth constraints, data compression is the primary tool that allows us to prioritize what information propagates from our resource-constrained sensor devices to our powerful but power-hungry machine consumers. The potential consequences of relying on compression methods designed for primate perceivers are profound, since this is tantamount to prioritizing by the degree of evolutionary benefit that a signal component provides above all else. For example, the blood-signaling hypothesis {cite}`changizi2006bare` posits that the ability to discriminate subtle differences in red hues associated with spectral modulation of skin provides an emotional and socio-sexual signal that is explanatory of the unique color sensitivities of human vision. A compression system that maximizes subjective quality to a human will therefore disproportionately prioritize information encoding minor differences in red hues compared to other details. Human auditory and visual systems are also characterized by what they do not prioritize, or discard altogether: infrared, infrasound, ultrasound, and ultraviolet, for example. The opportunity for machine perception systems to complement human abilities---detecting minute anomalies in medical images, avoiding vehicle collisions, or making scientific discoveries from hyperspectral satellite image data---would be vastly hindered by inheriting the biases of human subjective perception.

The conventional codec development lifecycle, involving years or even decades of standardization for specific applications across international committees, cannot keep up with the pace of innovation in robotics, IoT, and other types of sensing. Standards bodies like MPEG operate on an eight- to ten-year cadence, providing time for customized hardware ASICs to implement each new standard {cite}`wiegand2003overview,sullivan2012overview,bross2021overview,zhang2026learning`. However, machine perception systems, typically developed for and delivered to fully programmable CPUs, GPUs, or NPUs through pure software means, commonly progress through a major generation each year from a single developing entity alone {cite}`bao2022beit,peng2022beit,wang2023image,caron2021emerging,oquab2024dinov,simeoni2026dinov`. Additionally, compression standards have historically been organized by a handful of international organizations---JPEG, MPEG, ITU-T, AOM, and DICOM---often working in collaboration. In contrast, the major developments in machine perception systems are distributed across hundreds of universities, global technology conglomerates, and venture-capital-funded startups, fueled by unprecedented levels of capital investment and often in an environment of intense competition. With each new year, the number of new scientifically or commercially significant machine perception systems is virtually uncountable. At present, decades-old codecs like JPEG, MP3, and AVC {cite}`wallace1992jpeg,brandenburg1994iso,wiegand2003overview` remain the workhorse for the overwhelming majority of applications involving on-device encoding (e.g., streaming video feeds from aerial robots or streaming audio from a smartwatch), leading to a somewhat embarrassing outcome that is all-too common: cutting-edge, super-human AI foundation models on bleeding edge supercomputers fighting an uphill battle to ingest low quality, downsampled images compressed using antiquated methods operating at bitrates and on types of signals that they were never designed for.

The intentionally slow approach centered around standardization and efficient deployment via specialized ASICs and software is ill-suited for machine-oriented compression, and architectures should instead adopt the characteristics of their signal consumers: (1) end-to-end differentiability, to control behavior programmatically using isolated or joint deep learning optimization; (2) inference workflows that are amenable to programmable CPUs, NPUs, and MCUs; and (3) development via highly accessible and flexible frameworks, so that scientists and engineers are empowered to develop application-specific compression systems that meet the unique demands of their own domains, instead of choosing from a limited set of standards-defined options designed mostly for mass media broadcasting and telecommunications. This dissertation investigates ***machine-oriented compression***: compression systems designed for machine perception rather than human perception, supporting model training and inference directly on compressed representations without additional decoding overhead. {numref}`tbl:limitations_principles` pairs each limitation of conventional codecs with the machine-oriented design principle that addresses it; the paragraphs that follow expand on each principle, and Chapters 3--5 show how the three characteristics above are realized in codec architectures.

```{list-table} Limitations of conventional codecs and the machine-oriented design principles that address them.
:header-rows: 1
:widths: 50 50
:name: tbl:limitations_principles

* - Limitation of conventional codecs
  - Machine-oriented design principle
* - Distortion is minimized for human perception, even as ML systems---capable of perceiving minute, high-frequency details---become the primary signal consumers.
  - Preserve the features salient to machine perception, interoperating with unmodified pre-trained foundation models.
* - Signals must be fully decoded into redundant representations (pixels, audio samples) before other systems can use them, adding overhead at every stage of ML training and inference.
  - Train and run inference directly on compressed representations, with only partial decoding---reducing overhead at every stage of the ML pipeline and improving resolution scaling.
* - Formats are fixed to stereo audio and RGB image/video, excluding biomedical signals, hyperspectral and volumetric imagery, multi-perspective robot video, and other emerging modalities.
  - A modality-agnostic, end-to-end differentiable architecture that adapts to any sensor, signal type, environment, or downstream task.
* - Newer standards target mass broadcast: expensive cloud encoding in exchange for cheap decoding on consumer devices. The opposite case---cheap encoding on battery-powered devices---is neglected.
  - Invert the asymmetry: encoders simple enough for microcontrollers, with powerful cloud decoders that enhance and, when appropriate, resynthesize detail.
* - Changing a standard takes years of committee work, and abandoning standards means abandoning the hardware and software infrastructure built around them.
  - Bespoke codecs trained in hours on a single GPU, which remain compatible with existing infrastructure through one-time transcoding to standard formats.
```

**(1) Preserve machine-perceptual salience without adaptation.** A machine-oriented codec should be immediately useful for general-purpose applications: paired with pre-trained foundation models, it should preserve the features salient to machine perception without any task-specific tuning. Chapter 3 demonstrates this, applying a single pre-trained codec, unmodified, across image classification, colorization, document understanding, and music source separation.

**(2) Operate directly on compressed representations.** Rather than fully decoding a signal back to pixels or audio samples before inference, models can train and run directly on the compressed representation, with only lightweight partial decoding (e.g., only the entropy decoding step). This reduces overhead at every stage of the ML pipeline and improves how models scale to high resolutions, since the representation is inherently lower-dimensional than the signal it encodes. Increasing input resolution substantially improves the accuracy of modern perception models, but at a steep cost in memory and latency; compressed-domain learning attenuates this cost by replacing resolution reduction with dimensionality reduction (Chapter 3).

**(3) Adapt end to end to any sensor, signal, environment, or task.** Because every component is differentiable, the same codec can be adapted end to end---to a specific sensor, signal type, or environment, or jointly with a downstream ML application---whenever specialization is justified. Chapter 4 extends this to spatial audio arrays, hyperspectral remote sensing data, and 3D medical images; Chapter 7 trains both general-purpose and task-aware transcoding pipelines for a pre-trained, frozen encoder.

**(4) Make the encoder simple enough for MCUs.** Machine-oriented compression divides labor asymmetrically. The encoder reduces to a static bank of sub-critically sampled filters producing inherently lower-dimensional outputs, a simple and efficient pointwise companding nonlinearity, and precomputed entropy coding tables. There is no online rate-distortion optimization requiring multiple encoding passes; variable-rate and progressive coding come from an embarrassingly parallel encoder DAG with no decoder in the loop (Chapter 5). The encoders developed in Chapters 3 and 5 reduce to a single linear layer with fewer than 100k parameters and, at low bitrates, cost less to run than a JPEG encoder---within reach of commodity CPUs, NPUs, and MCUs rather than the GPUs and hardware-baked ASICs on which newer standards increasingly rely. The heavy computation moves to the cloud, where powerful DNN decoders enhance the signal and, when appropriate for the application, resynthesize useful details.

**(5) Make bespoke codecs accessible, and keep them compatible.** Encoders this simple are cheap to develop as well as to run: trained in hours on a single GPU, a codec for a new sensor or task requires limited resources rather than extensive research, engineering, or supercomputers. Yet decades of hardware and software systems have been built around standards like JPEG---codecs baked into ASICs and SoCs, fast dataloaders, ML frameworks, and web browsers---and would be perilous to discard. We show that a learned, sensor-embedded codec can instead coexist with this ecosystem: a one-time transcode in the cloud converts the learned representation into a standard file that every existing consumer can decode (Chapter 7).

## Organization

The remainder of this dissertation develops these ideas in six chapters, proceeding from measurement, to codec design, to complete systems.

Chapter 2, *[Machine Perceptual Quality](mpq.md)*, establishes how existing ML systems are impacted by lossy compression from conventional, neural, and generative codecs. Additionally, this chapter characterizes the current landscape and interactions of objective and perceptual distortion metrics; conventional, learned, and generative codecs; and machine learning foundation models. We also demonstrate why reversible or "lossless" compression is not an appropriate goal for machine perceptual signals, due to the realities of noisy data and fundamental limitations of the current generation of foundation models. Across image classification, segmentation, speech recognition, and music source separation, we find that deep similarity metrics---originally designed to predict human judgments---are strong predictors of machine perceptual quality, and that machine perception can match, and in some cases exceed, the lossless baseline even under severe lossy compression, which we define as compression ratios between 20:1 and 1000:1.

Chapter 3, *[Learned Compression for Compressed Learning](walloc.md)*, introduces the first machine-oriented codec design that combines the benefits of conventional transform coding with end-to-end learned compression using neural networks. Chapters 3 and 4 develop the first machine-oriented compression systems for audio, images, video, and other data types, based on encoding-efficient asymmetric autoencoders (EE-AAEs) that directly optimize the rate-distortion Lagrangian $D + \lambda R$ in such a way that the distortion can either be general-purpose---suitable for consumption by humans and pre-trained machine perception systems---or specialized to specific tasks. For audio and image signals, this design provides cheap encoding, high compression efficiency, and uniform dimension reduction to accelerate downstream models. Using codecs adopting this design, we propose a simple but universal strategy for compressed-domain learning and demonstrate improved resolution scaling properties compared to large patch transformers or highly strided CNNs. Together, our proposed codec design and compressed learning framework significantly improve the accuracy-computation trade-off for high-resolution discriminative ML applications.

Chapter 4, *[Lightweight, Versatile Codec Design](liveaction.md)*, expands on this framework with a lightweight, versatile, and asymmetric codec design. Using FFT-like structured operations in the analysis transform, linear attention in the synthesis transform, and simplified training objectives, we demonstrate state-of-the-art performance in terms of the rate-distortion-complexity tradeoff while maintaining a modality-agnostic architecture. To demonstrate the versatility of this approach, we train codecs for stereo and multi-channel spatial audio, RGB and hyperspectral images, video, and 3D medical images, achieving competitive performance in each case.

Chapter 5, *[Variable-Rate Compression and Projection-Pursuit Encoding](frappe.md)*, addresses a remaining obstacle to deploying learned codecs on resource-constrained sensors: rate adaptation. We propose FRAPPE, a residual autoencoding framework that uses the full input to predict the residual output via a projection-pursuit encoder. The encoding objective naturally sorts latent channels by importance, enabling zero-overhead variable-rate and progressive coding using a single set of encoder weights, while keeping the analysis path an embarrassingly parallel DAG of independent projections---no recurrence, no quantizer chain, and no decoder in the loop. At high compression ratios ($\sim$0.1 bpp), the resulting image codec provides higher perceptual quality than AVIF with 47$\times$ faster encoding, making it capable of real-time 1080p, 30 fps CPU-only encoding.

Chapter 6, *[Video and Real-Time Sensing](dedelayed.md)*, turns from individual signals to real-time systems, focusing on video understanding for robotics. In applications where predictions must be available on device by a guaranteed deadline, round-trip communication delay makes standard cloud inference a non-starter. Instead, we propose a framework that feeds temporally predictive outputs of a delayed remote model as side information to a local model with access to the current signal, leading to both accurate and on-time predictions.

Chapter 7, *[Sensor-Embedded Autoencoding with One-Time Transcode](seaotter.md)*, addresses compatibility with existing infrastructure. SEAOTTER pairs a sensor-embedded encoder with a one-time transcode performed in the cloud: a powerful decoder reconstructs the signal and re-encodes it as a standard JPEG file, so downstream consumers, from training pipelines to web browsers, need nothing more than an ordinary JPEG decoder. A naive transcode would degrade quality; instead, the JPEG color transform and quantization matrices are learned end to end, and the transcode increases accuracy across global, dense, and vision-language tasks relative to the underlying autoencoder.

Taken together, these chapters demonstrate that compression can provide more than a compromise between quality and bitrate: when compression systems are designed for machine perception from the start, encoder cost, compression efficiency, and compatibility need not be traded against one another. The result is a practical framework for sensing and perception under extreme power, bandwidth, latency, or accuracy constraints.

## References

```{bibliography}
:filter: docname in docnames
```