Introduction#
The staggering economic scale of the smartphone market has driven extraordinary advances in sensing: modern low-cost, low-power image sensors let small, battery-powered robots and wearables capture billions of pixels per second at a fidelity once reserved for earth-observation satellites, consuming on the order of \(10^{-11}\) joules per pixel (Chen et al., 2014, Kim et al., 2022). However, the systems we use to transmit, store, and analyze this superabundance of information have not kept up. Fully utilizing these information-dense signals on-device is prohibitive, because the most capable machine perception models require computation that scales super-linearly with resolution (Beyer, 2024); offloading the computation to the cloud, where power is abundant, instead demands extreme compression to fit on-device power and bandwidth budgets. For example, a 1080p 30 fps stream over a 25 Mbps Wi-Fi channel requires a compression ratio of about 60:1, and a 480p stream over a 1 Mbps Bluetooth Low Energy channel about 288:1 (Balasubramanian et al., 2009, Carroll and Heiser, 2010, Gupta et al., 2024, Tosi et al., 2017). It is common to instead use only low-resolution, heavily compressed feeds and discard the rest.
Media compression standards, or “codecs,” such as JPEG and MPEG reduce storage and transmission requirements by multiple orders of magnitude by discarding imperceptible details and eliminating signal redundancies. Historically, standards adopt carefully engineered energy-compacting transforms, prediction mechanisms, and entropy coding methods, meticulously designed to provide the greatest compression efficiency while maintaining tolerable computational cost. Data-driven approaches, or end-to-end learned compression (E2ELC), using neural networks have been studied for decades (Sonehara et al., 1989), and in recent years have come to dominate in terms of compression efficiency. Ideally, such approaches should increase the accessibility and practicality of designing application-specific compression systems that benefit from specialization to a specific signal, sensor type, or environment, without requiring years of laborious and expensive algorithm development, perceptual quality testing, and iteration. However, adoption of such methods, and therefore the realization of their benefits, has not occurred, even long after their superiority in terms of compression efficiency has been firmly established. The reasons for this are twofold: neural-network-based approaches are typically many times more expensive to encode, often differing by two or three orders of magnitude; and while they replace meticulously engineered transforms, quantization matrices, and prediction algorithms with a unified approach—gradient-based optimization of neural networks—they introduce a difficult-to-navigate landscape of loss functions, hyperparameters, and modality-specific neural architectures that must be meticulously engineered, and also require expensive GPU compute infrastructure. Thus, the vast majority of applications choose to adopt some existing standard even if the required bandwidth, target signal type, environment, or human-perception-oriented design are inappropriate.
However, these standards meet the compression ratios above only at severe perceptual cost (Jacobellis et al., 2024). More fundamentally, they are built on the zombie assumption that distortion should be optimized to reflect subjective quality to a human viewer or listener. In an earlier era, this objective accurately reflected the predominant use cases of multimedia compression. The validity of this assumption has long been put to rest by the proliferation of machines possessing superhuman perception, diffused throughout our society, and with unimaginable potential for benefit—more accurate medical diagnostics, safer transportation, and accelerated scientific discovery, to name a few.
Given irreducible power and bandwidth constraints, data compression is the primary tool that allows us to prioritize what information propagates from our resource-constrained sensor devices to our powerful but power-hungry machine consumers. The potential consequences of relying on compression methods designed for primate perceivers are profound, since this is tantamount to prioritizing by the degree of evolutionary benefit that a signal component provides above all else. For example, the blood-signaling hypothesis (Changizi et al., 2006) posits that the ability to discriminate subtle differences in red hues associated with spectral modulation of skin provides an emotional and socio-sexual signal that is explanatory of the unique color sensitivities of human vision. A compression system that maximizes subjective quality to a human will therefore disproportionately prioritize information encoding minor differences in red hues compared to other details. Human auditory and visual systems are also characterized by what they do not prioritize, or discard altogether: infrared, infrasound, ultrasound, and ultraviolet, for example. The opportunity for machine perception systems to complement human abilities—detecting minute anomalies in medical images, avoiding vehicle collisions, or making scientific discoveries from hyperspectral satellite image data—would be vastly hindered by inheriting the biases of human subjective perception.
The conventional codec development lifecycle, involving years or even decades of standardization for specific applications across international committees, cannot keep up with the pace of innovation in robotics, IoT, and other types of sensing. Standards bodies like MPEG operate on an eight- to ten-year cadence, providing time for customized hardware ASICs to implement each new standard (Bross et al., 2021, Sullivan et al., 2012, Wiegand et al., 2003, Zhang et al., 2026). However, machine perception systems, typically developed for and delivered to fully programmable CPUs, GPUs, or NPUs through pure software means, commonly progress through a major generation each year from a single developing entity alone (Bao et al., 2022, Caron et al., 2021, Oquab et al., 2024, Peng et al., 2022, Siméoni et al., 2026, Wang et al., 2023). Additionally, compression standards have historically been organized by a handful of international organizations—JPEG, MPEG, ITU-T, AOM, and DICOM—often working in collaboration. In contrast, the major developments in machine perception systems are distributed across hundreds of universities, global technology conglomerates, and venture-capital-funded startups, fueled by unprecedented levels of capital investment and often in an environment of intense competition. With each new year, the number of new scientifically or commercially significant machine perception systems is virtually uncountable. At present, decades-old codecs like JPEG, MP3, and AVC (Brandenburg, 1994, Wallace, 1992, Wiegand et al., 2003) remain the workhorse for the overwhelming majority of applications involving on-device encoding (e.g., streaming video feeds from aerial robots or streaming audio from a smartwatch), leading to a somewhat embarrassing outcome that is all-too common: cutting-edge, super-human AI foundation models on bleeding edge supercomputers fighting an uphill battle to ingest low quality, downsampled images compressed using antiquated methods operating at bitrates and on types of signals that they were never designed for.
The intentionally slow approach centered around standardization and efficient deployment via specialized ASICs and software is ill-suited for machine-oriented compression, and architectures should instead adopt the characteristics of their signal consumers: (1) end-to-end differentiability, to control behavior programmatically using isolated or joint deep learning optimization; (2) inference workflows that are amenable to programmable CPUs, NPUs, and MCUs; and (3) development via highly accessible and flexible frameworks, so that scientists and engineers are empowered to develop application-specific compression systems that meet the unique demands of their own domains, instead of choosing from a limited set of standards-defined options designed mostly for mass media broadcasting and telecommunications. This dissertation investigates machine-oriented compression: compression systems designed for machine perception rather than human perception, supporting model training and inference directly on compressed representations without additional decoding overhead. Table 1 pairs each limitation of conventional codecs with the machine-oriented design principle that addresses it; the paragraphs that follow expand on each principle, and Chapters 3–5 show how the three characteristics above are realized in codec architectures.
Limitation of conventional codecs |
Machine-oriented design principle |
|---|---|
Distortion is minimized for human perception, even as ML systems—capable of perceiving minute, high-frequency details—become the primary signal consumers. |
Preserve the features salient to machine perception, interoperating with unmodified pre-trained foundation models. |
Signals must be fully decoded into redundant representations (pixels, audio samples) before other systems can use them, adding overhead at every stage of ML training and inference. |
Train and run inference directly on compressed representations, with only partial decoding—reducing overhead at every stage of the ML pipeline and improving resolution scaling. |
Formats are fixed to stereo audio and RGB image/video, excluding biomedical signals, hyperspectral and volumetric imagery, multi-perspective robot video, and other emerging modalities. |
A modality-agnostic, end-to-end differentiable architecture that adapts to any sensor, signal type, environment, or downstream task. |
Newer standards target mass broadcast: expensive cloud encoding in exchange for cheap decoding on consumer devices. The opposite case—cheap encoding on battery-powered devices—is neglected. |
Invert the asymmetry: encoders simple enough for microcontrollers, with powerful cloud decoders that enhance and, when appropriate, resynthesize detail. |
Changing a standard takes years of committee work, and abandoning standards means abandoning the hardware and software infrastructure built around them. |
Bespoke codecs trained in hours on a single GPU, which remain compatible with existing infrastructure through one-time transcoding to standard formats. |
(1) Preserve machine-perceptual salience without adaptation. A machine-oriented codec should be immediately useful for general-purpose applications: paired with pre-trained foundation models, it should preserve the features salient to machine perception without any task-specific tuning. Chapter 3 demonstrates this, applying a single pre-trained codec, unmodified, across image classification, colorization, document understanding, and music source separation.
(2) Operate directly on compressed representations. Rather than fully decoding a signal back to pixels or audio samples before inference, models can train and run directly on the compressed representation, with only lightweight partial decoding (e.g., only the entropy decoding step). This reduces overhead at every stage of the ML pipeline and improves how models scale to high resolutions, since the representation is inherently lower-dimensional than the signal it encodes. Increasing input resolution substantially improves the accuracy of modern perception models, but at a steep cost in memory and latency; compressed-domain learning attenuates this cost by replacing resolution reduction with dimensionality reduction (Chapter 3).
(3) Adapt end to end to any sensor, signal, environment, or task. Because every component is differentiable, the same codec can be adapted end to end—to a specific sensor, signal type, or environment, or jointly with a downstream ML application—whenever specialization is justified. Chapter 4 extends this to spatial audio arrays, hyperspectral remote sensing data, and 3D medical images; Chapter 7 trains both general-purpose and task-aware transcoding pipelines for a pre-trained, frozen encoder.
(4) Make the encoder simple enough for MCUs. Machine-oriented compression divides labor asymmetrically. The encoder reduces to a static bank of sub-critically sampled filters producing inherently lower-dimensional outputs, a simple and efficient pointwise companding nonlinearity, and precomputed entropy coding tables. There is no online rate-distortion optimization requiring multiple encoding passes; variable-rate and progressive coding come from an embarrassingly parallel encoder DAG with no decoder in the loop (Chapter 5). The encoders developed in Chapters 3 and 5 reduce to a single linear layer with fewer than 100k parameters and, at low bitrates, cost less to run than a JPEG encoder—within reach of commodity CPUs, NPUs, and MCUs rather than the GPUs and hardware-baked ASICs on which newer standards increasingly rely. The heavy computation moves to the cloud, where powerful DNN decoders enhance the signal and, when appropriate for the application, resynthesize useful details.
(5) Make bespoke codecs accessible, and keep them compatible. Encoders this simple are cheap to develop as well as to run: trained in hours on a single GPU, a codec for a new sensor or task requires limited resources rather than extensive research, engineering, or supercomputers. Yet decades of hardware and software systems have been built around standards like JPEG—codecs baked into ASICs and SoCs, fast dataloaders, ML frameworks, and web browsers—and would be perilous to discard. We show that a learned, sensor-embedded codec can instead coexist with this ecosystem: a one-time transcode in the cloud converts the learned representation into a standard file that every existing consumer can decode (Chapter 7).
Organization#
The remainder of this dissertation develops these ideas in six chapters, proceeding from measurement, to codec design, to complete systems.
Chapter 2, Machine Perceptual Quality, establishes how existing ML systems are impacted by lossy compression from conventional, neural, and generative codecs. Additionally, this chapter characterizes the current landscape and interactions of objective and perceptual distortion metrics; conventional, learned, and generative codecs; and machine learning foundation models. We also demonstrate why reversible or “lossless” compression is not an appropriate goal for machine perceptual signals, due to the realities of noisy data and fundamental limitations of the current generation of foundation models. Across image classification, segmentation, speech recognition, and music source separation, we find that deep similarity metrics—originally designed to predict human judgments—are strong predictors of machine perceptual quality, and that machine perception can match, and in some cases exceed, the lossless baseline even under severe lossy compression, which we define as compression ratios between 20:1 and 1000:1.
Chapter 3, Learned Compression for Compressed Learning, introduces the first machine-oriented codec design that combines the benefits of conventional transform coding with end-to-end learned compression using neural networks. Chapters 3 and 4 develop the first machine-oriented compression systems for audio, images, video, and other data types, based on encoding-efficient asymmetric autoencoders (EE-AAEs) that directly optimize the rate-distortion Lagrangian \(D + \lambda R\) in such a way that the distortion can either be general-purpose—suitable for consumption by humans and pre-trained machine perception systems—or specialized to specific tasks. For audio and image signals, this design provides cheap encoding, high compression efficiency, and uniform dimension reduction to accelerate downstream models. Using codecs adopting this design, we propose a simple but universal strategy for compressed-domain learning and demonstrate improved resolution scaling properties compared to large patch transformers or highly strided CNNs. Together, our proposed codec design and compressed learning framework significantly improve the accuracy-computation trade-off for high-resolution discriminative ML applications.
Chapter 4, Lightweight, Versatile Codec Design, expands on this framework with a lightweight, versatile, and asymmetric codec design. Using FFT-like structured operations in the analysis transform, linear attention in the synthesis transform, and simplified training objectives, we demonstrate state-of-the-art performance in terms of the rate-distortion-complexity tradeoff while maintaining a modality-agnostic architecture. To demonstrate the versatility of this approach, we train codecs for stereo and multi-channel spatial audio, RGB and hyperspectral images, video, and 3D medical images, achieving competitive performance in each case.
Chapter 5, Variable-Rate Compression and Projection-Pursuit Encoding, addresses a remaining obstacle to deploying learned codecs on resource-constrained sensors: rate adaptation. We propose FRAPPE, a residual autoencoding framework that uses the full input to predict the residual output via a projection-pursuit encoder. The encoding objective naturally sorts latent channels by importance, enabling zero-overhead variable-rate and progressive coding using a single set of encoder weights, while keeping the analysis path an embarrassingly parallel DAG of independent projections—no recurrence, no quantizer chain, and no decoder in the loop. At high compression ratios (\(\sim\)0.1 bpp), the resulting image codec provides higher perceptual quality than AVIF with 47\(\times\) faster encoding, making it capable of real-time 1080p, 30 fps CPU-only encoding.
Chapter 6, Video and Real-Time Sensing, turns from individual signals to real-time systems, focusing on video understanding for robotics. In applications where predictions must be available on device by a guaranteed deadline, round-trip communication delay makes standard cloud inference a non-starter. Instead, we propose a framework that feeds temporally predictive outputs of a delayed remote model as side information to a local model with access to the current signal, leading to both accurate and on-time predictions.
Chapter 7, Sensor-Embedded Autoencoding with One-Time Transcode, addresses compatibility with existing infrastructure. SEAOTTER pairs a sensor-embedded encoder with a one-time transcode performed in the cloud: a powerful decoder reconstructs the signal and re-encodes it as a standard JPEG file, so downstream consumers, from training pipelines to web browsers, need nothing more than an ordinary JPEG decoder. A naive transcode would degrade quality; instead, the JPEG color transform and quantization matrices are learned end to end, and the transcode increases accuracy across global, dense, and vision-language tasks relative to the underlying autoencoder.
Taken together, these chapters demonstrate that compression can provide more than a compromise between quality and bitrate: when compression systems are designed for machine perception from the start, encoder cost, compression efficiency, and compatibility need not be traded against one another. The result is a practical framework for sensing and perception under extreme power, bandwidth, latency, or accuracy constraints.
References#
Niranjan Balasubramanian, Aruna Balasubramanian, and Arun Venkataramani. Energy consumption in mobile phones: a measurement study and implications for network applications. In Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, 280–293. 2009.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations. 2022. URL: https://openreview.net/forum?id=p-BhZSz59o4.
Lucas Beyer. On the speed of ViTs and CNNs. lb.eyer.be/a/vit-cnn-speed, 2024.
Karlheinz Brandenburg. ISO-MPEG-1 audio: a generic standard for coding of high-quality digital audio. Journal of the Audio Engineering Society, 42(10):780–792, 1994.
Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm. Overview of the versatile video coding (VVC) standard and its applications. IEEE Transactions on Circuits and Systems for Video Technology, 31(10):3736–3764, 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 9630–9640. 2021.
Aaron Carroll and Gernot Heiser. An analysis of power consumption in a smartphone. In 2010 USENIX Annual Technical Conference (USENIX ATC 10). 2010.
Mark A Changizi, Qiong Zhang, and Shinsuke Shimojo. Bare skin, blood and the evolution of primate colour vision. Biology Letters, 2(2):217, 2006.
Denis Guangyin Chen, Fang Tang, Man-Kay Law, and Amine Bermak. A 12 pj/pixel analog-to-information converter based 816× 640 pixel cmos image sensor. IEEE Journal of Solid-State Circuits, 49(5):1210–1222, 2014.
Agrim Gupta, Adel Heidari, Avyakta Kalipattapu, Ish Kumar Jain, and Dinesh Bharadia. 3 w's of smartphone power consumption: who, where and how much is draining my battery? In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, 2248–2250. 2024.
Dan Jacobellis, Daniel Cummings, and Neeraja J Yadwadkar. Machine perceptual quality: evaluating the impact of severe lossy compression on audio and image models. In Data Compression Conference. IEEE, 2024.
Sangwoo Kim, Taehyoung Kim, Kiwon Seo, and Gunhee Han. A fully digital time-mode CMOS image sensor with 22.9 pj/frame.pixel and 92db dynamic range. In 2022 IEEE International Solid-State Circuits Conference (ISSCC), volume 65, 1–3. IEEE, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. URL: https://openreview.net/forum?id=a68SUt6zFt.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366, 2022. URL: https://arxiv.org/abs/2208.06366.
Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seung Eun Yi, Michael Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Herve Jegou, Patrick Labatut, and Piotr Bojanowski. DINOv3. Transactions on Machine Learning Research, 2026. URL: https://openreview.net/forum?id=2NlGyqNjns.
Noboru Sonehara, Mitsuo Kawato, Sei Miyake, and Kunihiko Nakane. Image data compression using a neural network model. In International 1989 Joint Conference on Neural Networks, 35–41. 1989.
Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on Circuits and Systems for Video Technology, 22(12):1649–1668, 2012.
Jacopo Tosi, Fabrizio Taffoni, Marco Santacatterina, Roberto Sannino, and Domenico Formica. Performance evaluation of bluetooth low energy: a systematic review. Sensors, 17(12):2898, 2017.
Gregory K Wallace. The JPEG still picture compression standard. IEEE Transactions on Consumer Electronics, 38(1):xviii–xxxiv, 1992.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and others. Image as a foreign language: BEiT pretraining for vision and vision-language tasks. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19175–19186. 2023.
Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the H.264/AVC video coding standard. IEEE Transactions on Circuits and Systems for Video Technology, 13(7):560–576, 2003.
Wenzhuo Zhang, Luyi Qin, Xinxin Chen, Nianxiang Fu, Haodong Qu, Wenzhuo Ma, Junxi Zhang, and Zhenzhong Chen. Learning-enhanced video compression with capability beyond VVC. In 2026 IEEE International Symposium on Circuits and Systems (ISCAS), 244–248. 2026.