Abstract

Modern sensors—ranging from wearables and smartphones to medical devices and satellites—produce increasingly rich streams of high-resolution data. Existing media compression standards, or “codecs,” such as JPEG or MPEG can reduce storage and transmission requirements by orders of magnitude by discarding imperceptible details and eliminating redundancies.

However, these standards were designed under assumptions that no longer hold: (1) that a human, rather than a machine learning (ML) system, is the ultimate consumer of the signal; (2) that the signal must be fully decoded before any other system can use it; (3) that the signals of interest are limited to conventional media formats such as stereo audio and RGB images or video; (4) that encoding happens where power is plentiful and decoding where it is scarce; and (5) that applications can abide years—or decades—of standardization before benefiting from compression.

Under these assumptions, the vast majority of applications are faced with a difficult choice: (A) design and maintain a bespoke compression system, (B) use or repurpose an existing but ill-suited standard, or (C) rely exclusively on general-purpose, bit-reversible compression methods. Option (A) demands years of specialized algorithm development, perceptual quality testing, and expensive compute infrastructure, prohibitive without extraordinary resources. Adopting (B), an ill-suited but readily available standard, eliminates the engineering, implementation, and maintenance burden, but use outside of the intended design squanders bits on irrelevant details while discarding information relevant to the task at hand. Holding to (C), an uncompromising standard for signal preservation, too often ends with signals discarded entirely, since no compression method can indiscriminately preserve every signal minutia while meeting the application’s compute, power, bandwidth, and storage constraints.

To escape this trilemma, this dissertation proposes a design philosophy, attendant principles, and techniques for next-generation compression systems, which we refer to as machine-oriented compression. Machine-oriented compression is characterized by a design goal answering each of these assumptions: (1) preservation of features salient to machine perception, interoperating with unmodified pre-trained foundation models; (2) model training and inference directly on compressed representations, without additional decoding overhead; (3) end-to-end optimization and adaptation for specific sensors, signals, environments, and downstream tasks; and (4) lightweight encoding suitable to run on low power and programmable NPUs, CPUs, or even MCUs, rather than high power GPUs or hardware-baked ASICs, which newer compression standards increasingly rely on. And finally, (5) it aims to increase the accessibility of building bespoke codecs using data-driven design and limited resources rather than extensive research, engineering, or supercomputers, while remaining compatible, when appropriate, with existing codecs and the decades of hardware and software sedimented upon them, which would be perilous to discard.

This dissertation develops these principles as follows. Chapter 2 presents a philosophy for measuring machine perceptual quality of compressed signals based on task-specific performance metrics for downstream machine perception systems. Chapters 3 and 4 develop the first machine-oriented compression systems for audio, images, video, and other data types, based on encoding-efficient asymmetric autoencoders (EE-AAEs) that directly optimize the rate-distortion Lagrangian \(D + \lambda R\). Chapter 3 also introduces a framework for training and inference directly on compressed data, and demonstrates how this approach provides an efficiency multiplier when scaling the resolution of machine perception systems. Chapter 4 introduces several technologies that increase the performance, accessibility, and flexibility of EE-AAEs, and demonstrates the feasibility of developing machine-oriented compression systems for signals beyond stereo audio and RGB images/video, including spatial audio arrays, hyperspectral remote sensing data, and 3D medical images. Chapter 5 demonstrates an efficient and practical method for variable-rate and progressive compression with EE-AAEs, allowing the computation and bandwidth requirements of encoding to scale dynamically to the available resources. Chapter 6 extends machine-oriented compression to real-time systems such as video-based autonomous robots, whose safety, reliability, and latency constraints have long precluded cloud-based processing. Chapter 7 demonstrates a practical, negative-distortion transcoding system allowing machine-oriented compression systems to remain fully compatible with the entrenched hardware and software infrastructure of existing compression standards.

Together, these chapters inaugurate the machine-oriented compression design philosophy, instantiated across standard audiovisual multimedia as well as nonstandard signals ill-served by existing codecs, demonstrating that compression can enable profoundly new capabilities for sensing and perception under extreme power, bandwidth, latency, or accuracy constraints.