GGDLPC

GGDLPC#

pip install GGDLPC

GGDLPC is a lossless entropy coding method and package: linear predictive coding residuals are modeled with generalized Gaussian distributions, and every table the coder uses — statistic thresholds, per-cell symbol distributions, canonical Huffman codes, the zero-run mode — is generated at load time from six offline-fitted scalars per channel. One call encodes one integer tensor (values in \([-N, N]\), \(N \le 65536\); rank 1, 2, or 3) as one self-contained byte-padded bitstream, truncatable at any channel boundary. A differentiable rate proxy built from the same model supports gradient-based training of the transforms that feed the coder.

Memory model of the C engine#

The engine keeps no table storage of its own. A channel (glpc_channel_t) holds the six-scalar-derived state that is genuinely per channel — predictor weights, integer statistic edges, run parameters — and points at its cells; a cell (glpc_cell_t) holds its split configuration and points at an encoder table and/or a decoder table. Tables are direction-specific (glpc_enc_table_t: canonical codewords; glpc_dec_table_t: first-code/offset/count per length), owned by the caller, and shareable, so an exporter pools identical cells across channels and places the pool and the cell arrays in flash as const data generated offline; a deployment that only encodes builds no decoder tables at all. The resident RAM per channel is then the channel struct alone (≈0.45 kB at \(N = 31\) with -DGLPC_MAX_G=41, ≈0.9 kB at the default limits, on a 32-bit target), flat in bit depth and channel count. The Python bridge allocates the same objects per channel on the heap and pools tables across a Codec’s channels (Codec.table_pool); Codec(blob, enc=True, dec=False) builds an encode-only coder. GGDLPC.emit_c(blob, enc=, dec=, prefix=) generates that const data as C99 source — the pooled table arrays, per-channel cell arrays, and a glpc_channel_rec_t record per channel from which GLPC_CHANNEL_CONFIG_REC(&ch, &rec) fills the RAM struct at boot — with the codewords read back from the engine’s own glpc_tables_init, so there is one implementation of the canonical code.

Compile-time knobs (#ifndef-guarded in ggdlpc.h): GLPC_MAX_G (grid cells; 96 by default, 41 suffices at \(N = 31\)), GLPC_MAX_ALPHA (windowed alphabet, 64), GLPC_MAX_LEN (codeword bits, 63), and GLPC_CODE_BITS (codeword storage width, 64; the fitted tables floor cell probabilities at \(10^{-12}\), so real codewords reach 47 bits — a 32-bit build rejects such a table with GLPC_ERR_TABLE rather than truncating it). The bitstream does not depend on any knob.

Predictor fitting is selectable per data set (0.0.3): GGDLPC.fit takes keyword-only predictor ('ls', the default unconstrained ridge solve with intercept; 'ls_dc_invariant', weights summing to exactly one with no intercept, for records carrying large DC offsets; 'fixed'), bias, loss ('l2' or IRLS 'l1'), and demean; the defaults reproduce the original recipe value for value, and the coder cannot tell how the integer weights were chosen.

Walkthrough#

The chapters below are interactive demonstrations of how the coder works, not API references. Each notebook keeps its intermediate quantities in notebook scope (residuals, statistics, cell indices, code lengths, bitstream bytes), so added cells can inspect any step. Where the package relies on its C engine, the notebooks reimplement the same steps in plain Python and verify byte identity against the C stream.