N-gram compression outperforms zlib by 33%

An order-4 n-gram model estimates the probability of the next token in Python source, and that probability is fed into arithmetic coding for compression.

• A roughly 575 KB Flask .py file was reduced to 101 KB (0.176x), achieving 82.4% compression.
• On the same data, this was smaller than zlib 151 KB, lzma 152 KB, and zstd 147 KB.
• The implementation uses Python for the tokenizer and model, and Zig for the arithmetic coder, connected via ctypes.
• Encoding speed was about 75 seconds, roughly 1600x slower than zlib (0.05 seconds).
• The bottleneck is that per-token probability calculation happens in Python, with no caching or batching.