CUBR-0037 · transparency
Benchmark methodology
How every number on this site is measured — so you do not have to take our word for it. Corpora, exact flags, the metric, and the checks that make it impossible for us to quietly cheat.
1. Corpora — what we measure on, and why
We use the standard, publicly downloadable corpora the compression field has converged on. They are not picked to flatter Cubrim — they are the same files every serious compressor is judged against, spanning every data type including the ones where Cubrim currently loses.
Silesia (≈212 MB) — the modern reference set: executables, images, medical scans, structured binary, databases, and text. Deliberately diverse, so no single specialisation can win it.
enwik8 — the first 100 MB of an English Wikipedia dump, the canonical natural-language text benchmark (the basis of the Hutter Prize).
The Canterbury Corpus — the long-standing academic set used in compression research since 1997.
An early in-house 10-file set used during bring-up. It is kept only as a historical reference — leaderboard numbers there are NOT claimed as general results, precisely because a corpus you tune on can mislead (see the overfit note below).
2. Archivers and their exact flags
Every rival is run at its strongest practical setting, so the comparison is honest rather than rigged against weak baselines. These are the exact invocations — anyone can reproduce them:
| archiver | verified version | flags | Binary sha256 |
|---|---|---|---|
| cubrim | 0.3.2 | v0.3.2 published release binary sha256 b6c3cd251f7148c1895f5b85d30d06df8252a70afbd649e269f673a19e2a5768; tag v0.3.2 = commit 09ef2bbd00c359c5485a0ef4c6fd59f464382270, whose only difference from dfb195ef089db738e51153ad4532fdd583f247bf is the version bump, so both produce byte-identical output; full24 RT cmp=0; exactly five verified cell corrections | b6c3cd251f71… |
| gzip | 1.12 | -9 | afea077ce127… |
| bzip2 | 1.0.8 | -9 | 8e728b8ed28f… |
| xz | 5.4.5 | -9e | b5b163eb2732… |
| zstd | 1.5.5 | --ultra -22 | 7c5468b370f7… |
| brotli | 1.1.0 | -q 11 | 01969d4716e4… |
| lz4 | 1.9.4 | -12 | 87c0d5d060fd… |
| ppmd | 7-Zip 23.01 | 7z -m0=PPMd | 60fc00b4e1ed… |
| 7z | 23.01 | -m0=LZMA2 -mx9 | 60fc00b4e1ed… |
| rar | 7.00 | a -m5 | 338274d32151… |
The flags come from the validated live meta-35 payload. The database stores command labels, not package versions; the version column comes from the controlled 2026-07-26 benchmark-host package capture and the checksum-verified public Cubrim v0.3.2 binary. Cubrim runs its built-in competitive scheme selection.
Why some charts do not start at zero
Compression ratios on this corpus sit between roughly 0.19 and 0.38, so a bar chart anchored at zero would compress every real difference into the top third of the image and hide exactly what the comparison is about. Where an axis is truncated we label the break on the axis itself, and every exact ratio stays printed next to the bar, so nothing depends on reading a length. On narrow screens the same data is shown as two labelled groups instead of a broken axis.
What "competitive" means for Cubrim competitive
The other archivers in the table run at a fixed level (gzip -9, zstd --ultra -22). Cubrim has no single "level". For each file the encoder competitively tries several internal value-coding schemes — bit-packing, RLE codes, context Huffman, and the BWT family with geomix mixing — measures each, and writes the smallest one to the archive, tagging it with a one-byte scheme identifier. Because it picks the minimum across the candidates and the previous best, a new scheme can never make things worse — the selection is structurally regression-proof. So "competitive (built-in scheme selection)" in the Cubrim column is not a level but an architecture: the codec chooses the best scheme for the data at hand.
3. The metric
We report the compression ratio — compressed size divided by original size. Lower is better. File groups and the overall leaderboard are size-weighted: Σ compressed bytes ÷ Σ original bytes across the selected files, the same method used by Silesia/lzbench. This is not an average of per-file ranks; no speed weighting, no cherry-picked units.
4. Why we cannot quietly cheat
Three properties make the results self-checking. They are not promises — they are mechanical constraints baked into how the benchmark runs.
Round-trip byte-exact, automatically verified
For every file, Cubrim must decompress back to the original bit-for-bit. If a single byte differs, the result is invalid and discarded. A lossless compressor cannot fake a small number by losing data — the round-trip check catches it.
Competitive rail — no per-file tuning
Cubrim selects the smallest of its own internal schemes for each input via a fixed competitive rule. It does not hand-tune parameters to a specific file, so a good number reflects the codec, not a knob turned for that one case.
Every run carries a code_sha
Each benchmark is stamped with the exact git commit of the codec that produced it. The numbers are reproducible against that specific commit — not a moving, unverifiable target.
Reproduce it yourself
A packaged reproducer rebuilds the whole benchmark from public sources on your own hardware and checks every result against ours. Nine of the ten archivers must match byte-for-byte, including every Cubrim number; rar is held to its round trip and a bounded margin, because it stores file timestamps inside the archive. A mismatch is useful evidence — please report it.
Run the reproducer →5. Honesty as a principle
- We publish the current truth, including when an older review is stale. Meta-35 ranks Cubrim first in all six type aggregates: on text Cubrim is 0.176001 versus PPMd at 0.201379; on executables Cubrim is 0.244212, ahead of 7z at 0.274874, xz at 0.275453, and Brotli at 0.284998. The remaining measured losses are per-file: on xargs.1 Brotli is 0.346344 versus Cubrim at 0.380175; on nci xz is 0.043192 and Brotli 0.045293 versus Cubrim at 0.046335.
- We publish the dead ends. Hypotheses that did not work are kept as NO-GO cards in the evolution feed, with the measured reason they failed — not deleted.
- We admit overfitting openly. An early milestone (H-24) edged gzip on the tuned corpus but proved 2.2× worse than gzip on a disjoint holdout — we documented that regression rather than hiding it, which is exactly why we now lead with the world corpora.
- We re-validate when the dataset changes. Switching to a new corpus means re-measuring from scratch; a result is only as honest as the data it was last checked against.
6. Openness
The whole hypothesis race is public. Every idea, every measurement, and every verdict — accepted or rejected — is recorded in the evolution feed as it happens. There is no private leaderboard.