CUBR-0037 · transparency
Benchmark methodology
How every number on this site is measured — so you do not have to take our word for it. Corpora, exact flags, the metric, and the checks that make it impossible for us to quietly cheat.
1. Corpora — what we measure on, and why
We use the standard, publicly downloadable corpora the compression field has converged on. They are not picked to flatter Cubrim — they are the same files every serious compressor is judged against, spanning every data type including the ones where Cubrim currently loses.
Silesia (≈212 MB) — the modern reference set: executables, images, medical scans, structured binary, databases, and text. Deliberately diverse, so no single specialisation can win it.
enwik8 — the first 100 MB of an English Wikipedia dump, the canonical natural-language text benchmark (the basis of the Hutter Prize).
The Canterbury Corpus — the long-standing academic set used in compression research since 1997.
An early in-house 10-file set used during bring-up. It is kept only as a historical reference — leaderboard numbers there are NOT claimed as general results, precisely because a corpus you tune on can mislead (see the overfit note below).
2. Archivers and their exact flags
Every rival is run at its strongest practical setting, so the comparison is honest rather than rigged against weak baselines. These are the exact invocations — anyone can reproduce them:
| archiver | flags |
|---|---|
| cubrim | competitive-min corrected: CUBR-0043 CM19 rows + prior verified rows + fresh exact-317a323 replacements for nci/ptt5/kennedy.xls |
| gzip | -9 |
| bzip2 | -9 |
| xz | -9e |
| zstd | --ultra -22 |
| brotli | -q 11 |
| lz4 | -12 |
| ppmd | 7z -m0=PPMd |
| 7z | -m0=LZMA2 -mx9 |
| rar | a -m5 |
cubrim runs its built-in competitive scheme selection (it picks the smallest of its own schemes per input). The flags above are read directly from the published benchmark JSON, so they cannot drift from what was actually measured.
What "competitive" means for Cubrim competitive
The other archivers in the table run at a fixed level (gzip -9, zstd --ultra -22). Cubrim has no single "level". For each file the encoder competitively tries several internal value-coding schemes — bit-packing, RLE codes, context Huffman, and the BWT family with geomix mixing — measures each, and writes the smallest one to the archive, tagging it with a one-byte scheme identifier. Because it picks the minimum across the candidates and the previous best, a new scheme can never make things worse — the selection is structurally regression-proof. So "competitive (built-in scheme selection)" in the Cubrim column is not a level but an architecture: the codec chooses the best scheme for the data at hand.
3. The metric
We report the compression ratio — compressed size divided by original size. Lower is better. File groups and the overall leaderboard are size-weighted: Σ compressed bytes ÷ Σ original bytes across the selected files, the same method used by Silesia/lzbench. This is not an average of per-file ranks; no speed weighting, no cherry-picked units.
4. Why we cannot quietly cheat
Three properties make the results self-checking. They are not promises — they are mechanical constraints baked into how the benchmark runs.
Round-trip byte-exact, automatically verified
For every file, Cubrim must decompress back to the original bit-for-bit. If a single byte differs, the result is invalid and discarded. A lossless compressor cannot fake a small number by losing data — the round-trip check catches it.
Competitive rail — no per-file tuning
Cubrim selects the smallest of its own internal schemes for each input via a fixed competitive rule. It does not hand-tune parameters to a specific file, so a good number reflects the codec, not a knob turned for that one case.
Every run carries a code_sha
Each benchmark is stamped with the exact git commit of the codec that produced it. The numbers are reproducible against that specific commit — not a moving, unverifiable target.
5. Honesty as a principle
- We publish the losses. Cubrim currently leads the overall world aggregate, but the benchmark still shows where other archivers win by type or file — ppmd on text, 7z on exe/binary, and individual file losses inside otherwise winning types.
- We publish the dead ends. Hypotheses that did not work are kept as NO-GO cards in the evolution feed, with the measured reason they failed — not deleted.
- We admit overfitting openly. An early milestone (H-24) edged gzip on the tuned corpus but proved 2.2× worse than gzip on a disjoint holdout — we documented that regression rather than hiding it, which is exactly why we now lead with the world corpora.
- We re-validate when the dataset changes. Switching to a new corpus means re-measuring from scratch; a result is only as honest as the data it was last checked against.
6. Openness
The whole hypothesis race is public. Every idea, every measurement, and every verdict — accepted or rejected — is recorded in the evolution feed as it happens. There is no private leaderboard.