§5.2 The rzip compression algorithm 90
using small matches or using entropy encoding. I used bzip2[Seward 1998] for this.
From the above it should be clear that rzip is intended to be used to compress files, not streams of bytes. The matching algorithm requires that fast random access is pos-
sible in the file being compressed in order to find the block matches and the multiple stream encoding used in the output file requires random access when uncompressing.
The algorithm also has memory requirements which rise as the log of the file size in order to store the hash table entries. This logarithmic rise stops at a file size of 2
16 +11
after which memory usage is constant at 2
16
+ 11 × 2
15
hash entries. The use of a block sorting compression algorithm as the final stage of rzip greatly
simplified the design of rzip, and allowed rzip to perform well on a much wider range of files. The Lempel-Ziv compression method is not a state of the art compression
method although it is ideally suited to adaption for coarse-grained block matching us- ing ideas derived from rsync. Block sorting compression algorithms are much closer
to state of the art[Fenwick 1996], while being quite efficient. By using a block sorting compression algorithm as the final stage in rzip the algorithm can obtain good fine-
grained compression while also taking advantage of any long range block repetitions.
5.2.3 Testing rzip
A compression algorithm is useless unless it performs well on real data that people might want to compress. To test rzip I first looked at existing text compression data
sets, such as the Calgary Text Corpus[Witten and Bell 1990] but I found that the ex- isting test data sets contained files that are far too small to show the advantages of
long-distance block matches. To address this I put together a new Large Text Cor- pus[Tridgell 1998] consisting of five large files taken from real world applications.
The files are 1. A tar file of the source code to GNU Emacs version 20.2.
2. A tar file of the source code to the Linux kernel, version 2.1.100. 3. A tar file snapshot of my Samba work area on my workstation.
§5.2 The rzip compression algorithm 91
4. A snapshot of a two month archive of the linux-kernel mailing list mirror which I run on my machine.
5. A snapshot of the unsolicited commercial email caught by the “spam filter” on my machine over a two week period.
The Samba snapshot has a high degree of internal redundancy because it contains multiple copies of different versions of the source code of Samba in various states of
testing. Although each version differs slightly from the other versions, they contain a large amount of common text. The “spamfile” also contains a lot of widely spaced
redundancy, due to the nature of unsolicited commercial email. Such email tends to be very repetitive and the people who send such email usually send it to many addresses
on the one machine
5
. These files were chosen to represent a range of commonly compressed data while
being freely distributable so that other researchers can test different algorithms on the same data.
Table 5.2 shows the result of applying gzip
6
, bzip2
7
and rzip
8
to the five test files. The results show that rzip was able to find some long-distance redundancy in all the
files, giving improved compression ratios for each file. The small improvements for the Emacs and Linux source code are probably due to repeated copyright headers
and patterns of include files, while the improvement for the mailing list archive is probably a result of quoting and repeated mail headers and email signatures.
The improvements for the Samba snapshot and the spamfile are much larger. The Samba snapshot demonstrates the effectiveness of rzip at finding the common com-
ponents in a repetitive source code archive which I believe would be typical of a programmer’s home directory. The largest improvement is for the highly redundant
spamfile, which has a compression ratio almost three times larger than the Lempel-Ziv based gzip and almost twice that for the bzip2 block sorting compressor.
5
Several people have commented that the best way to compress spam is to delete it, and while I tend to agree it does make for a good example of data with a very high degree of internal redundancy.
6
gzip version 1.2.4 was used with maximum compression selected.
7
bzip2 version 0.1pl2 was used with maximum compression selected.
8
The rzip snapshot from July 2nd 1998 was used with the default options.
§5.3 Incremental backup systems 92
