An Overview of Deflate64
Deflate64 is an extended variation of the legacy Deflate procedure. The basic workings of the procedure remain unchanged from traditional Deflate, but the addressable size of the dictionary has been expanded from 32kbytes to 64 kbytes, offering improved performance.
Improved Compression Even at Slower Speeds
One of the most interesting things about Deflate64 is that it offers improved compression rates over Deflate, even when run at lower processing speeds. Deflate64 was designed by PKWARE and is included in the ZIP specification as compression method number 9. It is also sometimes referred to as Enhanced Deflate.
Deflate is supported by PKWARE, 7-Zip and Info-ZIP. It is not supported in the zlib project because it is considered to be a proprietary format. Deflate64 is covered by US patent 5051745. However, in spite of its proprietary nature, it is still popular in some open source circles.
Other Changes in Deflate64
In conventional Deflate, the distance codes 30 and 31 were not used. In Deflate64 those codes have been extended to address the full 64kbyte range. There were 14 extra bits assigned to each code. In addition, the last length code, 285, has been extended by 16 bits to remove the limitation of 258 byte sequences.
How Deflate64 Works
Compression takes place in two stages. The first stage involves matching duplicate strings and replacing them with pointers, while the second stage replaces symbols with other symbols that are weighted based on the frequency with which each symbol appears.
A tree is created, which contains space for up to 288 symbols. Within that tree, the first 256 blogs represent literal bytes or symbols, while the remainder contains special extra-bits.
Deflate64 relies on an encoder that runs during the compression stage looking for matching strings. There is a sliding scale that allows the user to select whether they want high speed compression or a higher compression level. The setting -9 offers the highest level of compression, while -0 instructs the encoder to simply store the data uncompressed.
Other Deflate encoders exist, and all Deflate encoders produce bitstreams which are compatible with each other. The encoded bit-stream may differ between implementations but each encoder can read and decompress other encoder’s streams.
Uses of Delfate64
Deflate64 is used in gzip, many zip files, and also PNG image files. It is also used in Crypto__ and in PuTTY’s sshzlib implementation. In addition, the Deflate Stream implementation used by the .NET framework makes use of Deflate64.