lib/3rdparty/bzip2/manual_2.html

   1 <HTML>
   2 <HEAD>
   3 <!-- This HTML file has been created by texi2html 1.54
   4      from manual.texi on 23 March 2000 -->
   5
   6 <TITLE>bzip2 and libbzip2 - How to use bzip2</TITLE>
   7 <link href="manual_3.html" rel=Next>
   8 <link href="manual_1.html" rel=Previous>
   9 <link href="manual_toc.html" rel=ToC>
  10
  11 </HEAD>
  12 <BODY>
  13 <p>Go to the <A HREF="manual_1.html">first</A>, <A HREF="manual_1.html">previous</A>, <A HREF="manual_3.html">next</A>, <A HREF="manual_4.html">last</A> section, <A HREF="manual_toc.html">table of contents</A>.
  14 <P><HR><P>
  15
  16
  17 <H1><A NAME="SEC2" HREF="manual_toc.html#TOC2">How to use <CODE>bzip2</CODE></A></H1>
  18
  19 <P>
  20 This chapter contains a copy of the <CODE>bzip2</CODE> man page,
  21 and nothing else.
  22
  23 </P>
  24
  25 <BLOCKQUOTE>
  26
  27
  28
  29 <H4><A NAME="SEC3" HREF="manual_toc.html#TOC3">NAME</A></H4>
  30
  31 <UL>
  32 <LI><CODE>bzip2</CODE>, <CODE>bunzip2</CODE>
  33
  34 - a block-sorting file compressor, v1.0
  35 <LI><CODE>bzcat</CODE>
  36
  37 - decompresses files to stdout
  38 <LI><CODE>bzip2recover</CODE>
  39
  40 - recovers data from damaged bzip2 files
  41 </UL>
  42
  43
  44
  45 <H4><A NAME="SEC4" HREF="manual_toc.html#TOC4">SYNOPSIS</A></H4>
  46
  47 <UL>
  48 <LI><CODE>bzip2</CODE> [ -cdfkqstvzVL123456789 ] [ filenames ...  ]
  49
  50 <LI><CODE>bunzip2</CODE> [ -fkvsVL ] [ filenames ...  ]
  51
  52 <LI><CODE>bzcat</CODE> [ -s ] [ filenames ...  ]
  53
  54 <LI><CODE>bzip2recover</CODE> filename
  55
  56 </UL>
  57
  58
  59
  60 <H4><A NAME="SEC5" HREF="manual_toc.html#TOC5">DESCRIPTION</A></H4>
  61
  62 <P>
  63 <CODE>bzip2</CODE> compresses files using the Burrows-Wheeler block sorting
  64 text compression algorithm, and Huffman coding.  Compression is
  65 generally considerably better than that achieved by more conventional
  66 LZ77/LZ78-based compressors, and approaches the performance of the PPM
  67 family of statistical compressors.
  68
  69 </P>
  70 <P>
  71 The command-line options are deliberately very similar to those of GNU
  72 <CODE>gzip</CODE>, but they are not identical.
  73
  74 </P>
  75 <P>
  76 <CODE>bzip2</CODE> expects a list of file names to accompany the command-line
  77 flags.  Each file is replaced by a compressed version of itself, with
  78 the name <CODE>original_name.bz2</CODE>.  Each compressed file has the same
  79 modification date, permissions, and, when possible, ownership as the
  80 corresponding original, so that these properties can be correctly
  81 restored at decompression time.  File name handling is naive in the
  82 sense that there is no mechanism for preserving original file names,
  83 permissions, ownerships or dates in filesystems which lack these
  84 concepts, or have serious file name length restrictions, such as MS-DOS.
  85
  86 </P>
  87 <P>
  88 <CODE>bzip2</CODE> and <CODE>bunzip2</CODE> will by default not overwrite existing
  89 files.  If you want this to happen, specify the <CODE>-f</CODE> flag.
  90
  91 </P>
  92 <P>
  93 If no file names are specified, <CODE>bzip2</CODE> compresses from standard
  94 input to standard output.  In this case, <CODE>bzip2</CODE> will decline to
  95 write compressed output to a terminal, as this would be entirely
  96 incomprehensible and therefore pointless.
  97
  98 </P>
  99 <P>
 100 <CODE>bunzip2</CODE> (or <CODE>bzip2 -d</CODE>) decompresses all
 101 specified files.  Files which were not created by <CODE>bzip2</CODE>
 102 will be detected and ignored, and a warning issued.
 103 <CODE>bzip2</CODE> attempts to guess the filename for the decompressed file
 104 from that of the compressed file as follows:
 105
 106 <UL>
 107 <LI><CODE>filename.bz2 </CODE> becomes <CODE>filename</CODE>
 108
 109 <LI><CODE>filename.bz  </CODE> becomes <CODE>filename</CODE>
 110
 111 <LI><CODE>filename.tbz2</CODE> becomes <CODE>filename.tar</CODE>
 112
 113 <LI><CODE>filename.tbz </CODE> becomes <CODE>filename.tar</CODE>
 114
 115 <LI><CODE>anyothername </CODE> becomes <CODE>anyothername.out</CODE>
 116
 117 </UL>
 118
 119 <P>
 120 If the file does not end in one of the recognised endings,
 121 <CODE>.bz2</CODE>, <CODE>.bz</CODE>,
 122 <CODE>.tbz2</CODE> or <CODE>.tbz</CODE>, <CODE>bzip2</CODE> complains that it cannot
 123 guess the name of the original file, and uses the original name
 124 with <CODE>.out</CODE> appended.
 125
 126 </P>
 127 <P>
 128 As with compression, supplying no
 129 filenames causes decompression from standard input to standard output.
 130
 131 </P>
 132 <P>
 133 <CODE>bunzip2</CODE> will correctly decompress a file which is the
 134 concatenation of two or more compressed files.  The result is the
 135 concatenation of the corresponding uncompressed files.  Integrity
 136 testing (<CODE>-t</CODE>) of concatenated compressed files is also supported.
 137
 138 </P>
 139 <P>
 140 You can also compress or decompress files to the standard output by
 141 giving the <CODE>-c</CODE> flag.  Multiple files may be compressed and
 142 decompressed like this.  The resulting outputs are fed sequentially to
 143 stdout.  Compression of multiple files in this manner generates a stream
 144 containing multiple compressed file representations.  Such a stream
 145 can be decompressed correctly only by <CODE>bzip2</CODE> version 0.9.0 or
 146 later.  Earlier versions of <CODE>bzip2</CODE> will stop after decompressing
 147 the first file in the stream.
 148
 149 </P>
 150 <P>
 151 <CODE>bzcat</CODE> (or <CODE>bzip2 -dc</CODE>) decompresses all specified files to
 152 the standard output.
 153
 154 </P>
 155 <P>
 156 <CODE>bzip2</CODE> will read arguments from the environment variables
 157 <CODE>BZIP2</CODE> and <CODE>BZIP</CODE>, in that order, and will process them
 158 before any arguments read from the command line.  This gives a
 159 convenient way to supply default arguments.
 160
 161 </P>
 162 <P>
 163 Compression is always performed, even if the compressed file is slightly
 164 larger than the original.  Files of less than about one hundred bytes
 165 tend to get larger, since the compression mechanism has a constant
 166 overhead in the region of 50 bytes.  Random data (including the output
 167 of most file compressors) is coded at about 8.05 bits per byte, giving
 168 an expansion of around 0.5%.
 169
 170 </P>
 171 <P>
 172 As a self-check for your protection, <CODE>bzip2</CODE> uses 32-bit CRCs to
 173 make sure that the decompressed version of a file is identical to the
 174 original.  This guards against corruption of the compressed data, and
 175 against undetected bugs in <CODE>bzip2</CODE> (hopefully very unlikely).  The
 176 chances of data corruption going undetected is microscopic, about one
 177 chance in four billion for each file processed.  Be aware, though, that
 178 the check occurs upon decompression, so it can only tell you that
 179 something is wrong.  It can't help you recover the original uncompressed
 180 data.  You can use <CODE>bzip2recover</CODE> to try to recover data from
 181 damaged files.
 182
 183 </P>
 184 <P>
 185 Return values: 0 for a normal exit, 1 for environmental problems (file
 186 not found, invalid flags, I/O errors, &#38;c), 2 to indicate a corrupt
 187 compressed file, 3 for an internal consistency error (eg, bug) which
 188 caused <CODE>bzip2</CODE> to panic.
 189
 190 </P>
 191
 192
 193
 194 <H4><A NAME="SEC6" HREF="manual_toc.html#TOC6">OPTIONS</A></H4>
 195 <DL COMPACT>
 196
 197 <DT><CODE>-c  --stdout</CODE>
 198 <DD>
 199 Compress or decompress to standard output.
 200 <DT><CODE>-d  --decompress</CODE>
 201 <DD>
 202 Force decompression.  <CODE>bzip2</CODE>, <CODE>bunzip2</CODE> and <CODE>bzcat</CODE> are
 203 really the same program, and the decision about what actions to take is
 204 done on the basis of which name is used.  This flag overrides that
 205 mechanism, and forces bzip2 to decompress.
 206 <DT><CODE>-z --compress</CODE>
 207 <DD>
 208 The complement to <CODE>-d</CODE>: forces compression, regardless of the
 209 invokation name.
 210 <DT><CODE>-t --test</CODE>
 211 <DD>
 212 Check integrity of the specified file(s), but don't decompress them.
 213 This really performs a trial decompression and throws away the result.
 214 <DT><CODE>-f --force</CODE>
 215 <DD>
 216 Force overwrite of output files.  Normally, <CODE>bzip2</CODE> will not overwrite
 217 existing output files.  Also forces <CODE>bzip2</CODE> to break hard links
 218 to files, which it otherwise wouldn't do.
 219 <DT><CODE>-k --keep</CODE>
 220 <DD>
 221 Keep (don't delete) input files during compression
 222 or decompression.
 223 <DT><CODE>-s --small</CODE>
 224 <DD>
 225 Reduce memory usage, for compression, decompression and testing.  Files
 226 are decompressed and tested using a modified algorithm which only
 227 requires 2.5 bytes per block byte.  This means any file can be
 228 decompressed in 2300k of memory, albeit at about half the normal speed.
 229
 230 During compression, <CODE>-s</CODE> selects a block size of 200k, which limits
 231 memory use to around the same figure, at the expense of your compression
 232 ratio.  In short, if your machine is low on memory (8 megabytes or
 233 less), use -s for everything.  See MEMORY MANAGEMENT below.
 234 <DT><CODE>-q --quiet</CODE>
 235 <DD>
 236 Suppress non-essential warning messages.  Messages pertaining to
 237 I/O errors and other critical events will not be suppressed.
 238 <DT><CODE>-v --verbose</CODE>
 239 <DD>
 240 Verbose mode -- show the compression ratio for each file processed.
 241 Further <CODE>-v</CODE>'s increase the verbosity level, spewing out lots of
 242 information which is primarily of interest for diagnostic purposes.
 243 <DT><CODE>-L --license -V --version</CODE>
 244 <DD>
 245 Display the software version, license terms and conditions.
 246 <DT><CODE>-1 to -9</CODE>
 247 <DD>
 248 Set the block size to 100 k, 200 k ..  900 k when compressing.  Has no
 249 effect when decompressing.  See MEMORY MANAGEMENT below.
 250 <DT><CODE>--</CODE>
 251 <DD>
 252 Treats all subsequent arguments as file names, even if they start
 253 with a dash.  This is so you can handle files with names beginning
 254 with a dash, for example: <CODE>bzip2 -- -myfilename</CODE>.
 255 <DT><CODE>--repetitive-fast</CODE>
 256 <DD>
 257 <DT><CODE>--repetitive-best</CODE>
 258 <DD>
 259 These flags are redundant in versions 0.9.5 and above.  They provided
 260 some coarse control over the behaviour of the sorting algorithm in
 261 earlier versions, which was sometimes useful.  0.9.5 and above have an
 262 improved algorithm which renders these flags irrelevant.
 263 </DL>
 264
 265
 266
 267 <H4><A NAME="SEC7" HREF="manual_toc.html#TOC7">MEMORY MANAGEMENT</A></H4>
 268
 269 <P>
 270 <CODE>bzip2</CODE> compresses large files in blocks.  The block size affects
 271 both the compression ratio achieved, and the amount of memory needed for
 272 compression and decompression.  The flags <CODE>-1</CODE> through <CODE>-9</CODE>
 273 specify the block size to be 100,000 bytes through 900,000 bytes (the
 274 default) respectively.  At decompression time, the block size used for
 275 compression is read from the header of the compressed file, and
 276 <CODE>bunzip2</CODE> then allocates itself just enough memory to decompress
 277 the file.  Since block sizes are stored in compressed files, it follows
 278 that the flags <CODE>-1</CODE> to <CODE>-9</CODE> are irrelevant to and so ignored
 279 during decompression.
 280
 281 </P>
 282 <P>
 283 Compression and decompression requirements, in bytes, can be estimated
 284 as:
 285
 286 <PRE>
 287      Compression:   400k + ( 8 x block size )
 288
 289      Decompression: 100k + ( 4 x block size ), or
 290                     100k + ( 2.5 x block size )
 291 </PRE>
 292
 293 <P>
 294 Larger block sizes give rapidly diminishing marginal returns.  Most of
 295 the compression comes from the first two or three hundred k of block
 296 size, a fact worth bearing in mind when using <CODE>bzip2</CODE> on small machines.
 297 It is also important to appreciate that the decompression memory
 298 requirement is set at compression time by the choice of block size.
 299
 300 </P>
 301 <P>
 302 For files compressed with the default 900k block size, <CODE>bunzip2</CODE>
 303 will require about 3700 kbytes to decompress.  To support decompression
 304 of any file on a 4 megabyte machine, <CODE>bunzip2</CODE> has an option to
 305 decompress using approximately half this amount of memory, about 2300
 306 kbytes.  Decompression speed is also halved, so you should use this
 307 option only where necessary.  The relevant flag is <CODE>-s</CODE>.
 308
 309 </P>
 310 <P>
 311 In general, try and use the largest block size memory constraints allow,
 312 since that maximises the compression achieved.  Compression and
 313 decompression speed are virtually unaffected by block size.
 314
 315 </P>
 316 <P>
 317 Another significant point applies to files which fit in a single block
 318 -- that means most files you'd encounter using a large block size.  The
 319 amount of real memory touched is proportional to the size of the file,
 320 since the file is smaller than a block.  For example, compressing a file
 321 20,000 bytes long with the flag <CODE>-9</CODE> will cause the compressor to
 322 allocate around 7600k of memory, but only touch 400k + 20000 * 8 = 560
 323 kbytes of it.  Similarly, the decompressor will allocate 3700k but only
 324 touch 100k + 20000 * 4 = 180 kbytes.
 325
 326 </P>
 327 <P>
 328 Here is a table which summarises the maximum memory usage for different
 329 block sizes.  Also recorded is the total compressed size for 14 files of
 330 the Calgary Text Compression Corpus totalling 3,141,622 bytes.  This
 331 column gives some feel for how compression varies with block size.
 332 These figures tend to understate the advantage of larger block sizes for
 333 larger files, since the Corpus is dominated by smaller files.
 334
 335 <PRE>
 336           Compress   Decompress   Decompress   Corpus
 337    Flag     usage      usage       -s usage     Size
 338
 339     -1      1200k       500k         350k      914704
 340     -2      2000k       900k         600k      877703
 341     -3      2800k      1300k         850k      860338
 342     -4      3600k      1700k        1100k      846899
 343     -5      4400k      2100k        1350k      845160
 344     -6      5200k      2500k        1600k      838626
 345     -7      6100k      2900k        1850k      834096
 346     -8      6800k      3300k        2100k      828642
 347     -9      7600k      3700k        2350k      828642
 348 </PRE>
 349
 350
 351
 352 <H4><A NAME="SEC8" HREF="manual_toc.html#TOC8">RECOVERING DATA FROM DAMAGED FILES</A></H4>
 353
 354 <P>
 355 <CODE>bzip2</CODE> compresses files in blocks, usually 900kbytes long.  Each
 356 block is handled independently.  If a media or transmission error causes
 357 a multi-block <CODE>.bz2</CODE> file to become damaged, it may be possible to
 358 recover data from the undamaged blocks in the file.
 359
 360 </P>
 361 <P>
 362 The compressed representation of each block is delimited by a 48-bit
 363 pattern, which makes it possible to find the block boundaries with
 364 reasonable certainty.  Each block also carries its own 32-bit CRC, so
 365 damaged blocks can be distinguished from undamaged ones.
 366
 367 </P>
 368 <P>
 369 <CODE>bzip2recover</CODE> is a simple program whose purpose is to search for
 370 blocks in <CODE>.bz2</CODE> files, and write each block out into its own
 371 <CODE>.bz2</CODE> file.  You can then use <CODE>bzip2 -t</CODE> to test the
 372 integrity of the resulting files, and decompress those which are
 373 undamaged.
 374
 375 </P>
 376 <P>
 377 <CODE>bzip2recover</CODE>
 378 takes a single argument, the name of the damaged file,
 379 and writes a number of files <CODE>rec0001file.bz2</CODE>,
 380        <CODE>rec0002file.bz2</CODE>, etc, containing the  extracted  blocks.
 381        The  output  filenames  are  designed  so  that the use of
 382        wildcards in subsequent processing -- for example,
 383 <CODE>bzip2 -dc  rec*file.bz2 &#62; recovered_data</CODE> -- lists the files in
 384        the correct order.
 385
 386 </P>
 387 <P>
 388 <CODE>bzip2recover</CODE> should be of most use dealing with large <CODE>.bz2</CODE>
 389        files,  as  these will contain many blocks.  It is clearly
 390        futile to use it on damaged single-block  files,  since  a
 391        damaged  block  cannot  be recovered.  If you wish to minimise
 392 any potential data loss through media  or  transmission errors,
 393 you might consider compressing with a smaller
 394        block size.
 395
 396 </P>
 397
 398
 399
 400 <H4><A NAME="SEC9" HREF="manual_toc.html#TOC9">PERFORMANCE NOTES</A></H4>
 401
 402 <P>
 403 The sorting phase of compression gathers together similar strings in the
 404 file.  Because of this, files containing very long runs of repeated
 405 symbols, like "aabaabaabaab ..."  (repeated several hundred times) may
 406 compress more slowly than normal.  Versions 0.9.5 and above fare much
 407 better than previous versions in this respect.  The ratio between
 408 worst-case and average-case compression time is in the region of 10:1.
 409 For previous versions, this figure was more like 100:1.  You can use the
 410 <CODE>-vvvv</CODE> option to monitor progress in great detail, if you want.
 411
 412 </P>
 413 <P>
 414 Decompression speed is unaffected by these phenomena.
 415
 416 </P>
 417 <P>
 418 <CODE>bzip2</CODE> usually allocates several megabytes of memory to operate
 419 in, and then charges all over it in a fairly random fashion.  This means
 420 that performance, both for compressing and decompressing, is largely
 421 determined by the speed at which your machine can service cache misses.
 422 Because of this, small changes to the code to reduce the miss rate have
 423 been observed to give disproportionately large performance improvements.
 424 I imagine <CODE>bzip2</CODE> will perform best on machines with very large
 425 caches.
 426
 427 </P>
 428
 429
 430
 431 <H4><A NAME="SEC10" HREF="manual_toc.html#TOC10">CAVEATS</A></H4>
 432
 433 <P>
 434 I/O error messages are not as helpful as they could be.  <CODE>bzip2</CODE>
 435 tries hard to detect I/O errors and exit cleanly, but the details of
 436 what the problem is sometimes seem rather misleading.
 437
 438 </P>
 439 <P>
 440 This manual page pertains to version 1.0 of <CODE>bzip2</CODE>.  Compressed
 441 data created by this version is entirely forwards and backwards
 442 compatible with the previous public releases, versions 0.1pl2, 0.9.0 and
 443 0.9.5, but with the following exception: 0.9.0 and above can correctly
 444 decompress multiple concatenated compressed files.  0.1pl2 cannot do
 445 this; it will stop after decompressing just the first file in the
 446 stream.
 447
 448 </P>
 449 <P>
 450 <CODE>bzip2recover</CODE> uses 32-bit integers to represent bit positions in
 451 compressed files, so it cannot handle compressed files more than 512
 452 megabytes long.  This could easily be fixed.
 453
 454 </P>
 455
 456
 457
 458 <H4><A NAME="SEC11" HREF="manual_toc.html#TOC11">AUTHOR</A></H4>
 459 <P>
 460 Julian Seward, <CODE>jseward@acm.org</CODE>.
 461
 462 </P>
 463 <P>
 464 The ideas embodied in <CODE>bzip2</CODE> are due to (at least) the following
 465 people: Michael Burrows and David Wheeler (for the block sorting
 466 transformation), David Wheeler (again, for the Huffman coder), Peter
 467 Fenwick (for the structured coding model in the original <CODE>bzip</CODE>,
 468 and many refinements), and Alistair Moffat, Radford Neal and Ian Witten
 469 (for the arithmetic coder in the original <CODE>bzip</CODE>).  I am much
 470 indebted for their help, support and advice.  See the manual in the
 471 source distribution for pointers to sources of documentation.  Christian
 472 von Roques encouraged me to look for faster sorting algorithms, so as to
 473 speed up compression.  Bela Lubkin encouraged me to improve the
 474 worst-case compression performance.  Many people sent patches, helped
 475 with portability problems, lent machines, gave advice and were generally
 476 helpful.
 477
 478 </P>
 479 </BLOCKQUOTE>
 480
 481 <P><HR><P>
 482 <p>Go to the <A HREF="manual_1.html">first</A>, <A HREF="manual_1.html">previous</A>, <A HREF="manual_3.html">next</A>, <A HREF="manual_4.html">last</A> section, <A HREF="manual_toc.html">table of contents</A>.
 483 </BODY>
 484 </HTML>