LIBARCHIVE_INTERNALS(3) | Library Functions Manual | LIBARCHIVE_INTERNALS(3) |
libarchive_internals
—
libarchive
library provides a flexible interface for
reading and writing streaming archive files such as tar and cpio. Internally,
it follows a modular layered design that should make it easy to add new
archive and compression formats.
The read and write APIs each have four layers: a public API layer, a format layer that understands the archive file format, a compression layer, and an I/O layer. The I/O layer is completely exposed to clients who can replace it entirely with their own functions.
In order to provide as much consistency as possible for clients, some public functions are virtualized. Eventually, it should be possible for clients to open an archive or disk writer, and then use a single set of code to select and write entries, regardless of the target.
archive
object to read entries and
bodies from an archive stream. Internally, the archive
object is cast to an archive_read
object, which holds
all read-specific data. The API has four layers: The lowest layer is the I/O
layer. This layer can be overridden by clients, but most clients use the
packaged I/O callbacks provided, for example, by
archive_read_open_memory(3),
and
archive_read_open_fd(3).
The compression layer calls the I/O layer to read bytes and decompresses them
for the format layer. The format layer unpacks a stream of uncompressed bytes
and creates archive_entry
objects from the incoming
data. The API layer tracks overall state (for example, it prevents clients
from reading data before reading a header) and invokes the format and
compression layer operations through registered function pointers. In
particular, the API layer drives the format-detection process: When opening
the archive, it reads an initial block of data and offers it to each
registered compression handler. The one with the highest bid is initialized
with the first block. Similarly, the format handlers are polled to see which
handler is the best for each archive. (Prior to 2.4.0, the format bidders were
invoked for each entry, but this design hindered error recovery.)
The client read callback is expected to provide a block of data on each call. A zero-length return does indicate end of file, but otherwise blocks may be as small as one byte or as large as the entire file. In particular, blocks may be of different sizes.
The client skip callback returns the number of bytes actually skipped, which may be much smaller than the skip requested. The only requirement is that the skip not be larger. In particular, clients are allowed to return zero for any skip that they don't want to handle. The skip callback must never be invoked with a negative value.
Keep in mind that not all clients are reading from disk: clients reading from networks may provide different-sized blocks on every request and cannot skip at all; advanced clients may use mmap(2) to read the entire file into memory at once and return the entire file to libarchive as a single block; other clients may begin asynchronous I/O operations for the next block on each request.
A subsequent call to the consume
()
function advances the read pointer. Note that data returned from a
read_ahead
() call is guaranteed to remain in place
until the next call to read_ahead
(). Intervening
calls to consume
() should not cause the data to
move.
Skip requests must always be handled exactly. Decompression handlers that cannot seek forward should not register a skip handler; the API layer fills in a generic skip handler that reads and discards data.
A decompression handler has a specific lifecycle:
__archive_read_register_compression
() function to
provide bid and initialization functions. This function returns
NULL
on error or else a pointer to a
struct decompressor_t
. This structure contains a
void * config slot that can be used for storing any
customization information.archive_read
object. The bid function is otherwise
stateless. In particular, it must not perform any I/O operations.
The value returned by the bid function indicates its suitability for handling this data stream. A bid of zero will ensure that this decompressor is never invoked. Return zero if magic number checks fail. Otherwise, your initial implementation should return the number of bits actually checked. For example, if you verify two full bytes and three bits of another byte, bid 19. Note that the initial block may be very short; be careful to only inspect the data you are given. (The current decompressors require two bytes for correct bidding.)
read_ahead
()
decompression method but not calling the consume
()
method. This allows each bidder to look ahead in the input stream. Bidders
should not look further ahead than necessary, as long look aheads put
pressure on the decompression layer to buffer lots of data. Most formats
only require a few hundred bytes of look ahead; look aheads of a few
kilobytes are reasonable. (The ISO9660 reader sometimes looks ahead by
48k, which should be considered an upper limit.)data_skip
() function.archive_read
, archive_write
,
and archive_write_disk
objects all contain an initial
archive
object which provides common support for a set
of standard services. (Recall that ANSI/ISO C90 guarantees that you can cast
freely between a pointer to a structure and a pointer to the first element of
that structure.) The archive
object has a magic value
that indicates which API this object is associated with, slots for storing
error information, and function pointers for virtualized API functions.
For example, libarchive's ISO9660 support operates very differently from most ISO9660 readers. The libarchive support utilizes a work-queue design that keeps a list of known entries sorted by their location in the input. Whenever libarchive's ISO9660 implementation is asked for the next header, checks this list to find the next item on the disk. Directories are parsed when they are encountered and new items are added to the list. This design relies heavily on the ISO9660 image being optimized so that directories always occur earlier on the disk than the files they describe.
Depending on the specific format, such approaches may not be possible. The ZIP format specification, for example, allows archivers to store key information only at the end of the file. In theory, it is possible to create ZIP archives that cannot be read without seeking. Fortunately, such archives are very rare, and libarchive can read most ZIP archives, though it cannot always extract as much information as a dedicated ZIP program.
libarchive
library first appeared in
FreeBSD 5.3.
libarchive
library was written by
Tim Kientzle ⟨kientzle@acm.org⟩.
January 26, 2011 | NetBSD 9.2 |