Roaring bitmap storage format

Bitmaps are persisted to disk using a file format very similar to the Roaring Bitmap format spec. Pilosa’s format uses 64-bit IDs, so it is not binary-compatible with the spec. Some parts of the format are simpler, and an additional section is included. Specific differences include:

roaring file format diagram

All values are little-endian. The first two bytes of the cookie is 12348, to reflect incompatibility with the spec, which uses 12346 or 12347. Container types are NOT inferred from their cardinality as in the spec. Instead, the container type is read directly from the descriptive header.