Skip to main content

Data files

Parquet​

pip install 'filerepack[parquet]' # or filerepack[data]
filerepack repack data.parquet
filerepack repack data.parquet --ultra # zstd level 22

Recompression uses PyArrow 19+ with zstd level 19 (22 with --ultra). Before publication, the candidate must retain physical and logical schema, nullability, nested field names, field identifiers, file/schema/field metadata, ordered values and floating-point bit patterns, including NaN and signed zero. Existing ARROW:schema bytes are retained; a missing Arrow schema may be added. Row-group boundaries, compression pages and statistics may be rebuilt.

Writing and verification use batches of at most 65,536 rows. This is a row limit, not a byte memory budget: large source row groups, wide rows, nested values and backend buffers can require more memory. External column chunks, sorting declarations and schemas whose physical representation cannot be retained (for example legacy INT96 timestamps or integer-backed decimals) are skipped with a reason. Missing PyArrow or failed verification also leaves the input unchanged. Install the updated parquet or data extra even if DuckDB is already installed.

Arrow/Feather, ORC and Avro also have qualified schema, metadata and value preservation checks. Arrow IPC retains file/stream framing. Unsupported variants remain unchanged; see the format profiles and quality evidence for their qualification limits.

SQLite and spatial containers​

SQLite compaction verifies schema, metadata, rows and rowids. .db requires the SQLite format 3 header. In-place processing needs --sqlite-offline, closed database users and no WAL, shared-memory or journal sidecars:

filerepack repack notes.sqlite --sqlite-offline
filerepack repack map.gpkg --sqlite-offline
filerepack repack tiles.mbtiles --sqlite-offline

For a distinct output, filerepack takes a consistent snapshot including committed WAL pages. It can publish that snapshot even when VACUUM provides no savings:

filerepack repack notes.sqlite --output-dir ./snapshots

A live WAL source cannot use a single-file --backup. The offline assertion cannot detect every idle external connection. Close users before an in-place run. .vscdb, .sqlitedb and .qgd use separate offline-only profiles and refuse sidecars; close their application before processing.

Columnar and scientific​

pip install 'filerepack[data]'
filerepack repack table.orc
filerepack repack events.avro
filerepack repack frame.feather

HDF5 needs h5repack; NetCDF4 uses its native Python writer. Both need the scientific extra:

pip install 'filerepack[scientific]'
filerepack repack model.h5
filerepack repack grid.nc

Scientific and project files​

pip install 'filerepack[fits,blend]'
filerepack repack observation.fits
filerepack repack volume.nrrd
filerepack repack scene.blend
filerepack repack project.qgz
filerepack repack auxiliary.qgd
filerepack repack events.ndjson
filerepack repack analysis.ipynb

These keep their extensions and verify their supported decoded content. FITS uses lossless tiled compression and can move a primary image to HDU 1 behind an empty primary HDU; use a reader with tiled-compression support. NRRD supports attached raw/gzip/bzip2 arrays. Blender verifies the full decoded stream without launching Blender. QGZ retains unrelated assets and validates the QGS project and QGD auxiliary database. JSON Lines keeps record boundaries and number/string spellings; notebook cells and outputs are retained.

Native input/decoded-payload limits and unsupported variants are described in Native format policies and limits.

Fonts​

pip install 'filerepack[fonts]'
filerepack repack icon.woff2

WOFF/WOFF2 require fontTools/Brotli for independent exact-table and metadata comparison. External woff2_compress / woff2_decompress can provide a candidate, but cannot replace the required fonts extra. DSIG-bearing fonts are skipped.

Compressed streams​

Web archives (WARC)​

filerepack repack archive.warc.gz
filerepack repack archive.warc
filerepack repack archive.warc.gz --dryrun
filerepack bulk ./captures --include-ext warc
filerepack repack archive.warc.gz --output-dir ./recompressed

The dependency-free writer supports WARC 1.0/1.1 with CRLF framing, valid header field names and unique, nonempty WARC-Type, WARC-Record-ID, WARC-Date and Content-Length fields. Each record becomes a separate level-9 gzip member, keeping random access. Whole-file gzip and records spread across gzip members are accepted as inputs and rewritten into this layout when savings policy permits. Use --allow-grow if a record-based layout is required even when it is larger. The full decoded bytes are verified: WARC and HTTP headers, encoded payloads, digests, unknown fields, folded headers, revisit references and record order stay exact. Gzip wrapper metadata can change. ARC conversion, payload optimization, header repair, missing-digest generation and Zstandard are outside this profile.

Standalone plain .warc converts to .warc.gz; the original is removed only after successful publication. Use --output-dir to retain it, or --no-convert-container to disable conversion. Plain WARC members inside another container are skipped so member names and types remain intact; nested .warc.gz members can be recompressed. Ordinary size and minimum-savings policies apply, so an existing optimized archive may remain unchanged.

Compressed offsets and lengths change after recompression. In-place requests with adjacent .cdx, .cdxj, .cdx.gz or .cdxj.gz files are skipped. A distinct unindexed output is allowed while retaining the source and its indexes. Indexes in other directories or external catalogues cannot be detected and must be regenerated for the result. Filerepack does not rewrite index files. Distinct WARC outputs are created only for accepted candidates; --allow-grow also permits writing a verified output when the input is already optimized.

Reads use bounded chunks and headers are limited to 1 MiB total and 64 KiB per line. The existing --format-max-decoded-bytes, --format-max-memory-bytes, --format-max-scratch-bytes, --format-max-nodes and --format-timeout limits apply cumulatively to parsing, encoding and verification. A successful rewrite reads the decoded bytes four times, so the default 512 MiB decoded-byte budget allows roughly 128 MiB of capture bytes. Raise the budget for larger archives:

filerepack repack large.warc.gz --format-max-decoded-bytes 8589934592 --format-timeout 600

Truncated records, invalid lengths, malformed framing, damaged gzip trailers or exhausted budgets leave the source unchanged.

Other compressed streams​

gzip (.gz or .gzip), xz, bz2, zst, brotli, lz4, lzip, lzma, lzo, and Unix .Z are recompressed with the matching CLI (pigz is preferred for gzip). If the payload is a tar, the tarball is walked first — see Archives.

Recompressed gzip keeps the leading member's file name, comment, modification time and OS byte, so gunzip -N restores the original name. Gzip input whose leading member has an FEXTRA field, such as BGZF (tabix/CSI-indexed .vcf.gz, .bed.gz) or dictzip, is left unchanged and reported as unsupported: a single recompressed member would invalidate compressed-offset indexes and random access.

filerepack repack dump.json.gz
filerepack repack dump.json.gzip

See Formats and Installation extras.

See preserving scientific formats for RDS/RData, HDF5/NetCDF, TIFF, checkpoints and SAV. Complete offline Zarr v2 stores use repack-store STORE --output-dir OUTPUT_PARENT; MAT and ZSAV remain explicitly experimental.