Safety
filerepack is designed to leave the source file untouched until a candidate is verified and actually smaller.
Write path
Encoders write in scratch space. A verified candidate is copied and checked into
a private file beside the destination, then published with os.replace for an
existing destination or a no-clobber link for a new one. Direct pack_* helpers
use the same publication boundary. Scratch can be on another filesystem; an
EXDEV failure never falls back to copying over the original.
A missing 7zz/7z leaves the source archive untouched. Packers must not
unlink user files before a successful rewrite.
Candidates, extraction directories and tool-generated auxiliary outputs belong to the current operation. Cleanup runs on rejection, dry-run and encoder or validator exceptions. A pre-existing derived sidecar is refused without being deleted. This cleanup policy does not establish stronger format preservation: the remaining data/media preservation work is tracked separately.
Candidate verification
Every writer must name a structural validator. Unknown keys, missing parsers or
decoders, warnings from the PDF structural checker, failed subprocesses and
truncated candidates refuse publication and log a reason. A recognizable header
is insufficient. Raster validation needs filerepack[validation] (Pillow);
media validation needs both ffprobe container inspection and complete ffmpeg
video/audio decoding. ZIP reads every member through its CRC check, tar checks
complete blocks and payloads, and other archives use their integrity tester.
Data formats need their respective Python parsers to read the candidate.
Separate source comparisons are enforced for decoded compressed-stream bytes, JPEG/PNG/GIF/WebP/TIFF decoded frames, Arrow/Feather/ORC schemas, metadata and ordered values, and lossless PDF objects and streams. Existing archive-manifest, Parquet, DICOM and native-format checks also remain in use. This does not establish complete preservation for every other data/image/audio/video writer; their remaining metadata, stream and frame contracts are still tracked separately. Unsupported raster decoders (including HEIC/JXL/EXR in a standard Pillow build), unknown ICNS resources and unsupported high-depth raster modes fail closed.
Some validators cap parser input at 256 MiB, decoded output at 512 MiB and raster frame count at 4096. Verifier processes have a 120-second timeout and capped captured output; PDF JSON comparison is capped at 16 MiB. Exceeding these bounds refuses the candidate. These are local verifier bounds, not a shared operation budget or a bound on every parser's internal memory allocation.
PDF protection and candidate policy
Protection inspection runs before image walking, qpdf and Ghostscript, including PDF-based Illustrator files. Encryption, signature fields, signature/protection markers anywhere in the object table, and uncertain inspection leave the source bytes untouched even when lossy options are requested. This inspection detects protection markers; it does not authenticate a cryptographic signature.
An available qpdf performs structural checks and inspects all PDF objects. With qpdf absent, pikepdf can inspect protection; lossless publication additionally requires qpdf 11+ JSON object/stream comparison. Missing verification refuses rewriting. The source document graph, metadata and decoded streams must match; object numbers, xref layout, compression and volatile trailer IDs are excluded. Unsupported stream filters or excessive inspection output refuse acceptance. JPEG/JPX image replacements also compare decoded sample size, mode and pixels.
Compression evaluates the original, a verified image-walk candidate, and qpdf
results from the original and walked files. It selects the smallest valid result;
an unsuccessful or larger qpdf candidate does not discard a valid walked one.
The original wins ties and larger compression results, including --allow-grow.
--pdf-linearize adds an explicit linearization constraint. Use --allow-grow
if a required linearized result is larger; ordinary savings thresholds still
apply. The flag also applies to a Ghostscript result, with the subsequent qpdf
stage required to preserve that explicitly lossy result. Linearization is never
forced by ordinary PDF compression.
Discard if not smaller
Output that is not smaller than the original is discarded unless --allow-grow.
--min-savings PCT keeps the result only if savings are at least that percent.
Backups and output directory
filerepack repack contract.docx --backup
filerepack repack contract.docx --backup --backup-dir ./backups
filerepack repack contract.docx --output-dir ./out
filerepack repack contract.docx --output-dir ./out --overwrite
--output-dir preserves the source. For ordinary supported inputs with no
accepted improvement, the destination receives a copy checked for byte identity
with the source. Scientific, NIB, CAR and WARC profiles create distinct outputs
only for accepted candidates, and refused processing creates no fallback copy.
SQLite distinct outputs use a consistent snapshot including committed WAL pages;
that snapshot may be published without compaction savings. Copying an unchanged
source does not establish that an already malformed source is a valid format.
Different existing output/conversion targets are refused unless --overwrite
is explicit. This includes the .mp4 target of video conversion and .7z when
RAR writing is unavailable. A required backup is never overwritten, and failure
to create it stops processing. Existing .bak files cause a conflict; a custom
backup directory uses fresh names for repeated backups. --backup-dir alone
does not enable backups.
Options, path aliases and conflicts are checked before copying or encoding. Concurrent filerepack operations reserve source/output/backup identities; another active owner causes a clear conflict. New output files are published without replacing a file that appears after preflight. Publication uses a candidate in the destination filesystem; unsupported publication operations fail while retaining the source.
Source bytes, file identity, link count, mode, mtime and ctime are checked again
after staging and immediately before publication. An existing output must also
remain unchanged even when --overwrite is authorized. A concurrent writer
causes a conflict. These checks cannot remove the short race between the final
check and rename/unlink, or coordinate applications that ignore filerepack
reservations. Conversion cleanup rechecks the source before deleting it. If the
source changes after the converted output is published, that output remains
and a cleanup conflict retains the new source bytes.
Filesystem attributes and links
Published files and backups retain the source's permission mode and mtime.
Extended attributes exposed by the Linux or macOS native APIs are copied and
verified, including binary values and macOS resource forks. Metadata application
or verification failure refuses publication and logs the reason. The default
policy covers the supported attributes; platforms without an xattr API do not
provide an xattr preservation guarantee. keep_meta controls incidental format
metadata separately; lossless JPEG/PNG already retain required orientation/color
information.
On macOS, a new staging file can receive a protected com.apple.provenance
attribute from the system. When the source has no such attribute, publication
retains the exact value observed immediately after staging-file creation. All
source attributes still have to match byte for byte. A provenance attribute
added later, a changed creation-time value, or a different source provenance
value does not receive an exception to verification.
Ownership, native ACLs, birth time, filesystem flags and Windows alternate data streams are outside this policy. Source reads may update atime. In-place file symlinks and sources with multiple hard links are refused, including aliases that resolve to an in-place request. A distinct output can read a linked source without replacing it. Directory symlink aliases remain supported.
Publication provides atomic visibility when the destination filesystem supports
replace/link. It does not guarantee persistence after a power loss: no file or
directory fsync contract is implemented. The library's durability="atomic"
is the only supported guarantee; stronger requests raise ValueError before
creating scratch, output or backup artifacts. Network and unusual filesystems
must provide the required publication primitives or the operation fails.
--dryrun measures savings in scratch space, leaving source, output and backup
files unchanged and creating no output or backup directories. Scratch space
also holds a private source copy during normal processing, so account for that
additional disk use with large inputs.
Scientific, NIB and CAR dry-runs inspect without encoding or measuring savings.
Archive extract limits
--max-extract-size skips archive extract if uncompressed size exceeds the
limit (0 disables). The default is 8 GB, and also 100× the archive size, so
a tiny zip bomb is not fully expanded.
Files that stay untouched
These are never extracted and rewritten:
- Signed installers:
deb,rpm,pkg,dmg - CAB (no qualified writer), ISO and AR /
a/lib - OpenType
.otffonts (ZIP ODF templates with that extension are packed) - Encrypted or digitally signed PDFs skip every rewriting path, including qpdf, pikepdf and Ghostscript
- DICOM instances that are signed, non-image, or use unsupported compressed transfer syntaxes; RLE Lossless sources can use the qualified GDCM path
Raw and bzip2-wrapped CPIO have a native preserving writer. Deep walking only optimizes safe regular single-link members; linked/special entries remain exact. See CPIO usage.