Skip to main content

Reports and resume

filerepack repack capture.warc.gz --report run.jsonl
filerepack bulk ./input --jobs 2 --report run.json --report-paths relative
filerepack bulk ./input --manifest run.manifest
filerepack bulk ./input --manifest run.manifest --resume

Reports are local version 1 documents independent of stdout --json and --csv. --report-format json|jsonl defaults to JSONL for .jsonl, JSON otherwise. The parent directory must exist; existing report files and aliases to inputs, outputs, backups or logs are refused. Report files and their spools are excluded from input discovery.

JSONL contains a run header, terminal item and member events, and a completion summary. Item details include bounded tool/worker evidence and structural or preservation checks, including rejected checks. Known executed tools receive read-only version probes after processing, cached by executable generation; an unavailable or unidentified version is explicitly null. Isolated workers record loaded library versions. Arguments and input paths are not collected as command evidence. More than 128 distinct evidence events is marked truncated. Each event is flushed separately. A crash may truncate the final line; complete earlier lines remain recoverable. JSON spools the same events with bounded memory and publishes {"schema_version":1,"events":[...]} atomically. A failed finalization retains a .partial-UUID.jsonl spool. complete describes whether scheduling finished; item statuses still identify individual failures.

Statuses are replaced, unchanged, skipped, unsupported, failed, predicted, and cancelled. Events distinguish actual publication from staged archive-member edits and dry-run predictions. A rejected outer candidate does not claim that its staged member savings were published.

Reports disclose absolute local paths by default. --report-paths relative uses the input directory (or single file's parent). redacted replaces path and member identities with run-specific opaque identifiers, removes detailed diagnostics and path-bearing settings, and retains reason codes. Relative paths can disclose locations outside the run root. A redacted report is not a resume manifest. Report write failures cause a non-success exit; already published files retain their actual outcomes.

--manifest maintains a separate private checkpoint. Its version 1 format is JSONL: a header, completion entries and a footer containing entry count and a SHA-256 checksum. A coordinator lock prevents simultaneous writers. Checkpoints are atomically replaced on the manifest filesystem. Malformed, incomplete or incompatible checkpoints are refused before dispatch.

--resume verifies streaming SHA-256 identities of current sources and outputs, effective settings, destinations, package/policy versions, toolchain identities, Python version and optional-library versions/absence. Installing or updating a reader/encoder extra invalidates a previous completion. In-place entries compare against post-publication source bytes. Distinct outputs require both source and output to match. Timestamps and sizes alone never authorize reuse. Invalidated entries return to normal destination safety rules; a stale output is not overwritten without --overwrite.

Only completed replaced and unchanged outcomes can be reused. Reports mark reused completion with reused: true, published: false, and zero new savings. Failed, cancelled, predicted, unsupported and skipped items are processed again. Dry-run can read a checkpoint and never persists new completion. A publication may finish before a checkpoint failure: the previous complete checkpoint stays valid, and a later invocation safely re-plans missing completion entries.

Converted in-place inputs can be reused through their discovered output filename only when the historical source is still absent and the output digest and execution fingerprint match. A recreated source invalidates that completion. Current discovery and size/extension filters are applied before reuse.

After a hard process crash, a manifest lock can remain. Confirm that its owning process has ended before removing that lock; preserve the last complete manifest. Do not promote a partial checkpoint or infer completion from a published output.

If the result spool cannot persist another row, bulk stops submitting work and drains the bounded outstanding jobs into an emergency in-memory tail. Prior complete spool rows and those terminal outcomes remain available; the invocation exits unsuccessfully. Disk-backed checkpoint updates currently rewrite the complete checkpoint, so very large runs can have substantial checkpoint I/O.

Single-file exit codes are 0 for accepted/unchanged/intentional-skipped work, 1 for failed/unsupported work or application validation failures, and 130 for interruption. Bulk exits 1 for fatal, fail-fast or persistence failure; a batch that continues after item failures exits 2. Argument-parser usage errors also exit 2. Dry-run predicts acceptance and records published: false.