pdf/tests/regression/README.md

Rendering-regression harness

Renders the committed corpus/ with the PDFium engine and compares each page to a frozen baseline using SSIM. A failure means this build renders differently from the blessed build — not that it disagrees with Acrobat.

Prerequisites

• The engine is built and copied to gateway/ (run scripts/build_cpp.ps1 -Preset win-local-pdfium).
• The gateway venv has numpy and Pillow (gateway/.venv).

Usage

# Check the current build against the frozen baseline (CI mode; non-zero exit on regression)
gateway/.venv/Scripts/python.exe tests/regression/run.py

# Re-freeze the baseline after an *intentional* rendering change (review the diff!)
gateway/.venv/Scripts/python.exe tests/regression/run.py --update

Options: --dpi 72, --max-pages 2, --threshold 0.990, --corpus, --baseline.

Large corpus (500+ real-world PDFs)

The corpus is pinned and shared via a manifest so every developer and CI run gets a byte-identical set. The 600 PDFs themselves are gitignored (≈68 MB); only the manifest (corpus-manifest.json, name + SHA-256 per file, ≈96 KB) is committed. Reproduce the exact corpus with:

python scripts/fetch_corpus.py --manifest tests/regression/corpus-manifest.json

This downloads each pinned file and verifies its SHA-256 — a mismatch is reported and skipped, so the corpus can never silently drift between machines. (scripts/fetch_corpus.py with no --manifest just grabs "latest" from pdf.js, which is not reproducible — use it only to refresh/regenerate the manifest.)

Two ways to use the corpus for regression:

# 1. Render-stability sweep — no baseline; passes as long as the engine never
#    crashes on any real-world PDF. The CI-friendly form of a 500+ regression.
python scripts/fetch_corpus.py
gateway/.venv/Scripts/python.exe tests/regression/run.py --sweep --corpus corpus/fuzz --max-pages 3

# 2. SSIM baseline over the large corpus (baseline is gitignored — it is derived
#    from gitignored inputs and is ~tens of MB, so it lives locally / as a CI
#    artifact, not in git).
... run.py --update --corpus corpus/fuzz --baseline tests/regression/baseline-large
... run.py          --corpus corpus/fuzz --baseline tests/regression/baseline-large

The committed baseline/ stays the small curated set (reviewable in PRs); baseline-large/ is the throwaway large-corpus reference. In CI the libFuzzer binary doubles as a sweep via pdfengine_fuzz -runs=0 corpus/fuzz/ (see .github/workflows/fuzz.yml).

How it works

• ssim.py — dependency-light SSIM (numpy only) via an integral-image box filter. Discriminates correctly: identical→1.0, shifted→~0.81, noise→~0.00, shape mismatch→0.0.
• run.py — enumerates corpus/**/*.pdf (excluding the gitignored corpus/fuzz/), renders to grayscale, and scores against baseline/*.png.
• baseline/ — committed PNGs. Because the baseline is the engine's own prior output, it is tied to this PDFium version; re-freeze deliberately when PDFium is upgraded.

Encrypted / intentionally-malformed fixtures are skipped (and reported), not failed.