Files

T

zaid a954df0c05 feat: gateway uv migration, conversion engine, and frontend updates

2026-09-09 12:54:37 +05:30

12 KiB

Raw Blame History

Gateway (FastAPI)

Phase 0 task "FastAPI service scaffolding" (Dev 1). Skeleton only — every PDF-touching route returns 501 until the pybind11 engine module in bindings/python/ lands at Gate G0b.

What the gateway eventually does (see docs/phase0.md and engine blueprint §8):

Auth, metadata, storage (S3 / MinIO), async job queue.
Bridges to the C++ engine via pybind11 (bindings/python/).
Hosts the API the React + TypeScript viewer in frontend/ calls.

Current state

gateway/
  pyproject.toml              # pinned FastAPI/uvicorn/pydantic + ruff/pytest
  app/
    __init__.py               # exports __version__
    main.py                   # FastAPI app factory
    config.py                 # pydantic-settings (PDFENGINE_* env vars)
    routers/                  # PEP 420 namespace pkg — no __init__.py needed
      health.py               # GET /health  → 200
      documents.py            # CRUD         → 501 (placeholder)
      render.py               # render/text  → 501 (placeholder)
      edits.py                # apply edits  → 501 (placeholder)
    services/                 # PEP 420 namespace pkg
      engine.py               # lazy pybind11 import wrapper (engine absent in Phase 0)
  tests/
    conftest.py
    test_health.py            # /health works without the engine
    test_placeholders.py      # every PDF route 501s until G0b lands

The gateway depends on no C++ today — the engine service module probes for import pdfengine lazily and reports engine_available: false via /health while the pybind11 module does not exist.

Local development

Requires Python 3.11+ (3.12 works). Build dirs are git-ignored.

Fast path with `uv` (Recommended — 10x faster)

# Windows
cd gateway
uv venv
.venv\Scripts\Activate.ps1
uv pip install -e ".[dev]"

# Linux/macOS
cd gateway
uv venv
source .venv/bin/activate
uv pip install -e ".[dev]"

Standard pip fallback

# Windows
cd gateway
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -e ".[dev]"

# Linux/macOS
cd gateway
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Run the service:

Using the startup script (Windows):

# Default port (8765)
powershell -File scripts/start_gateway.ps1

# Custom port (e.g. 8080)
powershell -File scripts/start_gateway.ps1 -Port 8080

Or run directly via uvicorn (cross-platform):

# Default port (8765)
uvicorn app.main:app --reload

# Custom port (e.g. 8080)
uvicorn app.main:app --reload --port 8080

→ http://127.0.0.1:/health

→ http://127.0.0.1:/docs (OpenAPI UI)

Document conversion API (OSS)

Wave 1+ converters live under app/routers/convert and app/services/convert. No ConvertAPI/Aspose/PyMuPDF — python-docx, openpyxl, reportlab, pypdf, and the C++ engine (when present) for text/layout/render.

Method	Path	Notes
GET	`/v1/convert/info`	Registered converter pairs + fidelity/gaps
GET	`/v1/convert/canconvert/{from}/to/{to}`	Capability probe
POST	`/v1/convert/{from}/to/{to}`	Sync convert (multipart `file`)
POST	`/v1/async/convert/{from}/to/{to}`	Returns `job_id`
GET	`/v1/jobs/{id}`	Job status
GET	`/v1/jobs/{id}/download`	Result bytes

Success responses include X-Fidelity: high|medium|lossy, optional X-Quality-Score (0–1), and X-Warnings.

Honesty (M2 + prod readiness): X-Quality-Score is output-based; live table score uses source-derived grids. Sync convert uses asyncio.to_thread; hard timeouts return 504. Soft cancel is between-pages only. Wave-1 Office stays lossy/medium — shippable, not ConvertAPI-class. Prod smoke: pdf/corpus/convert/real/ + tests/convert/test_prod_smoke.py. Frontend: pdf/frontend Conversion desk.

Accuracy path (Wave-1 climb): PDF→Office/text goes through layout → Intermediate Document Model (idm.v1) → formatters. Tables v3 (shared column schema + rulings + X-gap; under-detect preferred), span/font runs into DOCX/XLSX, headers/footers, multi-column reading order, RapidOCR + live Mistral OCR when configured. Corpus CI under pdf/corpus/convert/ — see docs/accuracy_climb.md. No ConvertAPI/Aspose; no LibreOffice.

Office→PDF (Path A — reportlab max): in-process, concurrent-safe, no LibreOffice. DOCX: BaseDocTemplate frames (incl. multi-column), font alias map, runs/lists/tables/nested tables/images/text boxes/HF. XLSX: number formats, fills, merges, print area/titles, freeze→repeat rows, column-band split, chart→reportlab Drawing (bar/line/pie). Fidelity stays medium (not Word/Excel print clone). SSIM harness is a structural proxy only (no raster SSIM in CI). Timeouts: CONVERT_TIMEOUT_SECONDS; OCR: CONVERT_OCR_PAGE_CAP.

Implemented pairs (honest fidelity): PDF↔DOCX/XLSX (lossy/medium), DOCX→PDF, PDF→TXT/MD/JSON/HTML/PNG/JPEG/TIFF, HTML→PDF, MD→HTML/PDF, images→PDF. pdf→json exports idm.v1.

Rejected in v1: legacy .doc, .docm, encrypted PDFs, unknown pairs (422).

Windows / multi-user note: DOCX/XLSX→PDF uses reportlab in-process so many concurrent converts stay light. Fidelity is medium (not Word/Excel print-identical).

pytest tests/convert -v

Lint, format, and test (the exact commands CI runs):

ruff check .
ruff format --check .
pytest

Configuration

Environment variables are read by app/config.py with the prefix PDFENGINE_:

Var	Default	Meaning
`PDFENGINE_ENVIRONMENT`	`dev`	`dev` / `staging` / `prod` — echoed in `/health`
`PDFENGINE_ENGINE_AVAILABLE`	`false`	Forces the engine-availability flag for testing
`PDFENGINE_CORS_ALLOWED_ORIGINS`	`http://localhost:5173,http://127.0.0.1:5173`	Comma-separated browser origins allowed to call the gateway.
`PDFENGINE_CORS_ALLOW_CREDENTIALS`	`false`	Enables cookies/credentials only for explicitly listed origins; wildcard is rejected.

Additionally, the gateway startup scripts and frontend configuration support:

Var	Default	Meaning
`PORT`	`8765`	Gateway listening port (used by `start_gateway.ps1`).
`VITE_GATEWAY_URL`	`http://127.0.0.1:8765`	URL of the gateway API (used by the frontend).
`CONVERT_TIMEOUT_SECONDS`	`120`	Hard convert timeout; exceeded runs return 504.
`CONVERT_OCR_PAGE_CAP`	`200`	Max pages OCR'd per document; capped runs report skipped pages in metadata/warnings.
`CONVERT_OCR_IMAGE_MAX_BYTES`	`15000000`	Per-page OCR raster budget; oversized pages are downsampled/compressed, never dropped.
`CONVERT_FIGURE_MAX_BYTES`	`4000000`	In-memory figure budget; large figures are compacted while retained.
`CONVERT_OCR_PAGE_TIMEOUT`	`30`	Per-page OCR timeout (seconds).
`CONVERT_LAYOUT_ML`	`0`	`1` enables optional Apache ONNX layout beside heuristics (fail-open).
`CONVERT_LAYOUT_ML_WEIGHTS`	`models/layout/v1/layout.onnx`	ONNX path (relative to gateway cwd).
`CONVERT_LAYOUT_ML_DEVICE`	`cpu`	`cpu` or `cuda`.
`CONVERT_LAYOUT_ML_DPI`	`150`	Page raster DPI for layout ML.
`CONVERT_LAYOUT_ML_PAGE_TIMEOUT`	`15`	Per-page ML soft timeout (seconds).
`CONVERT_LAYOUT_ML_MIN_SCORE`	`0.5`	Drop weak detection boxes.
`CONVERT_LAYOUT_ML_TABLE_STRUCTURE`	`0`	`1` enables optional table-structure ONNX (column separators only, fail-open).
`CONVERT_LAYOUT_ML_TABLE_STRUCTURE_WEIGHTS`	`models/layout/table/structure.onnx`	Structure ONNX path; missing ⇒ heuristics, no error.
`CONVERT_LAYOUT_ML_TABLE_STRUCTURE_MIN_SCORE`	`0.5`	Drop weak column/row boxes.
`CONVERT_LAYOUT_ML_TABLE_STRUCTURE_TIMEOUT`	`10`	Per-region structure timeout (seconds).
`CONVERT_OCR_ARABIC`	auto	Dual-pass Arabic rec; auto = on when `rec.onnx` exists.
`CONVERT_OCR_ARABIC_ADAPTIVE`	0 (startup script: 1)	When enabled, skip the Arabic pass for high-confidence Latin pages and retry it for empty/garbled/Arabic-looking EN output. Set 0/always for forced dual-pass behavior.
`CONVERT_OCR_ARABIC_WEIGHTS`	`models/ocr/ar/v5/rec.onnx`	Arabic PP-OCRv5 rec ONNX path.
`CONVERT_OCR_ARABIC_DICT`	`models/ocr/ar/v5/arabic_dict.txt`	Arabic charset dict.
`CONVERT_OCR_REC_MODEL`	(unset)	Optional Latin/CJK rec ONNX. Do not reuse this path for det/cls.
`CONVERT_OCR_DET_MODEL`	(unset)	Optional text det ONNX.
`CONVERT_OCR_CLS_MODEL`	(unset)	Optional angle cls ONNX.

Convert plugins default to 50 MB max upload (max_bytes). The pdf/frontend Conversion desk uses async jobs for files ≥ 8 MB.

Start gateway (recommended on Windows): use the helper so port 8080 is freed and the project venv is used (avoids WinError 10048 from an orphan python3.12 still listening):

cd pdf\gateway
.\start_convert_gateway.ps1

If you start manually and see only one usage of each socket address / errno 10048:

# Find and stop the holder
Get-NetTCPConnection -LocalPort 8080 -State Listen | Select OwningProcess
Stop-Process -Id <PID> -Force

Always start with .\.venv\Scripts\python.exe from pdf\gateway — not Windows Store python.exe.

Layout ML / public eval: see docs/ml_layout_free_stack.md. Generate smoke fixtures:

cd pdf
.\gateway\.venv\Scripts\python.exe scripts\convert\generate_public_smoke_fixtures.py
.\gateway\.venv\Scripts\python.exe -m pytest gateway\tests\convert -q -k "not scanned_page_count"

Enable layout ML later (PowerShell — quote URLs; no <placeholders>):

cd pdf
.\gateway\.venv\Scripts\python.exe scripts\convert\fetch_layout_onnx.py
# when you have a real Apache ONNX URL:
.\gateway\.venv\Scripts\python.exe scripts\convert\fetch_layout_onnx.py --url "https://YOUR-HOST/layout.onnx" --sha256 "YOUR_SHA256_HEX"
cd gateway
$env:CONVERT_LAYOUT_ML="1"
$env:CONVERT_LAYOUT_ML_WEIGHTS="models/layout/v1/layout.onnx"

A .env file in gateway/ is auto-loaded if present (it is git-ignored via the repo-wide .venv/ and Python rules — add .env to your local ignores if you keep secrets in it).

CI

The gateway job in .github/workflows/ci.yml runs on ubuntu-latest only — the gateway is pure Python with no OS-specific surface, so a cross-platform matrix would burn runner minutes catching nothing. The engine build matrix still covers Linux + macOS + Windows; once the pybind11 wheel is built per-platform there, the gateway job will install it and run integration tests against the real engine.

What is not here yet (and why)

Real document/render/edit routes — blocked on Gate G0b (PdfDocument / PdfPage interface contracts) and the pybind11 module. Stubbing them earlier would just lock us into a bad API.
Auth, storage, job queue — Phase 1 / 2 work. Leaving these out of Phase 0 keeps the surface area small enough to keep CI green while the engine is still being scaffolded.
Containerization (Dockerfile, compose) — added when there is something non-trivial to package. A FastAPI app with a /health route does not need an image yet.

12 KiB Raw Blame History Unescape Escape