Architecture

What this service is

A document conversion engine behind a FastAPI gateway. You send it a file and the format you want back; it decides everything else — whether to OCR, whether to reflow the text or hold its position, how to treat running headers, whether to look for tables — and returns the converted bytes.

It is deterministic. There is no language model anywhere in the pipeline. The only nondeterministic component is the OCR recogniser, and it is reached through exactly one function (app.services.ocr.recognize_image), which makes the rest of the system testable without stubbing anything.

Everything it depends on is permissively licensed free software, and every model it runs is freely redistributable. That is a hard constraint, not a preference: see limits.md.

The shape of a conversion

HTTP request
  │
  ├─ admission control ......... app/services/convert/admission.py
  │     refuses at capacity with 503 + Retry-After rather than accepting
  │     work the process cannot do
  │
  ├─ validation ................ app/services/convert/validate.py
  │     size, page count, magic bytes vs declared type, encryption,
  │     zip-bomb depth/ratio. Raises a typed ValidationError.
  │
  ├─ routing ................... app/services/convert/auto_policy.py
  │     reads the document and picks the reconstruction path
  │
  ├─ conversion ................ app/services/convert/pipeline.py
  │     enters four scopes, all ContextVar-based and all per-conversion:
  │       cancellation (cancel token + wall-clock deadline)
  │       document cache (readers, page text, geometry, rasters)
  │       policy decision
  │       options
  │     then calls exactly one plugin
  │
  ├─ output validation ......... app/services/convert/validation.py
  │     PDF must re-open; OOXML must contain its mandatory part
  │
  └─ HTTP response with X-Fidelity, X-Warnings, X-Request-Id

The document model in the middle

Every PDF-sourced conversion goes through one intermediate representation, the IDM (app/services/convert/idm/model.py): a Document of Pages of Blocks of TextSpans, with geometry in PDF points and the origin at the top left of the page. Formatters read the IDM and never the PDF.

This is what makes eleven output formats affordable. Adding one means writing a formatter against the IDM, not another PDF parser.

PDF ─► layout/pipeline.py ─► IDM ─┬─► formatters/docx_formatter.py  ─► .docx
                                  ├─► formatters/text_formatters.py ─► .md/.html/.txt
                                  ├─► formatters/xlsx_formatter.py  ─► .xlsx
                                  ├─► idm/serialize.py              ─► .json
                                  └─► writers/searchable_pdf.py     ─► searchable .pdf

How the IDM is built

app/services/convert/layout/pipeline.py, per page:

Geometry — glyph boxes and vector path operators. The native engine supplies these when it is built; otherwise layout/pypdf_geometry.py walks the content stream directly, resolving fonts, colours, text render modes and link annotations. Advance widths come from /Widths, repaired by repair_width_map when pypdf cannot build the map itself — subset TrueType fonts without /Encoding otherwise under-report by about 35%, and because producers draw a line as consecutive Tj operators the error accumulates across it.
Lines — glyphs coalesced into lines and spans (layout/glyphs.py).
Reading order — recursive XY-cut (layout/reading_order.py). A vertical cut is legal only through a band that no line's extent crosses, which is what keeps two-column pages from interleaving.
Paragraphs — lines grouped into blocks (layout/paragraphs.py).
Tables — three detectors, most reliable first: ruled lattice (tables_lattice.py), drawn rectangles (tables_rects.py), whitespace columns (tables_stream.py), each gated by table_plausibility.py.
Headers and footers — repeated-band detection across pages (layout/headers_footers.py).
OCR, when the page needs it (ocr/rebuild.py).
Optimisation — merge wrapped rows, reconcile cell fills, normalise colours, collapse echoed text (layout/idm_optimize.py).

Routing: the engine picks the path

auto_policy.py gathers cheap signals — document type, pages needing OCR, image coverage, word count, and how ruled the pages are — and decides:

Signal	Decision
clean digital text layer	no OCR
≥95% of pages need OCR	force OCR
brochure-like pages, or image-led with little prose	positioned emit (DOCX)
≥50 field-sized rectangles per page	positioned emit — it is a form
image-only, nothing to OCR	table detection off

The form threshold is measured, not guessed: forms score 84 and 368 field rectangles per page, the most heavily ruled prose in the corpus scores 33, and everything else is under 3. layout/form_signals.py records the two text-shape signals that were tried first and discarded because they discriminated backwards.

Every decision lands on document.meta.convert_policy, so "why did page 7 come out as a picture?" has an answer.

Concurrency and shared state

Per-conversion state is in ContextVars, so concurrent conversions cannot see or clear each other's: the document cache, the cancel token and deadline, the options, the policy decision, and the OCR detection plan. Each scope resets on exit, so a pooled worker thread cannot leak one job's state into the next.

Three things are process-global and therefore locked:

What	Where	Why locked
RapidOCR engines	`services/ocr.py` `_engine_lock`	RapidOCR sizes detection by mutating attributes on the shared engine; two conversions were overwriting each other's resolution
Font metric cache	`layout/pypdf_geometry.py` `_FONT_CACHE_LOCK`	eviction was a check-then-clear-then-write across conversions
PDFium rasterisation	`services/raster.py` `_render_lock`	PDFium is not re-entrant

Recognition results are cached across conversions (services/ocr.py::_RecognitionCache), keyed by a digest of the image plus the Arabic flag. This is the one deliberately cross-conversion cache, and it exists because recognition was being paid once per output format: one 42-page scan converted to eleven targets ran OCR eleven times.

Bounds

Nothing here is unbounded. Each limit degrades gracefully unless marked:

Bound	Value	Behaviour at the limit
input size	50 MB	reject (413)
page count	200	reject (413)
conversion deadline	120 s (1800 s ceiling)	reject (504)
soft deadline	80% of budget	stop early, return what is built
OCR page cap	50	warn, skip the rest
OCR wall budget	1200 s	stop starting pages, keep the finished ones
raster cache	64 MB	LRU eviction
geometry cache	12 pages	LRU eviction
page-text cache	64 pages	LRU eviction
OCR result cache	256 entries / 64 MB / 30 min	LRU + TTL
in-flight conversions	CPU count, min 2	503 + Retry-After
queued conversions	32	503 + Retry-After
job store	256 jobs / 512 MB	evict oldest finished job
raster pixels	80 M, 10 000 px edge	downscale
content stream	12 MB, 20 000 path ops	stop walking, use what was read

What is deliberately not here

Authentication, durable job persistence, and object storage are the surrounding platform's (docqube's). The job store is explicitly in-process and non-durable; a restart loses queued and completed jobs. A second, weaker copy of someone else's durability guarantee would be a liability, not a feature.

Error taxonomy

app/services/convert/errors.py. Codes 1–7 describe a bad input; 8–11 describe a conversion that did not complete. retryable on the response body tells a client whether trying again could plausibly work.

Code	HTTP	Retryable
`unsupported`	415	no
`encrypted`	400	no
`needs_ocr`	422	no
`malformed`	400	no
`resource_limit`	413	no
`missing_part`	400	no
`io`	400	no
`timeout`	504	yes
`cancelled`	409	no
`invalid_output`	500	no
`internal`	500	yes

8.3 KiB Raw Blame History Unescape Escape