Case study · 2026
Proper Papers
Every extracted value links back to the exact text it came from.
A document-intelligence app that turns service-agreement PDFs into structured, reviewable records. Missing values stay unknown; conflicts go to a human.
01
Problem and answer
The problem. Data pulled out of a contract PDF is only useful if someone can check where each value came from, and what the document actually says when it disagrees with itself.
The answer. Every found value carries a quote that must match the page text at its offsets; a mismatch fails the run.
- A missing value is recorded as unknown, with a reason.
- Clauses that disagree become a conflict listing every candidate with its own quote.
- A reviewer picks one candidate or marks it unknown. No free-text editing, no confidence score.
02
Workflow
- 01
Upload
PDF only, size-capped, encrypted PDFs refused. Scanned with ClamAV before anything is stored; any scanner failure refuses the upload.
- 02
Process
An explicit action. A run is queued in PostgreSQL; a worker parses the PDF in the sandbox and extracts the record with deterministic rules.
- 03
Review
Conflicts and missing required fields need a decision.
- 04
Approve
Freezes the run and the decisions into a new immutable version with a SHA-256. Approving again after a later decision makes the next version.
- 05
Export
Returns the stored JSON of one version after re-checking its hash.
Review
Conflicts in plain sight
Two effective dates, each quoted from the page. Nothing is guessed on the reviewer’s behalf.
Approve
An immutable version
Once every flagged field is decided, the record can be approved. Changing a decision afterwards does not edit the approved version; it is approved again as the next one.
03
Architecture
System shape
- Browser
- Next.js server
- FastAPI
- PostgreSQL 18row-level security
- Worker
- Parser containerno network
Python 3.14 (FastAPI, SQLAlchemy, Alembic), Next.js App Router (TypeScript), pdfminer.six. ClamAV scans every upload before it is stored.
Tenants are created only by an operator CLI, and the access token stays in a sealed HttpOnly cookie that only the server reads.
Deliberately left out: OCR, retrieval or vector search, a broker, object storage, caching, rate limiting and hosting.
04
Decisions and trade-offs
- AIsolated PDF processing
- Untrusted PDFs never run inside the API. Each document is parsed in a throwaway container with no network, a read-only root and resource limits, and there is no in-process fallback. A production runner that does not need the Docker socket is a release blocker.
- BDatabase-enforced tenant isolation
- Forced row-level security plus a membership check decide who sees a row, and the API refuses to use a runtime role that could bypass either. A development review found that a role belonging to the owner role slipped past the first guard and could read every tenant; the guard now checks role membership, and the regression tests fail against the old one. Known gap: RLS keyed on session settings does not stop arbitrary SQL run as the runtime role.
- CImmutable, versioned approvals
- Approval freezes the run and every decision into a new version with a SHA-256. Approved records cannot be updated or deleted, even by the runtime role, and export re-checks the hash. A later decision shows as “changed since version N”; approving again makes version N+1.
- Trade-offA Postgres job table, one worker
- Runs are queued with FOR UPDATE SKIP LOCKED in PostgreSQL instead of a message broker.
- Trade-offRules at runtime, not a model
- The model-assisted extractor failed its pre-registered gates, so the product stays rules-only (see the evaluation below).
05
Evaluation
Evaluated · not enabled in the product
Model assistance raised held-out accuracy from 0.63 to 0.96, but it failed gates fixed in advance, so it is not enabled.
The corpus is 80 synthetic PDFs with ground truth: a dev split the rules were written against, three held-out splits and a 10-case split for the model-assisted work. Held-out sets and pass/fail thresholds were committed before the extractor they judge was frozen.
The rules extractor is exact on the dev split. On held-out data its field accuracy is 0.63–0.77 with no wrong values and abstention recall of 1.0: it misses cases rather than inventing values.
A model-assisted extractor (the model proposes quoted spans for fields the rules left empty; each span is checked against the page text) was measured twice. Worst of three runs against the rules on the same split:
| Held-out split | Field accuracy | Evidence completeness | Wrong values | Silent conflicts |
|---|---|---|---|---|
| heldout-2 (16 cases) | 0.6339 → 0.9643 | 0.3538 → 0.9385 | 0 → 2 | 1 → 1 |
| heldout-3 (24 cases) | 0.7679 → 0.9762 | 0.4459 → 0.9459 | 0 → 0 | 2 → 3 |
Each attempt failed a gate fixed in advance (wrong values, then silent conflicts), so the product stays rules-only: it prefers missing a value to inventing one. Nothing was tuned on held-out data and there was no third attempt. Model responses are committed and CI replays them without an API key.
Limits: one author wrote the held-out documents and the extractors; the sets are small and more regular than real contracts. Read the numbers as directional.
06
Verification and limits
Tested beyond the happy path.
Proper Papers has automated tests across the API, the PDF parser and the frontend, covering tenant isolation, unsafe uploads, evidence validation, conflicting values and approval integrity.
I also completed a local end-to-end walkthrough, from document upload to multiple approval versions (the recording above shows the flow).
It is a working local portfolio application, not a deployed production service. It handles text-based service agreements rather than scanned PDFs, has had no external audit and no load testing, and one person can both decide and approve a record.
Test results, setup instructions and remaining production considerations are in the repository:README verification (opens in a new tab)demo walkthrough (opens in a new tab)deployment security (opens in a new tab)
07
Read the code
github.com/Novaav/proper-papers (opens in a new tab)
- Evidence handlingcontract.py (opens in a new tab)test_extraction.py (opens in a new tab)
- Tenant isolation0002_tenants_rls.py (opens in a new tab)test_rls_isolation.py (opens in a new tab)
- Sandboxsandbox.py (opens in a new tab)test_parser_sandbox.py (opens in a new tab)
- Review and immutabilityreview.py (opens in a new tab)test_review_schema.py (opens in a new tab)
- Evaluationevaluation.md (opens in a new tab)eval_scoring.py (opens in a new tab)
- Docssystem-design (opens in a new tab)security (opens in a new tab)demo (opens in a new tab)
Source-available under an all-rights-reserved licence, not open source: the code is public to view and evaluate. The repository is a curated snapshot, so its public history is not the full development history. Facts, figures and screenshots here come from the repository's README and demo assets (read 13 Oct 2026); this site did not re-run its tests or evaluations. There is no hosted demo.