doctriage
An LLM service that classifies insurance claim documents, and knows when to ask a person.
- Role
- Sole engineer: design, build, tests, deploy
- When
- May – Jul 2026, 4 weeks of build
- Stack
- TypeScript · Fastify · Claude Haiku 4.5 (Anthropic API) · AWS Bedrock (Titan Embeddings V2, Claude judge) · Postgres + pgvector · Zod · pino · Docker Compose · Nginx · GitHub Actions
- Size
- ~3,900 lines of source · 98 tests, none needing real credentials
The problem
Insurance claims arrive as PDFs: claim forms, medical reports, police reports, repair estimates. Each one has to be identified before it can be routed. An LLM can do that well most of the time. The engineering problem is the rest of the time: the model is unsure, times out, returns malformed JSON, or reads a document that has been written to manipulate it.
So the goal wasn't a chatbot wrapper. It was a backend service that treats the model as an unreliable upstream dependency and still produces results the business can trust.
Request pipeline
Every external call has retry with backoff, a timeout, a logger bound to the document ID, and a cost entry, whether the call succeeded, was queued for review, or failed.
Decisions and tradeoffs
A confidence gate, not "trust every valid response"
Through week 2, any schema-valid classification was stored. But passing schema validation and being trustworthy are different things. Results below CLASSIFICATION_CONFIDENCE_THRESHOLD (0.7) now go to a review queue and stay out of the document record until a person resolves them. That way the stored field only ever means "trusted".
Honest caveat: 0.7 is a starting point, not a measured value. The next experiment is to label results across the confidence range and see where accuracy actually drops.
pgvector instead of a dedicated vector database
Postgres was already running for workflow state. Adding pgvector meant no fourth database on one VPS, and it allows hybrid queries: a relational WHERE clause and vector similarity in one statement, such as "relevant chunks from open claims filed in the last 30 days".
Rejected: MongoDB vector search, which needs managed Atlas and doesn't work on self-hosted mongo:7. Accepted: pgvector's ANN indexing is less mature at very large scale, which doesn't matter at this size.
Separate classify, embed and query endpoints
Extraction is local and free. Classification and embedding are paid, rate-limited external calls. Bundling them would make an upload fail whenever the LLM was down. Keeping them separate also lets the eval harness re-run classification alone when a prompt changes, without re-embedding anything.
The production version of "process this document" is a queue and a worker. I left that out on purpose, because this project was about getting synchronous LLM-call patterns right first.
Haiku over Sonnet, with the missing half of the argument stated
Sonnet costs 3× as much as Haiku per token. Cost alone doesn't justify a model choice, though. The full answer is that Sonnet costs 3× more, its accuracy advantage on this task hasn't been measured yet, and measuring it is cheap.
Three layers of prompt-injection defense
Each document is untrusted input that gets read verbatim into a prompt. The defenses are independent: (1) prompt v3 tells the model that anything inside <document> is data, never instructions; (2) sanitizeForPrompt neutralizes fake turn markers and common injection phrasing before the text reaches any prompt, including the judge's; (3) Zod rejects any output that breaks the contract. An attack has to beat all three.
v3 became the default immediately as a security fix. Accuracy changes go through the eval harness first, as described below.
Correlation IDs threaded through the call graph
Each route creates request.log.child({ documentId }) once and passes it into every service call. That replaced separate per-module loggers, so a document's full path through the pipeline can be found with one grep, even with concurrent requests.
The eval that stopped a prompt change
Prompts are versioned files resolved through a registry, and every classification logs which version produced it. pnpm eval runs 20 hand-written fixtures through two prompt versions and scores them in three ways: exact match on document type, a confidence-band check, and an LLM judge on Bedrock for the free-text reasoning. The fixtures are deliberately written by hand, because LLM-generated ground truth would just grade the model against another model's opinion.
Prompt v2 was meant to improve confidence calibration. Here is what the harness reported:
| Metric (20 fixtures) | v1 | v2 |
|---|---|---|
| Document type, exact match | 100% | 100% |
| Confidence in expected band | 70% | 75% |
| Reasoning passes judge | 95% | 85% |
| Pairwise wins (ties 8, inconclusive 3) | 4 | 5 |
v2 calibrated slightly better but reasoned worse, so it wasn't promoted. The pairwise judge runs each comparison twice with v1 and v2 swapped. If it picks different winners, the result is recorded as inconclusive rather than counted, so position bias can't decide the outcome. Every run is saved as a timestamped JSON file, so the next prompt change has a baseline to compare against.
Cost, measured per request
Each classification, embedding and judge call records its token usage and cost. Failed calls are recorded too, because they're still billed. GET /metrics returns totals and a breakdown by pipeline stage. Prices are checked against the official Anthropic and Bedrock pricing pages, with the check date noted in the source.
| Classification | Per document | Per 1,000 |
|---|---|---|
| Claude Haiku 4.5 (measured) | $0.001292 | $1.29 |
| Claude Sonnet (same tokens, 3× price) | ~$0.0039 | ~$3.90 |
Running it
- Docker Compose on a VPS I manage, behind Nginx with auto-renewing Let's Encrypt certificates. The app port isn't reachable from outside the machine.
- A push to
maindeploys through GitHub Actions over SSH. - A cron job on the VPS polls
/healthand sends a webhook alert if it fails. - The full test suite runs with no API keys: every external dependency has an in-memory or mocked implementation behind a repository interface.
Known limitations
/healthshows that the process is up, not that Postgres is reachable. A dependency-aware check is the next fix.GET /documents/:idcan't yet tell "never classified" apart from "waiting for human review". This was a known tradeoff when the review queue was added.- The current default prompt, v3, still needs its own eval run to confirm the security change didn't cost accuracy.