How do I build a pipeline for mixed document types with OCR and human review?

A good mixed-document pipeline usually has 6 stages:

1. Ingest
2. Classify document type
3. Extract text/data
4. Run validation and confidence checks
5. Route low-confidence items to human review
6. Store outputs + feedback for continual improvement

Here’s a practical architecture.

---

1) Ingest documents from multiple sources

Accept files from:

• email attachments
• scanned PDFs
• office documents
• images
• APIs / SFTP / uploads

Normalize everything into a common intake format:

• save the original file
• generate a document ID
• record source, timestamp, sender, and metadata

Tip: keep the raw original immutable for auditability.

---

2) Preprocess the files

Before OCR or extraction:

• detect file type
• split multipage PDFs
• deskew / de-noise images
• rotate pages
• remove blank pages
• enhance contrast for scans

This improves OCR quality a lot.

---

3) Classify document types

For mixed documents, first determine what each file/page is:

• invoice
• receipt
• contract
• form
• ID card
• bank statement
• letter, etc.

You can do this with:

• a layout-aware ML model
• rules based on keywords
• LLM-assisted classification
• a hybrid approach

Best practice: classify at both:

• document level
• page level

This helps when one PDF contains multiple document types.

---

4) Extract text and structured data

Use different extraction methods depending on the file:

Digital PDFs / Office files

• extract embedded text directly
• preserve layout when possible

Scanned docs / images

• run OCR
• use layout-aware OCR if the document has tables, forms, or multi-column text

Structured documents

For forms and invoices:

• field extraction using templates or ML models
• table extraction
• key-value pair extraction

Store:

• raw OCR text
• word-level bounding boxes
• page coordinates
• extracted fields
• confidence scores

---

5) Add validation rules and confidence thresholds

Don’t rely on OCR alone. Add checks like:

• required fields present?
• date formats valid?
• totals match line items?
• page count within expected range?
• confidence below threshold?
• mismatch between document type and extracted fields?

Example:

• OCR confidence < 85% → send to review
• missing invoice total → send to review
• suspiciously low-quality scan → send to review

Use both:

• hard rules for deterministic failures
• soft thresholds for uncertainty

---

6) Human-in-the-loop review

Build a review interface for exceptions.

Review queue should show:

• original document
• OCR text overlay on image
• extracted fields
• confidence scores
• validation errors
• suggested values

Reviewers should be able to:

• correct OCR text
• confirm or change document type
• fix extracted fields
• mark document as unreadable / duplicate / invalid

Track:

• reviewer ID
• changes made
• timestamps
• reason codes

This creates an audit trail and training data for model improvement.

---

7) Feedback loop

Use reviewed corrections to improve the system:

• retrain document classifier
• tune OCR preprocessing
• update rules
• create templates for common document types
• improve reviewer routing thresholds

Measure:

• OCR accuracy
• field-level precision/recall
• review rate
• average review time
• false positive/negative rates

---

Reference architecture

A typical implementation looks like this:

Ingestion service
→ stores raw docs in object storage
→ sends job to queue

Preprocessing service
→ image cleanup, PDF splitting, normalization

Classification service
→ predicts doc type and page type

OCR / extraction service
→ OCR + field extraction + table parsing

Validation service
→ business rules + confidence scoring

Review service
→ human correction UI

Persistence layer

• object storage for originals
• database for metadata and extracted fields
• search index for text
• audit log for all actions

---

Recommended design pattern

Use an exception-based pipeline:

• process automatically when confidence is high
• send only uncertain items to human review

This is much cheaper and faster than reviewing everything manually.

---

Example decision flow

1. Document arrives
2. System detects PDF/image/email attachment
3. Preprocesses and classifies as invoice
4. OCR extracts text and fields
5. Validation finds total mismatch
6. Confidence on vendor name is low
7. Send to human review
8. Reviewer corrects values
9. Save corrections and final output
10. Use correction for future model improvement

---

Technology options

Depending on your stack:

• OCR: Tesseract, AWS Textract, Azure Form Recognizer, Google Document AI, PaddleOCR
• Workflow orchestration: Airflow, Prefect, Temporal, Step Functions
• Queues: SQS, RabbitMQ, Kafka
• Storage: S3 / Blob Storage
• Review UI: custom web app, low-code annotation tools, or enterprise IDP platforms
• Search/indexing: Elasticsearch, OpenSearch
• DB: Postgres

---

Best practices

• Keep original files unchanged
• Store page-level OCR output, not just final text
• Use page and document confidence scores
• Design for partial automation
• Log all human corrections
• Separate extraction logic from business validation
• Support batch processing and real-time ingestion
• Version your models, rules, and templates

---

If you want a minimal MVP

Start with:

1. upload endpoint
2. OCR
3. document classification
4. confidence scoring
5. review queue for low-confidence docs
6. export structured JSON

That gets you a working system fast.

---

If you want, I can also give you:

• a sample architecture diagram
• a step-by-step implementation plan
• or a tech stack recommendation for Python/AWS/Azure/GCP.