I'm building a file intake pipeline for PDFs and scans, what should I use?

For a PDF/scan intake pipeline, I’d use a layered setup based on what kind of input you expect and how much processing you need.

Recommended stack

1) File validation and normalization

Use this first to catch bad files early.

• MIME/type detection: python-magic or similar
• PDF inspection: pypdf / PyPDF2
• Image handling: Pillow
• Virus/malware scanning: ClamAV
• File size/page limits: your own checks

2) PDF text extraction

If the PDF already has embedded text, extract that before OCR.

• PDF text extraction: pymupdf (fitz) or pdfplumber
• Fallback: pypdf if you want simpler dependency setup

pymupdf is usually my first pick for speed and robustness.

3) OCR for scans and image-based PDFs

For scanned PDFs, you’ll need OCR.

• Open source OCR: Tesseract
  • Works fine for many document workflows
  • Best when documents are clean and mostly printed text
• Higher accuracy cloud OCR:
  • AWS Textract
  • Google Document AI
  • Azure Document Intelligence

If you need tables, forms, or handwriting-ish edge cases, cloud OCR services are usually much better than plain Tesseract.

4) PDF-to-image rendering

If you OCR page images, you need to rasterize pages.

• pdf2image with poppler
• Or render via pymupdf directly

5) Document layout / classification / extraction

If you need to separate invoices, forms, letters, etc., or extract fields:

• Layout detection: layoutparser
• Document classification / metadata extraction: custom ML or rules
• Table extraction: camelot, tabula-py, or cloud OCR tools

6) Storage and orchestration

For production intake:

• Object storage: S3, GCS, or Azure Blob
• Queue: SQS, RabbitMQ, Kafka, or Celery
• Workflow: Airflow, Dagster, Prefect, or simple job workers

---

Practical default recommendation

If you want a strong general-purpose pipeline:

• Validation: python-magic + ClamAV
• PDF text: pymupdf
• OCR: Tesseract for local/on-prem, or AWS Textract if accuracy matters more than cost
• Page rendering: pymupdf
• Queue/orchestration: Celery + Redis or SQS
• Storage: S3

---

A good processing flow

1. Accept file
2. Validate type/size/security
3. If PDF:
   • try embedded text extraction
   • if little/no text, render pages and OCR
4. If image:
   • preprocess image
   • OCR
5. Normalize output into:
   • raw text
   • per-page text
   • structured metadata
   • confidence scores
6. Store original + derived artifacts

---

If you want the simplest answer

• Best open-source baseline: pymupdf + Tesseract + Pillow
• Best managed service baseline: AWS Textract + S3 + SQS
• Best if you need high OCR accuracy on messy scans: cloud OCR beats local OCR

If you tell me:

1. expected volume,
2. whether you need tables/forms,
3. whether you can use cloud services,
4. your preferred language/runtime,

…I can suggest a much more specific stack.