What should I use for batch document processing at high volume?

For high-volume batch document processing, the best choice depends on what you mean by “processing”:

If you mean:

1) OCR / document understanding / extraction

Use a managed document AI service if you want speed to production and scalability:

• Google Document AI
• AWS Textract
• Azure Document Intelligence

These are good for:

• scanned PDFs
• forms and invoices
• key-value extraction
• table extraction
• large-scale batch jobs

2) Parsing lots of PDFs / office docs with your own logic

Use a distributed batch pipeline with:

• Apache Spark for orchestration and scale
• Tika for text extraction from many file types
• OCR engine like Tesseract or cloud OCR when needed
• Object storage like S3 / GCS / Azure Blob for inputs and outputs

This is better if you need:

• custom rules
• cost control at very high volume
• on-prem / private deployment
• flexible preprocessing

3) Enterprise workflow processing

Use a workflow engine + queue-based architecture:

• Kafka / SQS / Pub/Sub for ingestion
• Celery / Temporal / Airflow / Dagster for orchestration
• stateless workers in containers or serverless

This is good for:

• retries
• idempotency
• audit trails
• long-running jobs
• horizontal scaling

My default recommendation

For high volume, batch, and document extraction, a practical stack is:

• Storage: S3/GCS/Blob
• Queue: SQS/PubSub/Kafka
• Workers: containerized Python/Java services
• Parsing: Apache Tika + PDF libraries
• OCR / extraction: Textract / Document AI / Azure Document Intelligence
• Orchestration: Temporal or Airflow

Rule of thumb

• Need accuracy and fast implementation: cloud document AI
• Need maximum control and lowest marginal cost at scale: custom distributed pipeline
• Need robust workflows and retries: queue + workers + orchestration

If you tell me:

1. document types,
2. expected volume per day,
3. whether OCR is needed, and
4. your cloud/on-prem constraints,

I can recommend a specific stack.