How do I extract tables from document images with OCR?

To extract tables from document images with OCR, the usual pipeline is:

1. Preprocess the image

   • Convert to grayscale
   • Denoise
   • Deskew / rotate correctly
   • Increase contrast
   • Binarize if needed

2. Detect the table region

   • Use layout analysis or table-detection models
   • Or detect lines/rectangles with OpenCV
   • Crop the table area first

3. Detect rows and columns / cell boundaries

   • For ruled tables: find horizontal and vertical lines
   • For unruled tables: use text alignment, spacing, and ML-based table structure models

4. Run OCR on each cell or text block

   • OCR the whole table, then assign text to cells
   • Or OCR each detected cell separately for better accuracy

5. Reconstruct the table

   • Map OCR text into a grid
   • Handle merged cells, multi-line cells, missing borders, and spanning headers

6. Export the result

   • CSV, Excel, JSON, HTML, or a database table

Common tools

Open-source

• OpenCV: image cleanup, line detection, cell segmentation
• Tesseract OCR: basic OCR, works better after preprocessing
• PaddleOCR: stronger OCR, includes layout/table tools
• Camelot / Tabula: best for PDFs, not raw images
• LayoutParser: document layout and table detection
• Table Transformer (Microsoft): strong table structure detection

Cloud APIs

• AWS Textract
• Google Document AI / Vision
• Azure Document Intelligence

These are often easiest if you want high accuracy with minimal setup.

Simple OpenCV + Tesseract approach

If the table has visible grid lines:

• Threshold the image
• Extract horizontal lines using morphological operations
• Extract vertical lines similarly
• Combine them to find cell boxes
• OCR each cell

Pseudo-steps:

# 1. preprocess
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
th = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                           cv2.THRESH_BINARY_INV, 15, 10)

# 2. detect lines
horizontal = extract_horizontal_lines(th)
vertical = extract_vertical_lines(th)

# 3. find cell contours
grid = cv2.add(horizontal, vertical)
cells = find_cell_contours(grid)

# 4. OCR each cell
text = pytesseract.image_to_string(cell_img)

If the table has no borders

Use:

• OCR word boxes
• Row grouping by Y-coordinate
• Column grouping by X-coordinate
• Table-structure models such as:
  • PubTabNet-trained models
  • Table Transformer
  • PaddleOCR table recognition

This is harder because you must infer structure from text layout.

Best practical choices

• For scanned forms with lines: OpenCV + Tesseract or PaddleOCR
• For complex tables: AWS Textract / Azure Document Intelligence / Google Document AI
• For research or local ML: Table Transformer + OCR engine

Important tips

• Correct rotation first
• Use high-resolution scans if possible
• OCR each cell when possible, not the whole page
• Expect manual cleanup for merged cells and headers
• Evaluate with cell-level accuracy, not just OCR text accuracy

If you want, I can give you:

1. a Python example using OpenCV + Tesseract,
2. a PaddleOCR-based solution, or
3. a cloud API comparison.