Как пакетно распознавать и сканировать десятки тысяч PDF

Hybrid routing / parallel workers / SQLite cache / 30k-file script

Archive migration, contract audits, invoice filing — when you face tens of thousands of PDFs, the instinct is to install OCR software and click "batch." That works for hundreds of files; at 10k+ scale the bottleneck shifts from accuracy to disk I/O, redundant work, and no resume capability. This article describes a 2026-ready pipeline: detect text layers first, OCR only when needed, with multiprocessing and incremental cache to finish 30k documents overnight.

The headline takeaway: the fastest method is not the strongest OCR — it is avoiding OCR whenever possible. Native-text PDFs extract in milliseconds with PyMuPDF (50–200× faster than OCR). The old approach of OCR everything should become type-based routing.

Почему полный OCR не работает на десятках тысяч файлов

A 20-page scan takes 30–90 seconds to OCR. At 30k files with 8 cores, full OCR needs days to weeks. Yet 40–70% of batches are exported PDFs with intact text layers — page.get_text() finishes in milliseconds.

Classify before you recognize

Classification signals

Decision logic

• Text layer — any page with get_text() > 50 chars → text_native
• Scan signature — image-only pages → needs_ocr
• Encrypted / corrupt — log to error queue, don't block the pipeline

Design principle: split recognition into classify → extract → OCR fallback; each stage scales and resumes independently.

Три пути: извлечение, OCR, гибрид

Extract

Reads embedded font streams — default for bulk jobs.

OCR

Renders pages to bitmaps; CPU/GPU heavy — only for needs_ocr.

Hybrid

Walker → Classifier → Extract pool + OCR pool in parallel.

Самая быстрая архитектура: метаданные + параллельные воркеры

1. Walker — os.scandir collects paths without opening PDFs
2. Classifier — multiprocess first-page probe, writes file_type
3. Extract pool — handles text_native → JSONL / SQLite
4. OCR pool — consumes needs_ocr only; monitor load with Cmd + Activity Monitor

Сравнение инструментов

Продакшен-скрипт: 30 000 файлов за ночь

#!/usr/bin/env python3
import sqlite3, fitz
from pathlib import Path
from multiprocessing import Pool, cpu_count

DB = "pdf_index.sqlite3"
ROOT = Path("/data/invoices")

def classify(path: str) -> tuple:
    try:
        doc = fitz.open(path)
        sample = "".join(doc[i].get_text() for i in range(min(3, len(doc))))
        kind = "text_native" if len(sample.strip()) > 50 else "needs_ocr"
        doc.close()
        return (path, kind, "classified")
    except Exception as e:
        return (path, "error", str(e))

if __name__ == "__main__":
    files = [str(p) for p in ROOT.rglob("*.pdf")]
    with Pool(cpu_count()) as pool:
        for row in pool.imap_unordered(classify, files, chunksize=64):
            pass  # write to SQLite; OCR workers consume needs_ocr

On Apple Silicon, chunksize=64 saturates I/O better than default. Classification of 30k small PDFs on SSD: 20–40 minutes; extract adds 1–2 hours; OCR queue runs at pool capacity.

Настройка производительности

• MD5 incremental cache — skip unchanged files on reruns
• Avoid NFS for OCR — sync batch to local NVMe first
• Cap render DPI — 200–300 DPI is enough; 600 DPI is 4× slower
• Language packs — chi_sim for Chinese, eng for English; mixing hurts accuracy

FAQ

1. Stamps? — Text extract won't catch them; need OCR + detection models
2. Encrypted PDFs? — Decrypt first or log as skipped
3. GPU required? — No for extract; PaddleOCR ~3–5× faster on GPU
4. Full-text search? — JSONL → Elasticsearch, or SQLite FTS5 for mid scale

Итог: hybrid routing + parallelism + incremental cache beats any single OCR tool for 10k+ PDFs.