extract-pdf-stats

automating extraction of statistical data from State published data.

Extracts statistical data (tables / multi-column figures) out of PDF reports, including scanned documents that need OCR and pages whose table content is rotated 90° inside an otherwise portrait page.

Setup

python3 -m venv venv
./venv/bin/python3 -m pip install -r requirements.txt

Also requires these system packages (Debian/Ubuntu):

sudo apt install poppler-utils tesseract-ocr

Usage

./venv/bin/python3 extract_pdf_stats.py file1.pdf [file2.pdf ...] [options]
./venv/bin/python3 extract_pdf_stats.py --all          # every *.pdf in the cwd

Options:

• --out DIR — output directory (default: ./extracted)
• --dpi N — OCR render resolution (default: 300)
• --pages "1,5,10-15" — only process these pages (1-indexed, applies to every file given in that invocation — see examples below for per-file ranges)
• --save-images — also save the rotated/deskewed page image used for OCR, for spot-checking
• --text-threshold N — min native-text chars before a page counts as "digital" (default: 40)

There is no option to tell it about page orientation — every page's rotation (0/90/180/270°) and residual skew are detected and corrected automatically (see "How it handles each page" below), so a batch of mixed portrait/landscape PDFs, or a PDF with some pages rotated and some not, all runs through the same command with nothing special to flag.

Examples

Process every PDF in the current directory, full auto:

./venv/bin/python3 extract_pdf_stats.py --all

Process a specific set of PDFs by name (mix of digital and scanned is fine):

./venv/bin/python3 extract_pdf_stats.py "18-19_Home_School_Annual_Report (1).pdf" \
  20201230121817_1998_1999_Annual_Reportrv.pdf

Same page range applied across multiple PDFs in one run (e.g. checking just the first few pages of several reports before committing to a full run):

./venv/bin/python3 extract_pdf_stats.py report_1997.pdf report_1998.pdf report_1999.pdf --pages "1-3"

Different page ranges per PDF — --pages applies to every file named in a single invocation, so give each file its own command instead:

./venv/bin/python3 extract_pdf_stats.py report_1997.pdf --pages "5-9"
./venv/bin/python3 extract_pdf_stats.py report_1998.pdf --pages "12-24"

Re-run just a handful of pages you want to spot-check, saving the OCR source images alongside the CSVs so you can compare them directly:

./venv/bin/python3 extract_pdf_stats.py report_1998.pdf --pages "12,20,25" --save-images

Output layout, per input PDF:

extracted/<pdf-stem>/page_<n>.txt         reconstructed layout text
extracted/<pdf-stem>/page_<n>_table.csv   best-effort row/column reconstruction
extracted/<pdf-stem>/page_<n>.png         OCR source image (only with --save-images)
extracted/<pdf-stem>/manifest.json        per-page method/rotation/skew/table log

manifest.json records, per page: method (digital or ocr), rotation (0/90/180/270), skew (residual degrees corrected after that), table_found, columns, and table_source (ruled = real gridlines found and used, clustered = no gridlines, fell back to whitespace-position clustering). Use it to quickly spot which pages are worth a manual check.

How it handles each page

1. Born-digital text with a real ruled table — pdfplumber's line-based table extraction, with multi-line merged cells (a bordered box with no internal row rules) split back into individual rows.
2. Born-digital text, multi-column but no table lines (e.g. three County 1,234 blocks side by side) — shells out to pdftotext -layout, which lays text out on a character grid, then finds column boundaries as whitespace that's consistent down the page. Blank-line-separated blocks (title, table, footnote) are column-detected independently so unrelated layouts don't cross-contaminate each other's boundaries.
3. Scanned/image page — rendered to an image, then:
   • Coarse rotation: Tesseract's orientation-and-script detection snaps the page to 0/90/180/270° (handles a landscape chart/table baked sideways into an otherwise-portrait page).
   • Deskew: a photocopy/scan is rarely perfectly level even after that — long ruled lines are found via Hough transform and the page is rotated by the residual angle (typically a fraction of a degree up to a few degrees) so table gridlines end up pixel-aligned.
   • Gridline detection: if the page has a real ruled table, its row/column lines are located directly (tolerant of the small skew/scan noise that would otherwise break a naive single-pixel-wide line search) and each cell is cropped and OCR'd independently — far more reliable for dense many-column tables than inferring columns from word positions. Columns that come back mostly numeric get a second OCR pass with a digit-only whitelist, which fixes a class of misread (e.g. a degraded "2" read as "a") a general-alphabet pass can't.
   • If no gridlines are found (a form, chart, or plain paragraph text), it falls back to the same whitespace-position column clustering as case 2, using OCR word pixel positions instead of character columns.
   • Ruling lines are painted out before OCR in both cases — left in, they're frequently misread as stray |/+ characters.

Known limitations

• Dense, many-column scanned grid tables on a degraded photocopy (e.g. a 15+ column grade-by-county count) are still the hardest case: individual digit misreads happen even with the gridline + digit-whitelist passes above, though they're now isolated single-cell errors rather than the wholesale column misalignment an earlier version of this tool produced. Check manifest.json for table_source: "clustered" pages (no gridlines found) and any page where the column count looks off, and hand-check those against the saved page image (--save-images); the .txt file next to each .csv has the raw OCR text as a fallback.
• Bar/line charts are not parsed as charts — only their printed data-label text is recovered via OCR.
• The lattice (cell-by-cell) OCR path is slow: roughly 1.5–2 minutes per page for a dense table, since each of its (often 500+) cells is a separate Tesseract invocation, doubled for numeric columns by the digit-whitelist pass. Use --pages to scope a run instead of processing an entire report at once if you just need a few pages.