SystemFontProvider.NOTO_FONT_PATTERNS enumerates about two dozen Noto
families by name, and MultiFontManager only ever asked for those. Any
script outside that list fell through to the glyphless Occulta fallback
even when the correct font was installed, which is the common case on
macOS: it ships around a hundred script-specific Noto faces in
/System/Library/Fonts/Supplemental, almost none of which we knew how to
ask for. The reporter had the fonts and still got told to install them.
Add an optional GlyphSearchingFontProvider protocol (find_font_with_glyphs)
implemented by SystemFontProvider, BuiltinFontProvider and
ChainedFontProvider, and a new selection phase that uses it once the named
families fail. The system scan enumerates every Noto face, reusing the
existing variable-font/-Regular/-VF filename classification, and keeps
only the font it selects so a full scan does not retain every font file on
the system. Fonts found this way are remembered and retried by name for
later words. Capability detection is isinstance-based so third-party
providers keep working unchanged.
The warning itself was also unactionable: it named neither the characters
nor the script, which is why the reporter had to ask which package to
install. It now identifies what it could not render, e.g.
'Ꮳ' U+13E3 CHEROKEE LETTER TSA. A word mixing scripts that no single font
covers now gets a distinct message saying that installing fonts will not
help, rather than sending the user after fonts they already have.
Finally, the documented macOS install command was wrong: `brew install
font-noto` does not exist, since Homebrew has no single Noto package, only
a cask per family. The Fedora package name was also corrected, as
google-noto-fonts-common ships no actual fonts.
The test asserted that --mode skip preserves the structure tree, but with
the default --output-type auto the output runs through Ghostscript PDF/A
conversion, which discards /StructTreeRoot on Ghostscript 10.x (9.x kept
it). This failed on macOS CI and locally while passing on the Ubuntu
runners' Ghostscript 9.55. Pin the test to --output-type pdf so it exercises
OCRmyPDF's own structure-tree handling without the version-dependent GS step,
and document the caveat in advanced.md.
Ghostscript's PDF/A conversion re-embeds non-embedded CID (CJK) fonts by
substituting a system font, which corrupts the character-to-Unicode
mapping and silently destroys an existing text layer -- commonly the OCR
layer Adobe Acrobat adds to scanned CJK documents.
Detect non-embedded CID/Type0 fonts before conversion: with
--output-type auto (the default) downgrade to a regular PDF and preserve
the text layer; with an explicit --output-type pdfa* stop with an error
rather than emit corrupted output. Simple non-embedded fonts (e.g. Latin)
are left alone -- Ghostscript substitutes them without corrupting the
text, and they are far too common to treat as conversion blockers.
Use --output-type pdf to keep the existing text layer, or --force-ocr to
rebuild it with embedded fonts.
Writing the output PDF to stdout (ocrmypdf in.pdf -) previously relied on
an honor system: no in-process code -- third-party libraries, plugins, or
stray print() calls -- was supposed to write to stdout, enforced only
indirectly. A single accidental write to fd 1 would silently corrupt the
output PDF.
Enforce this at the OS level. At CLI startup, before plugins load or any
worker process/thread starts, save the real stdout via os.dup() and point
fd 1 at stderr, so stray writes are diverted to stderr while only the final
"produce the PDF" step writes to the preserved descriptor. Exposed as the
opt-in public API function configure_stdout_protection(), mirroring
configure_logging(); it is not enabled inside ocr() so in-process library
users keep their own stdout.
Also fix check_requested_output_file() to test the preserved real stdout
for tty-ness, since after the redirect sys.stdout reports stderr's status.
Fold unreleased v17.7.2 notes into v17.8.0.
Some PDFs use a /Name dictionary key in /DocumentInfo whose bytes are not
valid UTF-8/PDFDocEncoding, e.g. a Latin-1 /Saks#e5r. Older pikepdf raised
UnicodeDecodeError while iterating such a block, crashing the pipeline
during PDF/A conversion. repair_docinfo_nuls is documented to log and
continue on a malformed DocumentInfo block, so catch UnicodeDecodeError
alongside TypeError.
Add a mock-based unit test that drives the decode-error branch (current
pikepdf surrogate-escapes instead of raising) and an end-to-end test over
the reporter's file, committed as docinfo_latin1_key.pdf.
ReadTheDocs stopped injecting the GitHub edit context when it migrated to
Addons, so the sphinx_rtd_theme "Edit on GitHub" breadcrumb link vanished,
leaving only the static "View page source" (_sources/*.txt) copy. Set the
html_context explicitly so each page links back to its source on GitHub.
When TESSDATA_PREFIX points at a hand-assembled tessdata folder lacking
the configs/ subdirectory (e.g. files pulled from tessdata_best),
Tesseract prints "read_params_file: Can't open hocr/txt" and produces no
output. The runtime now surfaces a clear error (v17.5.0); add the
matching documentation: a new errors.md entry and a note on the
TESSDATA_PREFIX docs.
Since v16.4.3, OCRmyPDF forced pdfminer's read buffer to 256 MiB to work
around a pdfminer bug that mishandled tokens split across the buffer
boundary (gh #1361). On Windows this caused a severe performance
regression (gh #1662): CPython's BufferedReader.read(n) eagerly allocates
an n-byte buffer on every read, so pdfminer's thousands of seek+read
cycles each paid a ~30 ms 256 MiB allocation (this allocation is lazy and
effectively free on Linux). For a typical PDF the "Scanning contents"
phase went from ~5s on Linux to ~60s on Windows.
The underlying pdfminer bug was fixed upstream in pdfminer.six 20250327
(pdfminer/pdfminer.six#1030), with a follow-up for tokens split across
streams in 20260107 (pdfminer/pdfminer.six#1158). Remove the monkeypatch
entirely and raise the minimum pdfminer.six to 20260107 so we rely on the
upstream fix instead.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Harden the Docker images by dropping root privileges, and make the
bind-mount workflow less fiddly.
- Create a non-root `app` user (uid/gid 1000) in both images and add
`USER app` before the entrypoint, so ocrmypdf (and the
webservice/watcher) no longer run as root. This also fixes the
previously dangling `--chown=app:app`, which referenced a user that
was never created. The Ubuntu base ships a default `ubuntu`/1000 user,
so remove it first so `app` can take uid 1000 (parity with Alpine).
- Add `WORKDIR /data` (created and app-owned) so bind-mounted input and
output can be passed as relative paths without `--workdir`. The
webservice/watcher are now invoked by absolute path (`/app/*.py`)
since the working directory is no longer `/app`.
- Drop the redundant `ppa:alex-p/tesseract-ocr5` from the Ubuntu image:
Tesseract 5 ships in the Ubuntu archive as of 24.04, and the PPA had
no build for the 26.04 base, which broke the build outright.
- Rewrite docs/docker.md rootless-first: stdin/stdout piping as the
recommended permission-free path, then per-runtime volume guidance
(rootless Docker `--user 0:0`, Podman `--userns keep-id`, rootful
Docker as the special case). Update batch.md and the compose example
to match (absolute script paths, per-runtime `user:` guidance).
Adds a processing mode that removes the invisible (render mode 3) OCR text
layer in place. Unlike `--ocr-engine none --force-ocr`, it does not
rasterize the page, so images and visible content are preserved unchanged
and the output is smaller rather than larger. Options that require
rasterization or OCR (--deskew, --clean, --sidecar, etc.) are rejected.
Only invisible text is removed; text drawn as visible glyphs under an
opaque image (some OCR engines, and OCRmyPDF v2.2 and earlier) cannot be
removed this way, as documented.
Closes#1435.
When Tesseract reports a page with many diacritics, OCRmyPDF rewrites the
message to "lots of diacritics - possibly poor OCR", which hid the
original wording. Keep the interpreted hint but also emit Tesseract's raw
line at debug verbosity (-v 1) so users can see exactly what Tesseract
reported.
Closes#1566.
Ghostscript 10.x renders aliased glyphs that OCR frequently misreads as
extra word breaks or substituted characters. Enable text and graphics
anti-aliasing (-dTextAlphaBits=4 -dGraphicsAlphaBits=4) for the contone
raster devices, which empirically improves OCR accuracy on the
Ghostscript path, especially for small fonts at moderate DPI. The 1-bit
mono devices are excluded, since older Ghostscript rejects alpha bits on
them and pngmonod performs its own anti-aliased downscaling.
Also log which rasterizer rendered each page at debug verbosity and
clarify the --rasterizer help text, so quality reports are easier to
diagnose. The default rasterizer (auto) already prefers pypdfium2, which
anti-aliases; this primarily benefits --rasterizer ghostscript and
installs without pypdfium2.
Closes#1439.
The optimizer is best-effort: any image it cannot process is left
unchanged in the output, which remains valid. Previously, an extraction
failure (e.g. an exotic colorspace pikepdf cannot transcode) was logged
with log.exception, printing a full traceback at ERROR level that alarmed
users even though nothing was wrong with the output (issue #846).
Trap such failures with a concise warning that the image was left
unchanged and the output is still valid, and demote the traceback to
debug verbosity for diagnosis.
Ghostscript's `auto` image compression heuristic can transcode lossless
images to JPEG during PDF/A generation, which is surprising at
optimization levels that otherwise promise lossless-only operations
(issue #1124).
`--pdfa-image-compression=auto` (the default) now coerces to lossless at
-O0 so Ghostscript will not transcode lossless images to JPEG. -O1 and
above continue to defer to Ghostscript's heuristic; -O1 (the default
level) is kept as a historical exception because coercing it to lossless
substantially bloats output. Users wanting guaranteed lossless image
handling can pass --pdfa-image-compression=lossless or use -O0.
Also make `lossless` pass existing JPEGs through unchanged
(-dPassThroughJPEGImages=true) instead of re-encoding them with a
lossless codec, which only inflates already-lossy data.
Validate and repair the page-boundary boxes (MediaBox, CropBox, TrimBox,
ArtBox, BleedBox) of input PDFs in triage(), following PDF 2.0:
- Coerce coordinates written in invalid exponential notation, stored by
qpdf/pikepdf as strings (#1398).
- Normalize rectangles whose corners are given in reversed order, which
previously crashed with NegativeDimensionError (#1526).
- Clamp a crop/trim/art/bleed box that extends outside the MediaBox to
their intersection, or discard it when the intersection is empty, which
previously produced a zero-height effective page some viewers rejected
(#1400).
When a box is discarded, clamped, or reinterpreted, a warning recommends
visual inspection of the output. pdfinfo box reading shares the same
coercion helper so PdfInfo no longer crashes on malformed boxes.
Supersedes PR #1691. Thanks @ajdlinux.
Closes#1398, #1526, #1400
A tagged/structured PDF carries a logical structure tree
(/Root/StructTreeRoot, /MarkInfo) that maps marked page content to
semantic elements via MCIDs. When --force-ocr rasterizes pages or
--redo-ocr rewrites the text layer, those MCIDs are destroyed or
renumbered and the tree is left dangling. We cannot rebuild it to match
the new text, so discard it for force/redo modes, following the same
pattern as the thumbnail and search-index discards. --skip-text leaves
text pages untouched, so their structure is preserved.
Also broaden the default-mode "looks born-digital" stop signal to fire
on /StructTreeRoot, not just /MarkInfo/Marked, so structure-tree-only
PDFs are no longer silently OCR'd. The existing --tagged-pdf-mode ignore
escape hatch is unchanged.
A page may carry an optional /Thumb image XObject (ISO 32000-2 12.3.4),
a miniature rendering used only as a navigation aid. OCRmyPDF alters
page appearance (deskew, clean, rasterize, re-render) and plugins may
edit pages arbitrarily, so a retained thumbnail would be stale and no
longer match its page. Modern viewers generate thumbnails on demand, so
there is no loss of functionality.
OcrGrafter.finalize() now strips /Thumb from every page before saving,
alongside the existing search-index discard, covering both the OCR and
hOCR pipelines. Orphaned thumbnail XObjects are garbage-collected on save.
Adobe Acrobat can embed a proprietary full-text search index in the
document catalog at /Root/PieceInfo/SearchIndex. Only Acrobat reads it;
other viewers ignore it and search the text on the fly. Any change to a
PDF invalidates the index, so once OCRmyPDF rewrites the document the
retained index is stale and returns incorrect search results in Acrobat.
OcrGrafter.finalize() now discards it before saving (covering both the
OCR and hOCR pipelines), preserving any other PieceInfo owner data and
dropping an empty PieceInfo. Modern viewers rebuild a search index on
demand, so there is no loss of search capability.
Closes#1615. The token 'end' (case-insensitive) is now accepted as an
alias for the document's last page, e.g. --pages 3-end. Resolution is
deferred until the page count is known from the input PDF.
Expose Ghostscript's -dJPEGQ and image downsampling switches as
advanced, plugin-scoped options for tuning PDF/A output, without
polluting the central OcrOptions registry. The optimizer's existing
--jpeg-quality remains the recommended JPEG quality control.
- GhostscriptOptions gains jpeg_quality and jpeg_maxdpi fields and CLI
args (advanced help text). jpeg_quality=0 is honored as Ghostscript's
maximum compression rather than being silently coerced to the default.
- _exec.ghostscript.generate_pdfa() forwards both values; when
jpeg_maxdpi is set, downsample threshold is pinned at 1.0.
- _get_plugin_options falls back to extra_attrs for namespaced fields
so plugins can own their options without registering them centrally.
- Documentation explains the rationale: Ghostscript is the legacy path
(pypdfium + verapdf is preferred in v17+), the optimizer is the
supported file-size lever, and lowering quality is almost always a
better trade than downsampling.
The API previously clobbered PIL.Image.MAX_IMAGE_PIXELS unconditionally
on every call, so host applications (e.g. Paperless-NGX) that configured
the PIL limit before invoking ocrmypdf.ocr() saw their setting silently
overwritten with the 250 MP default. Make max_image_mpixels default to
None and only apply the override when the caller explicitly sets it.
The CLI default of 250 MP is unchanged.
Fixes#1665
poppler/pdftotext does not carry Tz (horizontal scaling) across
BT/ET boundaries, causing words to appear on separate lines.
Replace per-word BT blocks (via fpdf2's cell/set_stretching API)
with a single BT block per line using raw PDF operators. Each
non-last word gets a trailing space with Tz calculated to span
exactly to the next word's start position.
The previous fix (e62e73e4) only corrected text_rendering_mode →
text_mode in the single-page Fpdf2PdfRenderer, but the main OCR
pipeline uses Fpdf2MultiPageRenderer which still had the old
attribute name. Since fpdf2 has no text_rendering_mode property,
setting it silently created a no-op attribute while text_mode stayed
at FILL — so 3 Tr (invisible text) was never emitted.
Fixes#1631, #1632
- Prioritize uv over pip throughout, with uv as the recommended installer
- Update repology badges: Debian 13, Ubuntu 24.04, Fedora 40/41
- Make Python 3.12 the default (3.11 still supported)
- Promote Homebrew as full-featured option for macOS and Linux
- Add dependency summary table aligned with maintainers.md
- Document uharfbuzz and fonts-noto requirements
- Remove outdated warnings and simplify 32-bit section
Python 3.11 is now the minimum supported version. This aligns with
the codebase's use of StrEnum (introduced in 3.11) and removes
compatibility shims that were only needed for older versions.
Document the new v17 API style where OcrOptions can be passed directly
to ocr(). Mark the positional argument style as legacy API for <v17
compatibility. Update examples to use modern syntax.
Establish clear separation between user-facing optional dependencies
and developer-only dependency groups:
**Optional Dependencies (user features):**
- watcher: File watching service for batch processing
- webservice: Streamlit-based web UI
- Installable via: uv sync --extra <name> or pip install ocrmypdf[name]
**Dependency Groups (developer tools):**
- test: Testing infrastructure (merged from test + extended_test)
- docs: Documentation building tools
- streamlit-dev: Enhanced Streamlit development tools
- dev: General development tools (mypy, ipykernel)
- Installable via: uv sync --group <name> (uv only, NOT pip)
Breaking changes for developers:
- pip install -e .[test] no longer works → use uv sync --group test
- pip install -e .[docs] no longer works → use uv sync --group docs
- pip install -e .[extended_test] removed → merged into test group
No breaking changes for end users:
- pip install ocrmypdf[watcher] still works
- pip install ocrmypdf[webservice] still works
Updated:
- CI/CD workflows to use uv sync --group test
- Docker images to exclude test dependencies
- Documentation to recommend uv with pip as fallback
- pyproject.toml with clear comments explaining both systems
