From 14ac9b05609819e80b3e3dbbbedb3b8173a79d42 Mon Sep 17 00:00:00 2001 From: "James R. Barlow" Date: Sun, 11 Jun 2023 13:37:53 -0700 Subject: [PATCH 1/2] docs: some reorganizing --- docs/advanced.rst | 23 ++++++++++-- docs/cloud.rst | 87 ++++++++++++++++++++++++++++++++++++++++++++ docs/index.rst | 3 +- docs/optimizer.rst | 39 +++++++++++++++----- docs/pdfsecurity.rst | 78 --------------------------------------- 5 files changed, 138 insertions(+), 92 deletions(-) create mode 100644 docs/cloud.rst diff --git a/docs/advanced.rst b/docs/advanced.rst index 5061c575..f4297a6d 100644 --- a/docs/advanced.rst +++ b/docs/advanced.rst @@ -310,16 +310,31 @@ stable user interface. They may be imported from - The program was interrupted by pressing Ctrl+C. +.. _tmpdir: + +Changing temporary storage location +=================================== + +OCRmyPDF generates many temporary files during processing. + +To change where temporary files are stored, change the ``TMPDIR`` +environment variable for ocrmypdf's environment. (Python's +``tempfile.gettempdir()`` returns the root directory in which temporary +files will be stored.) For example, one could redirect ``TMPDIR`` to a +large RAM disk to avoid wear on HDD/SSD and potentially improve +performance. + +On Windows, the ``TEMP`` environment variable is used instead. + Debugging the intermediate files ================================ OCRmyPDF normally saves its intermediate results to a temporary folder and deletes this folder when it exits, whether it succeeded or failed. -If the ``-k`` argument is issued on the command line, OCRmyPDF will keep -the temporary folder and print the location, whether it succeeded or -failed (provided the Python interpreter did not crash). An example -message is: +If the ``--keep-temporary-files`` (``-k```) argument is issued on the +command line, OCRmyPDF will keep the temporary folder and print the location, +whether it succeeded or failed. An example message is: .. code-block:: none diff --git a/docs/cloud.rst b/docs/cloud.rst new file mode 100644 index 00000000..32d8ad18 --- /dev/null +++ b/docs/cloud.rst @@ -0,0 +1,87 @@ +.. _ocr-service: + +================== +Online deployments +================== + +OCRmyPDF is designed to be used as a command line tool, but it can be +used in a web service. This document describes some considerations for +doing so. + +A basic web service implementation is provided in the source code +repository, as ``misc/webservice.py``. It is only demonstration quality +and is not intended for production use. + +OCRmyPDF is not designed for use as a public web service where a +malicious user could upload a chosen PDF. In particular, it is not +necessarily secure against PDF malware or PDFs that cause denial of +service. For further discussino of security, see :ref:`security`. + +OCRmyPDF relies on Ghostscript, and therefore, if deployed +online one should be prepared to comply with Ghostscript's Affero GPL +license, and any other licenses. + +Setting aside these concerns, a side effect of OCRmyPDF is that it may +incidentally sanitize PDFs containing certain types of malware. It +repairs the PDF with pikepdf/libqpdf, which could correct malformed PDF +structures that are part of an attack. When PDF/A output is selected +(the default), the input PDF is partially reconstructed by Ghostscript. +When ``--force-ocr`` is used, all pages are rasterized and reconverted +to PDF, which could remove malware in embedded images. + +Limiting CPU usage +------------------ + +OCRmyPDF will attempt to use all available CPUs and storage, so +executing ``nice ocrmypdf`` or limiting the number of jobs with the +``--jobs`` argument may ensure the server remains responsive. Another option +would be to run OCRmyPDF jobs inside a Docker container, a virtual machine, +or a cloud instance, which can impose its own limits on CPU usage and be +terminated "from orbit" if it fails to complete. + +Temporary storage requirements +------------------------------ + +OCRmyPDF will use a large amount of temporary storage for its work, +proportional to the total number of pixels needed to rasterize the PDF. +The raster image of a 8.5×11" color page at 300 DPI takes 25 MB +uncompressed; OCRmyPDF saves its intermediates as PNG, but that still +means it requires about 9 MB per intermediate based on average +compression ratios. Multiple intermediates per page are also required, +depending on the command line given. A rule of thumb would be to allow +100 MB of temporary storage per page in a file – meaning that a small +cloud servers or small VM partitions should be provisioned with plenty +of extra space, if say, a 500 page file might be sent. + +To change the temporary directory, see :ref:`tmpdir`. + +On Amazon Web Services or other cloud vendors, consider setting your +temporary directory to `empheral +storage `__. + +Timeouts +-------- + +To prevent excessively long OCR jobs consider setting +``--tesseract-timeout`` and/or ``--skip-big`` arguments. ``--skip-big`` +is particularly helpful if your PDFs include documents such as reports +on standard page sizes with large images attached - often large images +are not worth OCR'ing anyway. + +Document management systems +--------------------------- + +If you are looking for a full document management system, consider +`paperless-ngx `__, +which is a web application that uses OCRmyPDF to automatically OCR and +archive documents. + +Commercial OCR alternatives +--------------------------- + +The author also provides professional services that include OCR and +building databases around PDFs, and is happy to provide consultation. + +Abbyy Cloud OCR is viable commercial alternative with a web services +API. Amazon Textract, Google Cloud Vision, and Microsoft Azure +Computer Vision provide advanced OCR but have less PDF rendering capability. \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index c5d1da7a..682c91b7 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -20,7 +20,6 @@ image processing and OCR to existing PDFs. introduction release_notes installation - optimizer languages jbig2 @@ -29,9 +28,11 @@ image processing and OCR to existing PDFs. :maxdepth: 2 cookbook + optimizer docker advanced batch + cloud performance pdfsecurity errors diff --git a/docs/optimizer.rst b/docs/optimizer.rst index 4d3f8bbb..82768a85 100644 --- a/docs/optimizer.rst +++ b/docs/optimizer.rst @@ -13,14 +13,33 @@ tuned. Optimization occurs after OCR, and only if OCR succeeded. It does not perform other possible optimizations such as deduplicating resources, consolidating fonts, simplifying vector drawings, or anything of that nature. -Optimization ranges from ``-O0`` through ``-O3``, where ``0`` disables -optimization and ``3`` implements all options. ``1``, the default, performs only -safe and lossless optimizations. (This is similar to GCC's optimization -parameter.) The exact type of optimizations performed will vary over time. +.. list-table:: Title + :widths: 33 6 60 + :header-rows: 1 -PDF optimization requires third-party, optional tools for certain optimizations. -If these are not installed or cannot be found by OCRmyPDF, optimization will not -be as good. + * - Optimization level + - Shorthand + - Description + * - ``--optimize 0`` + - ``-O0`` + - Disable most optimizations. + * - ``--optimize 1`` (default) + - ``-O1`` + - Safe and lossless optimizations. + * - ``--optimize 2`` + - ``-O2`` + - Safe and lossy optimizations. + * - ``--optimize 3`` + - ``-O3`` + - Aggressive lossy optimizations. + +The exact type of optimizations performed will vary over time, and depend on +the availability of third-party tools. + +Despite optimizations, OCRmyPDF might still increase the overall file size, +since it must embed information about the recognized text, and depending on the +settings chosen, may not be able to represent the output file as compactly as +the input file. Optimizations that always occurs ================================ @@ -37,12 +56,14 @@ Fast web view OCRmyPDF automatically optimizes PDFs for "fast web view" in Adobe Acrobat's parlance, or equivalently, linearizes PDFs so that the resources they reference are presented in the order a viewer needs them for sequential display. This -reduces the latency of viewing a PDF both online and from local storage. This -actually slightly increases the file size. +reduces the latency of viewing a PDF both online and from local storage, in +exchange for a slight increase in file size. To disable this optimization and all others, use ``ocrmypdf --optimize 0 ...`` or the shorthand ``-O0``. +Adobe Acrobat might not report the file as being "fast web view". + Lossless optimizations ====================== diff --git a/docs/pdfsecurity.rst b/docs/pdfsecurity.rst index 40ea55cb..a3e849fc 100644 --- a/docs/pdfsecurity.rst +++ b/docs/pdfsecurity.rst @@ -54,84 +54,6 @@ into the existing PDF or it may essentially reconstruct ("re-fry") a visually identical PDF that may be quite different at the binary level. That said, OCRmyPDF is not a tool designed for sanitizing PDFs. -.. _ocr-service: - -Using OCRmyPDF online or as a service -===================================== - -OCRmyPDF is not designed for use as a public web service where a -malicious user could upload a chosen PDF. In particular, it is not -necessarily secure against PDF malware or PDFs that cause denial of -service. OCRmyPDF relies on Ghostscript, and therefore, if deployed -online one should be prepared to comply with Ghostscript's Affero GPL -license, and any other licenses. - -Setting aside these concerns, a side effect of OCRmyPDF is that it may -incidentally sanitize PDFs containing certain types of malware. It -repairs the PDF with pikepdf/libqpdf, which could correct malformed PDF -structures that are part of an attack. When PDF/A output is selected -(the default), the input PDF is partially reconstructed by Ghostscript. -When ``--force-ocr`` is used, all pages are rasterized and reconverted -to PDF, which could remove malware in embedded images. - -OCRmyPDF should be relatively safe to use in a trusted intranet, with -some considerations: - -Limiting CPU usage ------------------- - -OCRmyPDF will attempt to use all available CPUs and storage, so -executing ``nice ocrmypdf`` or limiting the number of jobs with the -``-j`` argument may ensure the server remains available. Another option -would be to run OCRmyPDF jobs inside a Docker container, a virtual machine, -or a cloud instance, which can impose its own limits on CPU usage and be -terminated "from orbit" if it fails to complete. - -Temporary storage requirements ------------------------------- - -OCRmyPDF will use a large amount of temporary storage for its work, -proportional to the total number of pixels needed to rasterize the PDF. -The raster image of a 8.5×11" color page at 300 DPI takes 25 MB -uncompressed; OCRmyPDF saves its intermediates as PNG, but that still -means it requires about 9 MB per intermediate based on average -compression ratios. Multiple intermediates per page are also required, -depending on the command line given. A rule of thumb would be to allow -100 MB of temporary storage per page in a file – meaning that a small -cloud servers or small VM partitions should be provisioned with plenty -of extra space, if say, a 500 page file might be sent. - -To check temporary storage usage on actual files, run -``ocrmypdf -k ...`` which will preserve and print the path to temporary -storage when the job is done. - -To change where temporary files are stored, change the ``TMPDIR`` -environment variable for ocrmypdf's environment. (Python's -``tempfile.gettempdir()`` returns the root directory in which temporary -files will be stored.) For example, one could redirect ``TMPDIR`` to a -large RAM disk to avoid wear on HDD/SSD and potentially improve -performance. On Amazon Web Services, ``TMPDIR`` can be set to `empheral -storage `__. - -Timeouts --------- - -To prevent excessively long OCR jobs consider setting -``--tesseract-timeout`` and/or ``--skip-big`` arguments. ``--skip-big`` -is particularly helpful if your PDFs include documents such as reports -on standard page sizes with large images attached - often large images -are not worth OCR'ing anyway. - -Commercial alternatives ------------------------ - -The author also provides professional services that include OCR and -building databases around PDFs, and is happy to provide consultation. - -Abbyy Cloud OCR is viable commercial alternative with a web services -API. Amazon Textract, Google Cloud Vision, and Microsoft Azure -Computer Vision provide advanced OCR but have less PDF rendering capability. - Password protection, digital signatures and certification ========================================================= From d0d49ce9895d7a4cf8009666b3a92979a083ffff Mon Sep 17 00:00:00 2001 From: "James R. Barlow" Date: Mon, 12 Jun 2023 01:00:01 -0700 Subject: [PATCH 2/2] docs: Prefer long arguments --- docs/batch.rst | 30 +++++++++++++++--------------- docs/languages.rst | 12 ++++++------ 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/batch.rst b/docs/batch.rst index 07443244..3236d460 100644 --- a/docs/batch.rst +++ b/docs/batch.rst @@ -151,14 +151,14 @@ The watcher service is included in the OCRmyPDF Docker image. To run it: .. code-block:: bash docker run \ - -v :/input \ - -v :/output \ - -v :/archive \ - -e OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1 \ - -e OCR_ON_SUCCESS_ARCHIVE=1 \ - -e OCR_DESKEW=1 \ - -e PYTHONUNBUFFERED=1 \ - -it --entrypoint python3 \ + --volume :/input \ + --volume :/output \ + --volume :/archive \ + --env OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1 \ + --env OCR_ON_SUCCESS_ARCHIVE=1 \ + --env OCR_DESKEW=1 \ + --env PYTHONUNBUFFERED=1 \ + --interactive --tty --entrypoint python3 \ jbarlow83/ocrmypdf \ watcher.py @@ -170,13 +170,13 @@ original to ``/archive``. The parameters to this image are: :header: "Parameter", "Description" :widths: 50, 50 - "``-v :/input``", "Files placed in this location will be OCRed" - "``-v :/output``", "This is where OCRed files will be stored" - "``-v :/archive``", "Archive processed originals here" - "``-e OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1``", "Define environment variable ``OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1`` to place files in the output in ``{output}/{year}/{month}/{filename}``" - "``-e OCR_ON_SUCCESS_ARCHIVE=1``", "Define environment variable ``OCR_ON_SUCCESS_ARCHIVE`` to move processed originals" - "``-e OCR_DESKEW=1``", "Define environment variable ``OCR_DESKEW`` to apply deskew to crooked input PDFs" - "``-e PYTHONBUFFERED=1``", "This will force ``STDOUT`` to be unbuffered and allow you to see messages in docker logs" + "``--volume :/input``", "Files placed in this location will be OCRed" + "``--volume :/output``", "This is where OCRed files will be stored" + "``--volume :/archive``", "Archive processed originals here" + "``--env OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1``", "Define environment variable ``OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1`` to place files in the output in ``{output}/{year}/{month}/{filename}``" + "``--env OCR_ON_SUCCESS_ARCHIVE=1``", "Define environment variable ``OCR_ON_SUCCESS_ARCHIVE`` to move processed originals" + "``--env OCR_DESKEW=1``", "Define environment variable ``OCR_DESKEW`` to apply deskew to crooked input PDFs" + "``--env PYTHONBUFFERED=1``", "This will force ``STDOUT`` to be unbuffered and allow you to see messages in docker logs" This service relies on polling to check for changes to the filesystem. It may not be suitable for some environments, such as filesystems shared on a diff --git a/docs/languages.rst b/docs/languages.rst index ded61e11..ff9467c9 100644 --- a/docs/languages.rst +++ b/docs/languages.rst @@ -70,13 +70,13 @@ This enables these languages for all packages (e.g. including aspell). # Display a list of all Tesseract language packs equery uses app-text/tessdata_fast - + # Add English and German language support for Tesseract only echo 'app-text/tessdata_fast l10n_de l10n_en' >> /etc/portage/package.use - + # Add global English and German language support (the `l10n_` from equery has to be omitted) echo L10N="de en" >> /etc/portage/make.conf - + # update system to reflect changed USE flags emerge --update --deep --newuse @world @@ -101,7 +101,7 @@ derived Docker image as Windows users ============= -The Tesseract installer provided by Chocolatey currently includes only English language. -To install other languages, download the respective language pack (``.traineddata`` file) -from https://github.com/tesseract-ocr/tessdata/ and place it in +The Tesseract installer provided by Chocolatey currently includes only English language. +To install other languages, download the respective language pack (``.traineddata`` file) +from https://github.com/tesseract-ocr/tessdata/ and place it in ``C:\\Program Files\\Tesseract-OCR\\tessdata`` (or wherever Tesseract OCR is installed).