📚 Calibre-Web-Automated-Book-Downloader

An intuitive web interface for searching and requesting book downloads, designed to work seamlessly with Calibre-Web-Automated. This project streamlines the process of downloading books and preparing them for integration into your Calibre library.

✨ Features

• 🌐 User-friendly web interface for book search and download
• 🔄 Automated download to your specified ingest folder
• 🔌 Seamless integration with Calibre-Web-Automated
• 📖 Support for multiple book formats (epub, mobi, azw3, fb2, djvu, cbz, cbr)
• 🛡️ Cloudflare bypass capability for reliable downloads
• 🐳 Docker-based deployment for quick setup

🖼️ Screenshots

🚀 Quick Start

Prerequisites

• Docker
• Docker Compose
• A running instance of Calibre-Web-Automated (recommended)

Installation Steps

1. Get the docker-compose.yml:

   curl -O https://raw.githubusercontent.com/calibrain/calibre-web-automated-book-downloader/refs/heads/main/docker-compose.yml
   

2. Start the service:

   docker compose up -d
   

3. Access the web interface at http://localhost:8084

⚙️ Configuration

Environment Variables

Application Settings

Variable	Description	Default Value
FLASK_PORT	Web interface port	8084
FLASK_HOST	Web interface binding	0.0.0.0
DEBUG	Debug mode toggle	false
INGEST_DIR	Book download directory	/cwa-book-ingest
TZ	Container timezone	UTC
UID	Runtime user ID	1000
GID	Runtime group ID	100
CWA_DB_PATH	Calibre-Web's database	None
ENABLE_LOGGING	Enable log file	true
LOG_LEVEL	Log level to use	info
SESSION_COOKIE_SECURE	Secure cookie enforcement - Use for HTTPS connections only	false
CALIBRE_WEB_URL	Custom WebUI library link	None
BYPASS_WARMUP_ON_CONNECT	Warm up Cloudflare bypasser when first client connects	true

If you wish to enable authentication, you must set CWA_DB_PATH to point to Calibre-Web's app.db, in order to match the username and password.

Set CALIBRE_WEB_URL to your Calibre-Web / Booklore base URL. A ‘Go to library’ button will appear in the Web UI for quick access while downloading, and it also provides library access when CWA-BD is installed as a mobile PWA.

If logging is enabled, log folder default location is /var/log/cwa-book-downloader Available log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL. Higher levels show fewer messages.

Note that if using TOR, the TZ will be calculated automatically based on IP.

Download Settings

Variable	Description	Default Value
MAX_RETRY	Maximum retry attempts	3
DEFAULT_SLEEP	Retry delay (seconds)	5
MAIN_LOOP_SLEEP_TIME	Processing loop delay (seconds)	5
SUPPORTED_FORMATS	Supported book formats	epub,mobi,azw3,fb2,djvu,cbz,cbr
BOOK_LANGUAGE	Preferred language for books	en
AA_DONATOR_KEY	Optional Donator key for Anna's Archive fast download API	``
USE_BOOK_TITLE	Use book title as filename instead of ID	false
PRIORITIZE_WELIB	When downloading, download from WELIB first instead of AA	false
ALLOW_USE_WELIB	Allow usage of welib for downloading books if found there	true

If you change BOOK_LANGUAGE, you can add multiple comma separated languages, such as en,fr,ru etc.

Use the following environment variables to set specific folders in which to download different content types (Book, Magazine, Comic, etc.):

Variable	Description	Default Value
INGEST_DIR_BOOK_FICTION	Book (fiction) folder name	``
INGEST_DIR_BOOK_NON_FICTION	Book (non-fiction) folder name	``
INGEST_DIR_BOOK_UNKNOWN	Book (unknown) folder name	``
INGEST_DIR_MAGAZINE	Magazine folder name	``
INGEST_DIR_COMIC_BOOK	Comic book folder name	``
INGEST_DIR_AUDIOBOOK	Audiobook folder name	``
INGEST_DIR_STANDARDS_DOCUMENT	Standards document folder name	``
INGEST_DIR_MUSICAL_SCORE	Musical score folder name	``

If no specific path is set for a content type the default is INGEST_DIR.
Remember to map the specified paths to where your instance of Calibre-Web-Automated (CWA) will find them, e.g.:

volumes:
    - /tmp/data/calibre-web/comicbook-ingest:/cwa-comicbook-ingest

if INGEST_DIR_COMIC_BOOK=/cwa-comicbook-ingest and your CWA is configured to use /tmp/data/calibre-web/comicbook-ingest for comic books.

AA

Variable	Description	Default Value
AA_BASE_URL	Base URL of Annas-Archive (could be changed for a proxy)	https://annas-archive.org
USE_CF_BYPASS	Disable CF bypass and use alternative links instead	true

If you are a donator on AA, you can use your Key in AA_DONATOR_KEY to speed up downloads and bypass the wait times. If disabling the cloudflare bypass, you will be using alternative download hosts, such as libgen or z-lib, but they usually have a delay before getting the more recent books and their collection is not as big as aa's. But this setting should work for the majority of books.

Network Settings

Variable	Description	Default Value
AA_ADDITIONAL_URLS	Proxy URLs for AA (, separated)	``
HTTP_PROXY	HTTP proxy URL	``
HTTPS_PROXY	HTTPS proxy URL	``
CUSTOM_DNS	DNS configuration	auto
USE_DOH	Use DNS over HTTPS	false

Proxy Configuration

For proxy configuration, you can specify URLs in the following format:

# Basic proxy
HTTP_PROXY=http://proxy.example.com:8080
HTTPS_PROXY=http://proxy.example.com:8080

# Proxy with authentication
HTTP_PROXY=http://username:password@proxy.example.com:8080
HTTPS_PROXY=http://username:password@proxy.example.com:8080

DNS Configuration

The CUSTOM_DNS setting controls how DNS resolution works. By default, it is set to auto which provides automatic failover for reliable connectivity.

Auto Mode (Default)

When CUSTOM_DNS=auto, the application starts with your system's default DNS. If DNS resolution fails, it automatically rotates through alternative providers using DNS over HTTPS (DoH):

1. System DNS (initial)
2. Cloudflare (1.1.1.1)
3. Google (8.8.8.8)
4. Quad9 (9.9.9.9)
5. OpenDNS (208.67.222.222)

This automatic rotation helps bypass ISP-level blocks and DNS issues without any manual configuration.

Manual DNS Configuration

If you prefer to use a specific DNS configuration, you can override the auto behavior:

1. Preset DNS Providers: Use one of these predefined options:

   • google - Google DNS (8.8.8.8, 8.8.4.4)
   • quad9 - Quad9 DNS (9.9.9.9, 149.112.112.112)
   • cloudflare - Cloudflare DNS (1.1.1.1, 1.0.0.1)
   • opendns - OpenDNS (208.67.222.222, 208.67.220.220)

2. Custom DNS Servers: A comma-separated list of DNS server IP addresses

   • Example: 127.0.0.53,127.0.1.53 (useful for PiHole)
   • Supports both IPv4 and IPv6 addresses

When using preset providers, you can optionally enable DNS over HTTPS with USE_DOH=true:

CUSTOM_DNS=cloudflare
USE_DOH=true

Note: When using custom IP addresses, the USE_DOH flag is ignored since DoH requires a known provider endpoint.

Custom configuration

Variable	Description	Default Value
CUSTOM_SCRIPT	Path to an executable script that tuns after each download	``

If CUSTOM_SCRIPT is set, it will be executed after each successful download but before the file is moved to the ingest directory. This allows for custom processing like format conversion or validation.

The script is called with the full path of the downloaded file as its argument. Important notes:

• The script must preserve the original filename for proper processing
• The file can be modified or even deleted if needed
• The file will be moved to /cwa-book-ingest after the script execution (if not deleted)

You can specify these configuration in this format :

environment:
  - CUSTOM_SCRIPT=/scripts/process-book.sh

volumes:
  - local/scripts/custom_script.sh:/scripts/process-book.sh

Volume Configuration

volumes:
  - /your/local/path:/cwa-book-ingest
  - /cwa/config/path/app.db:/auth/app.db:ro

Note - If your library volume is on a cifs share, you will get a "database locked" error until you add nobrl to your mount line in your fstab file. e.g. //192.168.1.1/Books /media/books cifs credentials=.smbcredentials,uid=1000,gid=1000,iocharset=utf8,nobrl - See https://github.com/crocodilestick/Calibre-Web-Automated/issues/64#issuecomment-2712769777

Mount should align with your Calibre-Web-Automated ingest folder.

Variants:

🧅 Tor Variant

This application also offers a variant that routes all its traffic through the Tor network. This can be useful for enhanced privacy or bypassing network restrictions.

To use the Tor variant:

1. Get the Tor-specific docker-compose file:

   curl -O https://raw.githubusercontent.com/calibrain/calibre-web-automated-book-downloader/refs/heads/main/docker-compose.tor.yml
   

2. Start the service using this file:

   docker compose -f docker-compose.tor.yml up -d
   

Important Considerations for Tor:

• Capabilities: This variant requires the NET_ADMIN and NET_RAW Docker capabilities to configure iptables for transparent Tor proxying.
• Timezone: When running in Tor mode, the container will attempt to determine the timezone based on the Tor exit node's IP address and set it automatically. This will override the TZ environment variable if it is set.
• Network Settings: Custom DNS, DoH, and HTTP(S) proxy settings (CUSTOM_DNS, USE_DOH, HTTP_PROXY, HTTPS_PROXY) are ignored when using the Tor variant, as all traffic goes through Tor.

External Cloudflare resolver variant

This variant allows the application to use an external service to bypass Cloudflare protection, instead of relying on the built-in bypasser. This is useful if you already have a dedicated Cloudflare resolver (such as FlareSolverr or compatible services like ByParr) running elsewhere.

How it works:

• When enabled, all requests that require Cloudflare bypass are sent to your external resolver service.
• The application communicates with the resolver using its API.

Configuration

Variable	Description	Default Value
EXT_BYPASSER_URL	The full URL of your external resolver (required)
EXT_BYPASSER_PATH	API path for the resolver (usually /v1)	/v1
EXT_BYPASSER_TIMEOUT	Timeout for page loading (in milliseconds)	60000

Important

This feature follows the same configuration of the built-in Cloudflare bypasser, so you should turn on the USE_CF_BYPASS configuration to enable it.

To use the External Cloudflare resolver variant:

1. Get the extbp-specific docker-compose file:

   curl -O https://raw.githubusercontent.com/calibrain/calibre-web-automated-book-downloader/refs/heads/main/docker-compose.extbp.yml
   

2. Start the service using this file:

   docker compose -f docker-compose.extbp.yml up -d
   

Compatibility:

This feature is designed to work with any resolver that implements the FlareSolverr API schema, including ByParr and similar projects.

Internal vs External Bypasser

The internal bypasser (default) is custom-designed for this application's specific needs. It handles session management, cookie persistence, and retry logic optimized for book downloading workflows. For most users, this provides the most reliable experience out of the box.

The external bypasser is better suited if you:

• Already run FlareSolverr/ByParr for other services and want to consolidate
• Need to share bypass infrastructure across multiple applications
• Want to offload browser automation to a dedicated, more powerful container

If you're unsure which to use, start with the default internal bypasser.

🏗️ Architecture

The application consists of a Flask backend with a React-based frontend:

Backend

• Flask Application: Python-based backend (app.py, backend.py) providing REST API and WebSocket support
• Download Manager: Handles book search, download requests, and queue management (downloader.py, book_manager.py)
• Network Layer: Cloudflare bypass and proxy support (cloudflare_bypasser.py, network.py)

Frontend

• React + TypeScript: Modern web interface built with Vite (src/frontend)
• Real-time Updates: WebSocket integration for live download status
• Responsive UI: TailwindCSS-based design for mobile and desktop

For frontend development, use the provided Makefile:

make install  # Install dependencies
make dev      # Start development server
make build    # Build for production

If you run the docker compose file, the frontend will be built and served automatically. But if you run the frontend dev server it will supercede the docker compose frontend.

🏥 Health Monitoring

Built-in health checks monitor:

• Web interface availability
• Download service status
• Cloudflare bypass service connection

Checks run every 30 seconds with a 30-second timeout and 3 retries. You can enable by adding this to your compose :

HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
    CMD curl -s http://localhost:8084/api/status || exit 1

📝 Logging

Logs are available in:

• Container: /var/logs/cwa-book-downloader.log
• Docker logs: Access via docker logs

🤝 Contributing

Contributions are welcome! Feel free to submit a Pull Request.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

⚠️ Important Disclaimers

Copyright Notice

While this tool can access various sources including those that might contain copyrighted material (e.g., Anna's Archive), it is designed for legitimate use only. Users are responsible for:

• Ensuring they have the right to download requested materials
• Respecting copyright laws and intellectual property rights
• Using the tool in compliance with their local regulations

Duplicate Downloads Warning

Please note that the current version:

• Does not check for existing files in the download directory
• Does not verify if books already exist in your Calibre database
• Exercise caution when requesting multiple books to avoid duplicates

💬 Support

For issues or questions, please file an issue on the GitHub repository.