e2c0867cb0
* Implement media thumbnail previews for file attributes
* Replace PyMuPDF with pdf2image for PDF thumbnail generation
* create utility file for thumbnail generation
* Refactor thumbnail generation logic
* Add backfill command for missing file thumbnails and update docker_entrypoint script
* Added poppler dependency (needed for pdf2img) to dev setup docs.
* Fixed file card grid alignment with uniform thumbnail heights.
* Fix thumbnail backfill command test: filename desync via direct path
The test saved a file at a hardcoded source_path, then created an
EntityAttribute with file_value=source_path. AttributeModel.save() calls
generate_unique_filename() to add a timestamp suffix to file_value.name,
which desyncs the attribute's stored path from the on-disk path —
file_value.size then raises FileNotFoundError inside the backfill command.
Pass the file bytes via ContentFile(..., name='...') instead. The model
now owns the full filename-generation lifecycle and the on-disk and
in-database paths stay in sync.
Applied symmetrically to the dry-run test for consistency, even though
its assertions happened to pass without seeing the underlying mismatch.
* Switch thumbnail backfill from Docker entrypoint to render-time lazy gen
The Docker entrypoint's call to ``backfill_attribute_thumbnails`` had
unbounded startup cost for users with many file attributes, paid
overhead on every container restart even when nothing needed
generation, and was a permanent ongoing entry point for what's really
a transient migration need.
Replace with synchronous lazy generation triggered the first time a
file card is rendered for an attribute that lacks a thumbnail. Each
file attribute pays the generation cost once, on first view, spread
across actual usage instead of forced into startup.
- ``package/docker_entrypoint.sh``: drop the backfill invocation.
- ``AttributeModel.ensure_thumbnail()``: model method that generates
if missing, no-op otherwise.
- ``{% ensure_thumbnail attribute %}``: template tag wrapper for
call-from-template ergonomics.
- ``file_card.html``: invoke the tag once at the top, before any
``has_thumbnail`` / ``thumbnail_url`` reads.
The ``backfill_attribute_thumbnails`` management command stays
available for users who want to warm the cache explicitly. Its
counters remain accurate because the new render-time path is
isolated to ``ensure_thumbnail()`` — ``has_thumbnail`` stays a
pure existence check.
* Harden PDF thumbnail generation against pathological input
The 20MB pre-render byte cap was the only protection against expensive
thumbnail generation, but PDF rendering cost doesn't correlate well
with file size — a small PDF can produce a multi-GB pixel buffer at
pdf2image's 200-DPI default, and a crafted PDF can hang the underlying
pdftoppm subprocess indefinitely.
Three protections, all on the PDF path:
- ``size=(640, 640)`` passed to ``convert_from_bytes()``: caps the
rasterized output dimensions, keeping pre-resize memory bounded.
640 gives roughly a 2x oversample of the 320x320 thumbnail for
LANCZOS quality without runaway buffers.
- ``timeout=30`` threaded through to the pdftoppm subprocess so
pathological content can't hang generation.
- 10MB per-mime-type source-byte cap for PDFs, separate from the
existing 20MB cap for images.
Constants exposed as class attributes on ``AttributeThumbnail`` so
they're discoverable and tunable in one place.
* Install poppler-utils in CI for pdf2image-dependent tests
pdf2image shells out to pdftoppm (from the poppler-utils package) to
rasterize PDFs. The Docker image already installs poppler-utils, but
the GitHub Actions workflow runner did not, so
test_generate_thumbnail_best_effort_pdf_success passes locally and in
Docker but failed in CI on the ``assertTrue(generated)`` check.
Add an apt-get install step for poppler-utils so CI exercises the
same PDF-rendering surface as production.
* Add tests for AttributeModel.ensure_thumbnail() lazy-generation hook
ensure_thumbnail() wraps AttributeThumbnail.generate_thumbnail_best_effort()
with three branches that aren't transitively covered by the existing
generator tests or by the backfill command tests:
- supported file, thumbnail missing -> generates
- thumbnail already present -> short-circuits before instantiating the
generator (verified by mocking AttributeThumbnail)
- unsupported mime type -> short-circuits at the supports check
Co-located with the existing generate_thumbnail_best_effort_* tests
so a future reader finds the lazy hook's coverage in the same place
as the generator's direct coverage.
---------
Co-authored-by: Anthony Cassandra <github@cassandra.org>
7.3 KiB
Development Setup (one-time setup for external devevelopment)
Quick Setup (Recommended)
For a streamlined setup experience, we provide an automated setup script that handles most of the configuration for you.
Prerequisites
- Fork the repository on GitHub (see manual steps below for details)
- Clone your fork locally
- Ensure Python 3.11 is installed
Automated Setup
After cloning your fork, run the setup script from the project root:
cd home-information
./deploy/dev-setup.sh
This script will:
- Configure git settings and remotes
- Generate environment variables
- Create and activate a Python virtual environment
- Install all required packages
- Initialize the database
- Run validation tests
The script is interactive and will prompt you for necessary information. It's safe to run multiple times if needed.
Manual Setup (Alternative)
If you prefer to set up manually or need more control over the process, follow these detailed steps (details below):
git clone https://github.com/${YOURUSERNAME}/home-information.git
cd home-information
make env-build-dev
python3.11 -m venv venv
. ./init-env-dev.sh
pip install -r src/hi/requirements/development.txt
cd src
./manage.py check
./manage.py migrate
./manage.py hi_createsuperuser
./manage.py hi_creategroups
./manage.py runserver
Fork the Repository
- Sign into your GitHub account (required).
- Go to the main repository on GitHub: https://github.com/cassandra/home-information
- Click the "Fork" button in the upper-right corner. (You will be forking from the
stagingbranch.) - This creates a copy of the repository in the your GitHub account (keep same name if you can for simplicity).
- The forked repo will be located at https://github.com/${YOURUSERNAME}/home-information.git (if you kept the same repo name).
Local Repository Setup
Decide where you want to download the code to (adjust the following):
PROJ_DIR="${HOME}/proj"
mkdir -p $PROJ_DIR
cd $PROJ_DIR
Clone your fork to your local development environment:
git clone https://github.com/${YOURUSERNAME}/home-information.git
# Or use the SSH URL if you have SSH keys set up:
git clone git@github.com:${YOURUSERNAME}/home-information.git
Now change into that directory and configure the repo including adding the source as the "upstream" target:
cd home-information
git config --global user.name "${YOUR_NAME}"
git config --global user.email "${YOUR_EMAIL}"
git remote add upstream https://github.com/cassandra/home-information.git
Your "origin" should already be pointing to your forked repository, but check this and the "upstream" settings:
git remote -v
# Expect
origin https://github.com/${YOURUSERNAME}/home-information.git (fetch)
origin https://github.com/${YOURUSERNAME}/home-information.git (push)
upstream https://github.com/cassandra/home-information.git (fetch)
upstream https://github.com/cassandra/home-information.git (push)
If your origin is not set properly, re-verify after setting with:
git remote add origin git@github.com:${YOURUSERNAME}/home-information.git
# If no SSH keys were added to GitHub, you'll need this instead:
git remote set-url origin https://github.com/${YOURUSERNAME}/home-information.git
Environment Setup
Generate the environment variable file and review with the command below. The file will contain sensitive secrets and are stored in a .private directory. Also note that administrative credentials created during this next step.
make env-build-dev
This generates an environment variable file that we "source" before running:
$PROJ_DIR/.private/env/development.sh
This .private directory and its files should not be checked into the code repository. There is an existing .gitignore entry to prevent this. The adminstrative credentials generated can also be seen in that file.
Next, create the Python virtual environment.
cd $PROJ_DIR
python3.11 -m venv venv
Now source the environment and virtual environment with this convenience script:
. ./init-env-dev.sh
In the future, just source'ing this script is all you need to set things up for development (virtual env and env vars).
Next, install all the app's required Python packages (make sure you are in the virtual env).
pip install -r src/hi/requirements/development.txt
App and Database Initializations
Initialize the database and add the admin users and groups.
cd $PROJ_DIR/src
./manage.py check
./manage.py migrate
./manage.py hi_createsuperuser
./manage.py hi_creategroups
It is a good idea to run the tests to validate that you can and that the installation seem fine.
cd $PROJ_DIR/src
./manage.py test
Running
Ensure that a local Redis server is running (see the Dependencies page for installation instructions). Note: Redis is only a manual dependency for local development — Docker deployments bundle it automatically. Then:
cd $PROJ_DIR/src
./manage.py runserver
Then, visit: http://127.0.0.1:8411 to access the app.
Daily Development Commands
Once your environment is set up, these are the common commands for daily development work:
Environment Activation
# Daily development setup (run this first each day)
. ./init-env-dev.sh # Sources virtual env and environment variables
Django Management
cd $PROJ_DIR/src
# Database operations
./manage.py migrate
./manage.py makemigrations
./manage.py check
# Testing
./manage.py test # Run all tests
./manage.py test weather.tests # Run specific app tests
# User management
./manage.py hi_createsuperuser
./manage.py hi_creategroups
# Development server
./manage.py runserver # Runs on http://127.0.0.1:8411
Code Quality
# Linting and formatting (from development.txt requirements)
black src/ # Format code
flake8 --config=src/.flake8-ci src/ # Lint code with CI configuration
autopep8 --in-place --recursive src/ # Auto-format
Docker Operations
# Build and run in containers
make docker-build
make docker-run-fg # Foreground
make docker-run # Background
make docker-stop
Documentation Preview
To preview GitHub-flavored markdown locally before committing, we recommend VS Code with the Markdown Preview Github Styling extension. It renders markdown with GitHub's CSS locally — no API calls or additional dependencies needed.
If you already use VS Code for this project, install the extension and open
any .md file with Ctrl+Shift+V to preview (or Ctrl+K V for side-by-side).
If you use a different editor for development, you can open this project in a separate VS Code window just for markdown editing:
code -n $PROJ_DIR
Getting Started
If you want to familiarize yourself with how to use the app before diving into the code, see the Getting Started Page.
A look through these docs might also be a good starting point: