- Configure Allow2Ban in Rack::Attack to ban IPs requesting >500 pages per day for 1 week (7 days).
- Add honeypot route /dont-crawl-me disallowed in robots.txt and configure Fail2Ban in Rack::Attack to ban IPs visiting it for 7 days upon 1 hit.
- Update PhotosController#index to raise ActiveRecord::RecordNotFound when page parameter is out of bounds, returning 404 Not Found to crawlers instead of 200 OK.
- Add request and controller specs for Rack::Attack rules and pagination 404 responses.
Co-authored-by: CloCkWeRX <365751+CloCkWeRX@users.noreply.github.com>
Adds the `sitemap_generator` gem to generate a sitemap.
- Includes static routes and dynamic routes for active crops, plantings, seeds, photos, posts, and members.
- Implements a caching mechanism to ensure the sitemap is generated no more frequently than every 72 hours.
- Updates `robots.txt` to point to the new sitemap.
- Includes a unit test for the caching Rake task.
New route to /robots.txt and some specific logic to switch
on if the subdomain is staging or not.
Staging will not allow any crawlers at all, production does not
have any current limitations
