Jake Yesbeck 507e5a0ebc Robots controller and custom robots.txt files ...

New route to /robots.txt and some specific logic to switch
on if the subdomain is staging or not.

Staging will not allow any crawlers at all, production does not
have any current limitations

2015-05-04 23:11:41 -07:00

..

404.html

…

422.html

…

500.html

…

favicon.ico

…