4 Commits

Author	SHA1	Message	Date
Jake Yesbeck	0d55b54371	Use a constant for the default robots.txt filename	2015-05-05 09:05:00 -07:00
Jake Yesbeck	81ce6ed8a3	Make sure file exists before attempting to render it	2015-05-05 08:58:33 -07:00
Jake Yesbeck	0f8d1e7db1	Change the render to render the file directly ... Also assert that the response is the actual file contents instead of mocking the method call	2015-05-04 23:37:40 -07:00
Jake Yesbeck	507e5a0ebc	Robots controller and custom robots.txt files ... New route to /robots.txt and some specific logic to switch on if the subdomain is staging or not. Staging will not allow any crawlers at all, production does not have any current limitations	2015-05-04 23:11:41 -07:00