Also assert that the response is the actual file contents instead of mocking the method call
New route to /robots.txt and some specific logic to switch on if the subdomain is staging or not. Staging will not allow any crawlers at all, production does not have any current limitations