New route to /robots.txt and some specific logic to switch on if the subdomain is staging or not. Staging will not allow any crawlers at all, production does not have any current limitations