mirror/wayback-classic
1
0
Fork 0
mirror of https://github.com/ticky/wayback-classic.git synced 2025-12-23 22:17:56 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
develop
wayback-classic/robots.txt
Jessica Stokes ed8d826666 Just in case, block crawlers from all the dynamic content
2021-05-23 22:34:59 -07:00

4 lines
55 B
Plaintext
Raw Permalink Blame History

User-agent: *
Disallow: /cgi-bin/
Disallow: /README.md
Reference in New Issue View Git Blame Copy Permalink