mirror/growstuff
1
0
Fork 0
mirror of https://github.com/Growstuff/growstuff.git synced 2026-05-12 01:36:26 -04:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #4592 from Growstuff/CloCkWeRX-patch-2

Browse Source 
Ban Semrush
This commit is contained in:
Daniel O'Connor
2026-04-27 17:19:53 +09:30
committed by GitHub
parent 66cc98051f a2bb6c7162
commit fc4276cacf
1 changed files with 20 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
public/robots.txt
View File

@@ -123,13 +123,27 @@ Disallow: /
User-agent: WebReaper
Disallow: /
# Per their statement, semrushbot respects crawl-delay directives
# We want them to overall stay within reasonable request rates to
# the backend (20 rps); keeping in mind that the crawl-delay will
# be applied by site and not globally by the bot, 5 seconds seem
# like a reasonable approximation
# Semrush seem to crawl everything.
User-agent: SemrushBot
Crawl-delay: 5
Disallow: /
User-agent: SiteAuditBot
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: SemrushBot-SI
Disallow: /
User-agent: SemrushBot-SWA
Disallow: /
User-agent: SplitSignalBot
Disallow: /
User-agent: SemrushBot-OCOB
Disallow: /
#
# Friendly, low-speed bots are welcome viewing pages, but not