diff options
| author | Eltrac <hi@guhub.cn> | 2025-11-01 09:34:26 +0800 |
|---|---|---|
| committer | Eltrac <hi@guhub.cn> | 2025-11-01 09:34:26 +0800 |
| commit | 3cde8adc23de7afe867f00419adf30334aa8e5a7 (patch) | |
| tree | 394a11a3ae7b4b725776292b28ad6eb1c9735a61 | |
| parent | 9196e7eb3eea0fc5ecc8ce1d275345f95301e629 (diff) | |
chore(SEO): allow archive.org bot
Former-commit-id: ac0d843edb68e9806277cafc3b68289934b03005
| -rw-r--r-- | static/robots.txt | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/static/robots.txt b/static/robots.txt index cc5537d4..6c7c5503 100644 --- a/static/robots.txt +++ b/static/robots.txt @@ -26,6 +26,9 @@ Allow: / User-agent: facebookexternalhit Allow: / +User-agent: archive.org_bot +Allow: / + # Block known AI scrapers User-agent: GPTBot Disallow: / |