diff options
| author | Eltrac <hi@guhub.cn> | 2025-05-17 09:09:36 +0800 |
|---|---|---|
| committer | Eltrac <hi@guhub.cn> | 2025-05-17 09:09:36 +0800 |
| commit | ee8f25324164c845a7767d93b31ebd48571a3607 (patch) | |
| tree | c32ebdb1f1f2ae626d5ef7f4c23ee81ddc7d3773 /static | |
| parent | 0f4438046ed60e040afdfc3276d883d85dc2092c (diff) | |
chore: add robots.txt
Former-commit-id: 9267f11dbdbb1eb073865f06babfd0909837fe6f
Diffstat (limited to 'static')
| -rw-r--r-- | static/robots.txt | 61 |
1 files changed, 61 insertions, 0 deletions
diff --git a/static/robots.txt b/static/robots.txt new file mode 100644 index 00000000..cc5537d4 --- /dev/null +++ b/static/robots.txt @@ -0,0 +1,61 @@ +User-agent: Googlebot +Allow: / + +User-agent: Bingbot +Allow: / + +User-agent: DuckDuckBot +Allow: / + +User-agent: Baiduspider +Allow: / + +User-agent: Yandex +Allow: / + +# Allow link preview bots +User-agent: TelegramBot +Allow: / + +User-agent: Discordbot +Allow: / + +User-agent: Twitterbot +Allow: / + +User-agent: facebookexternalhit +Allow: / + +# Block known AI scrapers +User-agent: GPTBot +Disallow: / + +User-agent: ChatGPT-User +Disallow: / + +User-agent: CCBot +Disallow: / + +User-agent: ClaudeBot +Disallow: / + +User-agent: anthropic-ai +Disallow: / + +User-agent: Bytespider +Disallow: / + +User-agent: Amazonbot +Disallow: / + +User-agent: omgili +Disallow: / + +User-agent: Diffbot +Disallow: / + +# Disallow all others +User-agent: * +Disallow: / + +Sitemap: https://www.geedea.pro/sitemap.xml |