robots.txt

robots.txt File

A text file that tells web crawlers which pages they can and can't access. Without an Allow rule for GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, these AI crawlers can't index your site.

mytrend.ai · Last updated: 16 July 2026

Frequently Asked Questions

Where should robots.txt be placed?
At the root of your domain, e.g. https://example.com/robots.txt — it must be at that exact path for crawlers to find it.

Ready to put robots.txt into practice?

mytrend.ai automatically builds every GEO component for your business — Schema.org, llms.txt, FAQPage.

Free AI Test Back to Glossary
© 2026 mytrend.ai & TomrisBT Bilişim — All rights reserved. Powered by TomrisBT