robots.txt tells crawlers where they may go, an XML sitemap lists the pages you want found, and llms.txt is an optional file that summarises your site for AI tools. The first two matter for every website; llms.txt is new and not required by Google.
What is robots.txt?
A text file at /robots.txt with rules for bots. A typical small website allows everything and points to the sitemap:
User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
One wrong line — Disallow: / — blocks the whole site. robots.txt controls crawling, not indexing: to keep a page out of results use a noindex tag instead.
What is an XML sitemap?
A file (usually /sitemap.xml) listing the important URLs with their last-modified dates. Submit it in Google Search Console. Include only pages that return 200 and that you want indexed.
What is llms.txt?
A proposed Markdown file at /llms.txt that gives AI tools a short description of the site and links to the key pages. Google has said it doesn’t use it, and AI providers have not confirmed they rely on it — but it is cheap to add and harmless.
Should I allow AI crawlers?
If you want to be cited in AI answers, allow the AI search bots (for example OAI-SearchBot, PerplexityBot, Claude-SearchBot). Training bots (GPTBot, ClaudeBot, Google-Extended) are a separate choice.
How do I check them?
- Open
yoursite.com/robots.txtandyoursite.com/sitemap.xmlin the browser. - Use Search Console → Settings → robots.txt report, and Sitemaps.
- Run the Technical SEO and AEO & GEO tools in the course.
Want to learn by doing? Message us on WhatsApp at +91 73548 20386 — you get your login the same day.
