Robots.txt Explained for Beginners
A robots.txt file is a small text file at the root of your website, at yourdomain.com/robots.txt. It gives instructions to crawlers, the programs that search engines use to read your site. It is simple, but a mistake in it can seriously damage your visibility.
What it does
Robots.txt tells well-behaved crawlers which areas of your site they should or should not request. It controls crawling, not indexing. That distinction matters: a page blocked in robots.txt can still appear in search results if other sites link to it, though without a description. To keep a page out of search, use a noindex tag on the page itself, and do not block it in robots.txt, because crawlers must be able to read the tag.
A basic example
User-agent: *
Disallow: /admin/
Sitemap: https://yourdomain.com/sitemap.xmlThis says all crawlers should avoid the admin area and tells them where to find your XML sitemap.
The key directives
- User-agent: names the crawler the rules apply to. An asterisk means all.
- Disallow: lists a path crawlers should not visit.
- Allow: permits access to a path inside a blocked area.
- Sitemap: points to your sitemap file.
What you might block
- Admin and login areas
- Internal search result pages
- Cart and checkout pages on a store
- Duplicate or filtered versions of pages
What not to block
Never block pages you want to rank. Also avoid blocking the CSS and JavaScript files your pages need, since search engines use them to render and understand your layout. A very common and costly mistake is leaving a "Disallow: /" line from a development site, which blocks the entire site from crawlers.
It is not security
Robots.txt is public and relies on crawlers choosing to obey it. Do not use it to hide sensitive information. Use proper access controls, as covered in keeping your website secure.
How to test it
Visit your robots.txt in a browser to confirm it loads. Use the testing and reporting tools in Search Console to see whether important pages are blocked.
Do you need one?
A small site with nothing to block can work without it, but having a simple file that references your sitemap is good practice.
Check after launch
When moving from a test site to the live one, double-check robots.txt. See our website maintenance checklist.
The bottom line
Keep robots.txt simple, block only what truly should be skipped, and review it after any major change.