Website Robots.txt Audit Checklist

A robots.txt file can guide crawlers away from utility or duplicate paths, but it is not a general privacy control and it does not replace page-level indexing decisions. Audit it against the pages the business actually needs found.

Check production access

Confirm that important service, location, article, product, inventory, and contact pages are not blocked by a broad rule. Make sure staging restrictions do not accidentally ship to production.

Review blocked paths

List search results, filters, parameters, admin paths, previews, duplicate utilities, scripts, styles, and assets. Block only what the site has a reason to exclude, and test whether a page still renders usefully.

Check syntax and sitemap reference

Validate directives, host behavior where relevant, line formatting, sitemap URL, and changes from the prior version. Keep the file accessible at the expected root path.

The sitemap audit and technical SEO audit provide companion checks.

Test after deployments

Review the live file after a migration, platform change, plugin update, or domain change. Compare crawler reports, search data, rendered pages, and customer paths before closing the task.

Do not use it for private information

Keep confidential material protected by authentication and authorization. A robots rule is a crawl instruction, not a security boundary.