Robots.txt Generator
Select which pages search engines should index or ignore, add your sitemap URL, and generate a complete robots.txt file ready to upload to your website root.
Robots.txt Generator
Generate a robots.txt file for your website
Generated robots.txt contains User-agent directives (Allow/Disallow) and an optional Sitemap reference. Format: User-agent: [name] Disallow: [URL path] Allow: [URL path] Sitemap: [URL]
How Robots.txt Works
The robots.txt file is a text file at your website's root that tells search engine crawlers which pages or files the crawler can or cannot request from your site. It is part of the Robots Exclusion Protocol (REP).
**Syntax format**: ``` User-agent: Googlebot Disallow: /private/ Allow: /private/public-page.html
User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml ```
**Key directives**: - `User-agent`: Specifies which crawler the rule applies to. `*` applies to all crawlers. - `Disallow`: URL paths that the crawler should not request. Use `/` to disallow everything, `/private/` to disallow a subdirectory. - `Allow`: Explicitly permits access to a specific file or subpath even if a parent directory is disallowed. - `Sitemap`: References your XML sitemap file to help crawlers discover your pages.
**Important notes**: - robots.txt is a guideline, not a guarantee. Well-behaved crawlers follow it, but malicious bots may ignore it. - Use `Disallow: /` sparingly - this hides your entire site from search engines. - The `Allow` directive is needed when you disallow a directory but want to permit specific files within it (common for WordPress sites). - Keep the file size under 500 KB and under 500 lines (Google limit). - Update when your site structure changes.
- Generates valid, crawlable robots.txt syntax
- Visual checkbox selection of page types
- Optional sitemap inclusion
- One-click copy for upload
Worked Example
Example: Generating a robots.txt for a typical informational website:
Crawler: All (Googlebot + Bingbot + others)
Generated robots.txt: ``` User-agent: * Disallow: /admin/ Disallow: /search/ Disallow: /cart/ Disallow: /checkout/ Allow: /" Sitemap: https://example.com/sitemap.xml ```
**What this does**: - Allows all crawlers to index the home page and all content pages (") - Blocks admin directory (prevents indexing of dashboard/management pages) - Blocks search result pages (prevents thin content from appearing in search) - Blocks cart/checkout pages (prevents transactional pages from indexing) - Points crawlers to the sitemap for efficient discovery
**Typical SEO setup**: - Home page: Always allowed (Disallow nothing for /) - Admin/private: Always disallowed - Search results: Usually disallowed (thin content, pagination issues) - Checkout/cart: Always disallowed (privacy, transactional) - Sitemap: Always included if you have one generated
Common Robots.txt Patterns
- **Blog-only site**: Disallow /admin/, /search/, /track/ - **E-commerce site**: Disallow /cart/, /checkout/, /account/, /search/ - **Personal blog**: Disallow /search/, optionally Disallow /tag/ - **Development/staging**: Disallow / (entire site while developing) - **Multilingual site**: Keep /, disallow language duplicates if using hreflang correctly
Related Calculators
Percentage Calculator
Find percentages, calculate % of a value, and percentage increase/decrease.
Open calculator →Meta Tag Extractor
Extract title, description, and Open Graph tags from any URL.
Open calculator →Keyword Density Analyzer
Analyze keyword frequency and density in any text content.
Open calculator →Schema Markup Generator
Generate JSON-LD schema markup for articles, products, organizations, and more.
Open calculator →Frequently Asked Questions
Does robots.txt prevent a page from appearing in Google search?
No. robots.txt only controls crawling (whether Googlebot visits the page). To prevent a page from appearing in Google search results, use the `noindex` meta tag or HTTP header. You can block crawling with robots.txt AND noindex the page for maximum effect.
Can I have multiple User-agent blocks in one robots.txt?
Yes. Multiple User-agent blocks can exist in one file, each with their own Disallow/Allow directives. This is useful for targeting specific crawlers differently.
My robots.txt is blocking Google from indexing my home page. How do I fix it?
Ensure your robots.txt does not have `Disallow: /` or `Disallow: /` at the top. The home page should be allowed for all crawlers (either no Disallow rule, or an explicit `Allow: /` directive).
Looking for a different calculator?
CalculatorPlus has free tools across finance, construction, math, health and more — each one showing the formula and a worked example.