Micro
Tools
Home
All Tools
About
Menu
Home
All Tools
About
SEO Tools
Robots.txt Generator
Robots.txt Generator
Create robots.txt files to control search engine crawling
Robots.txt Generator
Create robots.txt files to control search engine crawling
Allow all web crawlers to access all content
Sitemap URL
Disallow Paths (one per line)
Crawl Delay (seconds)
Custom Rules
Generated Robots.txt
Copy
Download
# Robots.txt generated by Micro Tools User-agent: * Allow: /
Common Robots.txt Patterns
Block admin areas:
/admin/, /wp-admin/
Block search results:
/search/, /?s=
Block private files:
/private/, /temp/
Block duplicate content:
/tag/, /category/
Important Notes
• Place robots.txt in your website's root directory
• Test your robots.txt using Google Search Console
• Robots.txt is publicly accessible - don't use it to hide sensitive content
• Some crawlers may ignore robots.txt directives