A "robots.txt" text file is basically just a simple text file made with any text editor such as NotePad
It is located in the Websites root directory that specifies for the Search Engine's Crawlers and Spiders or Bots what website pages and files you want or do not want them to crawl or index. Usually website owners strive to be noticed by search engines, but there are cases when it's not needed or wanted. For instance, if you store sensitive data or you want to save bandwidth by not indexing heavy pages with images
When a crawler accesses a website, it requests for a file named "/robots.txt". If such a file is found, the crawler checks it for the website indexation instructions
NOTE: there can be only one robots.txt file for the website. Robots.txt file for addon domains or sub-domains need to be placed in the corresponding document root
The robots txt file is created in your web-site's root folder "yourwebsite.com/robot.txt"
You can use any text editor to make or edit a robots text file
The basic syntax for the robots txt file
>> User-agent: [The name of the robot for which you are writing these rules]
>> Disallow: [page, folder or path where you want to hide]
>> Allow: [page, folder or path where you want to unhide]
If you want to allow crawl everything, then use this code (All Search Engine)
User-agent: *
Disallow:
If you want to Disallow to crawl everything (All search Engine)
User-agent: *
Disallow: /
If you want to Disallow for the specific folder (All search Engine)
User-agent: *
Disallow: /folder name/
If you want to Disallow for the specific file (All search Engine)
User-agent: *
Disallow: /filename.html
If you want to Disallow for a folder but allow the crawling of one file in that folder (All search Engine)
User-agent: *
Disallow: /folderxyz/
Allow: /folderxyz/anyfile.html
Allow only one specific robot access in website
User-agent: *
Disallow: /
User-agent: Googlebot
Disallow:
To exclude a single robot
User-agent: BadBotName
Disallow: /
If you want to allow for the sitemap file crawling
User-agent: *
Sitemap: http://www.yourwebsite.com/sitemap.xml
WordPress Robots txt File
User-agent: *
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /readme.html
Disallow: /xmlrpc.php
Allow: /wp-content/uploads/
Sitemap: https://yourwebsite.com/sitemap.xml
Tip – Do not disallow files in the robots txt file that you want Bots to crawl or especially to hide, By doing this you are telling everyone about those files, We would recommend putting them inside a folder and Hide that folder
Other common mistakes are typos - misspelled directories, user-agents, missing colons after User-agent and Disallow, etc. When your robots.txt files get more and more complicated, and it's easy for an error to slip in
Tweet Share Pin Tumble Email