Search Engine Optimization
What is robots.txt ?
2025
31-Aug-2019
Updated on 31-Aug-2019
Anurag Dwivedi
20-Nov-2020Robot.txt is a file where you put the important content under the tag and invite the google bot to crawl the content and give it a top priority. Its primary role is to manage crawler traffic to your site and usually to keep a page off google depending on the file type.
if you want to block your page from search results use another method such as password protection or a non-index directive.
Most of us know that robot.txt has the only advantage but it has limitation too :
robot.txt is not supported by other search engines.
a robtted page can still be indexed if linked to from other sites
Sport11
13-Oct-2019Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.
Neel Patel
30-Sep-2019Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website.
Anonymous User
31-Aug-2019The Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.
A Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.
IF we want to block all search engine robots from crawling our website, just put the following code:
User-agent: *
Disallow: /
IF we want to block Google from crawling our website, just put the following code:
User-agent: Googlebot
Disallow: /
This is important to decide the location of Robots.txt very carefully, or else errors might occur while displaying the website.