What is robots.txt ?

Question

Anurag Dwivedi · Answer

Robot.txt is a file where you put the important content under the tag and invite the google bot to crawl the content and give it a top priority. Its primary role is to manage crawler traffic to your site and usually to keep a page off google depending on the file type.

if you want to block your page from search results use another method such as password protection or a non-index directive.

Most of us know that robot.txt has the only advantage but it has limitation too :

robot.txt is not supported by other search engines.

a robtted page can still be indexed if linked to from other sites

Sport11 · Answer

Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.

Neel Patel · Answer

Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website.

Anonymous User · Answer

The Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit. A Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display. IF we want to block all search engine robots from crawling our website, just put the following code: User-agent: * Disallow: / IF we want to block Google from crawling our website, just put the following code: User-agent: Googlebot Disallow: / This is important to decide the location of Robots.txt very carefully, or else errors might occur while displaying the website.

forum

What is robots.txt ?

Can you answer this question?

4 Answers