---
title: "What is robots.txt ?"  
description: "What is robots.txt ?"  
author: "Anonymous User"  
published: 2019-08-31  
updated: 2019-08-31  
canonical: https://www.mindstick.com/forum/95315/what-is-robots-txt  
category: "search engine marketing"  
tags: ["seo", "smo"]  
reading_time: 2 minutes  

---

# What is robots.txt ?

[Search Engine Optimization](https://www.mindstick.com/articles/12862/best-search-engine-optimization-techniques-for-your-seo-ranking)

## Replies

### Reply by Anurag Dwivedi

Robot.txt is a file where you put the important content under the tag and invite the google bot to crawl the content and give it a top priority. Its primary role is to manage crawler traffic to your site and usually to keep a page off google depending on the file type.

if you want to block your page from search results use another method such as password protection or a non-index directive.

Most of us know that robot.txt has the only advantage but it has limitation too :

robot.txt is not supported by other search engines.

a robtted page can still be indexed if linked to from other sites

### Reply by Sport11

Robots.txt is one way of telling the [Search](https://www.mindstick.com/articles/65368/best-smo-services-company-in-hyderabad-improve-search-rankings) [Engine](https://www.mindstick.com/interview/33775/what-is-a-javascript-engine) Bots about the web pages on your website which you do not want them to visit.

### Reply by Neel Patel

Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website.

### Reply by Anonymous User

The **Robots.txt** is one way of telling the [Search Engine](https://www.mindstick.com/articles/12941/link-building-and-search-engine-rankings-how-are-they-connected) Bots about the web pages on your website which you do not want them to visit.

A Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.

IF we want to block all search engine robots from crawling our website, just put the following code:

User-agent: *

Disallow: /

IF we want to block Google from crawling our website, just put the following code:

User-agent: Googlebot

Disallow: /

This is important to decide the location of Robots.txt very carefully, or else errors might occur while displaying the website.


---

Original Source: https://www.mindstick.com/forum/95315/what-is-robots-txt

Copyright © MindStick Software Pvt. Ltd. This Markdown version is provided for developers, AI systems, and offline reading.