The robots.Txt file is a new function that was recently brought to the Google Search Console. This document provides you with data approximately how Google is crawling and indexing your website. 

The report is split into 3 sections:

Crawled URLs: This section shows you a listing of all the URLs on your internet site that Google has crawled.

Not Crawled URLs: This section suggests you a listing of all the URLs on your website that Google has not crawled.

Blocked URLs: This segment shows you a list of all of the URLs for your internet site that Google has been blocked from crawling.

The robots.Txt record is a valuable tool that allows you to improve your internet site's SEO. By the usage of the document, you may become aware of any URLs which might be being blocked from crawling and make certain that they are reachable to Google. You can also use the record to perceive any URLs which might be being crawled too frequently and take steps to reduce the variety of times they're crawled.

How to get entry to the robots.Txt document

The robots.Txt record is to be had in the Google Search Console. To get right of entry to it, comply with these steps:

• Sign in to the Google Search Console.
• Click at the "Search Console" hyperlink within the pinnacle navigation bar.
• Click on the "Robots.Txt" link inside the left-hand navigation bar.

What statistics are included in the robots.Txt report?

Once you have accessed the robots.Txt file, you'll see a listing of all of the URLs in your internet site that Google has crawled. You can even see a list of all the URLs on your website that Google has no longer crawled.

For every URL, you may see the subsequent statistics:

• Status: The popularity of the URL. This can be "Crawled," "Not Crawled," or "Blocked."
• Crawl Date: The date that the URL changed into remaining crawled through Google.
• Last Fetch: The date that the URL turned into final fetched by way of Google.
• Block Reason: The motive why the URL is blocked, if it's miles blocked.

How to apply the robots.Txt report

The robots.Txt report is a treasured device that lets you improve your internet site's search engine marketing. Here are a few approaches that you could use the report:

Identify URLs which are being blocked from crawling: If you notice that any URLs on your website are being blocked from crawling, you can make sure that they are reachable to Google. You can try this through adding them for your robots.Txt document.

Identify URLs which are being crawled too regularly: If you see that any URLs on your internet site are being crawled too often, you can take steps to lessen the variety of instances they are crawled. You can do that by means of including a move slowly-postpone directive to your robots.Txt report.

Monitor your website's search engine optimization: The robots.Txt file can help you to display your internet site's search engine optimization. You can use the file to tune how Google is crawling and indexing your internet site over time.

Conclusion

The robots.Txt file is a precious tool that assists you to improve your website's search engine marketing. By the usage of the document, you can pick out any URLs that are being blocked from crawling and make certain that they may be on hand to Google. You can also use the document to perceive any URLs which might be being crawled too often and take steps to lessen the wide variety of instances they're crawled.