How to create SEO friendly robots.txt file?

Question

Amrith Chandran · Answer

The robots.txt file helps search engines understand which parts of your website to crawl and index. Creating an SEO-friendly robots.txt file ensures that search engines can access important pages while blocking unnecessary pages. Here's how to create it,

Key Instruction

User-Agent:
Specifies the search engine (e.g., Googlebot) to which the rule applies. Use * to apply to all crawlers.

User-agent: *

Disallow: Tells crawlers not to visit certain pages or directories.

Disallow: /private/

Allow: Overrides the Disallow directive for specific pages or files under a disallowed directory.

Allow: /private/page1.html

Sitemap: Points to your sitemap location, helping search engines find all pages.

Sitemap: https://www.example.com/sitemap.xml

Best Practices

Don't block important pages: Make sure pages with key content (e.g., blog posts, product pages) aren't blocked.

Block irrelevant or duplicate pages: Block admin, login, and other non-essential pages to prevent indexing.

Test it: Use a tool like Google Search Console's robots.txt checker to make sure your file is correct

Example of a Simple SEO-Friendly Robots.txt

User-agent: *
Disallow: /admin/
Disallow: /login/
Allow: /public/
Sitemap: https://www.example.com/sitemap.xml

Also, Read: What is canonical tag in SEO ?

interview

How to create SEO friendly robots.txt file?

Can you answer this question?

1 Answers