What is the robots.txt?

The robots.txt is a text file on a web server that follows the Robots Exclusion Standard Protocol. It holds instructions for the web crawlers that scan a website in order to index it for a search engine.

Do search engines have to follow the instructions in the robots.txt?

No. What stands in the robots.txt file is only a reading recommendation from the site owner to the search engine. Large search engines such as Google, Bing and Yahoo agreed in 2008 to follow the instructions in a robots.txt, but there is never a hundred percent guarantee.

Where do I put the robots.txt file on my web server?

The file has to sit in the root directory, meaning the topmost directory of the web server. It is therefore always reachable by adding /robots.txt to the URL after the top level domain.

Example: https://taismo.de/robots.txt

How is a robots.txt file structured?

The syntax is compact and rests on three basic elements:

• User-agent:
  • This element sets which bot the following instructions apply to.
    • Example: googlebot
  • If an instruction is meant for every bot, a * can be used.
    • Example: User-agent: *
• Allow:
  • This element names a directory or a specific page that should be crawled.
• Disallow:
  • This element names a directory or a specific page that should not be crawled.

Example:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Explanation: this snippet carries instructions for every crawler. They should leave the folder wp-admin alone, with the exception of the PHP file admin-ajax.php.

What effect does the robots.txt have on SEO?

A robots.txt that is set up wrongly can keep parts of your website out of the search results, because they are shut out from crawling. If certain pages do not rank the way you want them to, the instructions in that file are worth a look.

The robots.txt can also pass further SEO relevant information to the web crawler. Among other things, the path to the sitemap can be stored in it. How that works is covered by the sitemap entry of our SEO glossary.

Whether your robots.txt blocks anything it should not, and whether it points crawlers to the right sitemap, is one of the points we check in an SEO audit.