Skip to main content
loading="lazy" loading="lazy" loading="lazy" loading="lazy" Copy link
loading="lazy"

Table of Contents

What is robots.txt?
Are search engine operators obliged to follow the instructions in the robots.txt file?
Where should I store the robots.txt file on my web server?
How is a robots.txt file structured?
What influence does the robots.txt file have on SEO?

What is robots.txt?

Robots.txt is a text file located on a web server. It is structured according to the Robots Exclusion Standard Protocol. It contains instructions for aweb crawler that scans the page in order to index it for a search engine.

Are search engine operators obliged to follow the instructions in the robots.txt file?

No! The information in the robots.txt file is merely arecommendationfrom the website operator to the search engine. Major search engines such as Google, Bing, and Yahoo agreed in 2008 to follow the instructions in robots.txt. However, there is never a 100% guarantee.

Where should I place the robots.txt file on my web server?

The file must be located in the root directory, i.e., the top-level directory, of the web server. The file is therefore always accessible if you add/robots.txt to theURLafter the top-level domain.  to the URL.

Example: https://taismo.de/robots.txt

How is a robots.txt file structured?

The syntax is very straightforward and consists of three basic elements:

  • User agent:
    • This element specifies which bot the following instructions apply to.
      • Example:googlebot
    • If the instruction is to apply to all bots, * can be inserted. 
      • Example:User-agent: *
  • Allow:
    • The element specifies a directory or a specific page to be crawled.
  • Disallow:
    • The element specifies a directory or a specific page that not be crawled.

Example:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Explanation:

This code snippet contains instructions for all crawlers. These are intended to exclude the folders wp-admin , with the exception of the PHP file admin-ajax.php.

What influence does the robots.txt file have on SEO?

An incorrectly configured robots.txt file can prevent certain content on your website from appearing on theSERP because it is excluded from crawling. If certain pages are not ranking as desired, the instructions in the file should be examined.

The robots.txt file can be used to provide additionalSEO-relevantinformation to the web crawler. Among other things, the path tothe sitemap can be stored can be stored.

You can find out how to do this in our glossary article onthe sitemap.