What is a crawler?

A crawler, also called a web crawler, is a software program used above all by search engines to browse the World Wide Web automatically and build an index. The name describes the way it works: the program moves from link to link through the web and reads one page after the other.

While it works through websites, the crawler analyzes and indexes their content so that search engines can read it faster later on. It lays the groundwork for everything a search engine does afterwards.

How does a web crawler work?

A crawler is programmed for repeating actions and therefore works automatically.

The information it finds while crawling gets assigned to certain categories and is then indexed, which makes it available for retrieval and evaluation. Texts, images and videos are part of that analysis, and so are the links a page contains.

Different types of crawlers

There are many different crawlers. Well known examples are the Googlebot and the Bingbot, and Alexa Internet ran a bot of its own called ia_archiver.

Crawlers can also carry out other tasks, such as data mining. The web is then searched for specific pieces of information, email addresses for example. Black hat SEOs use this in part to send spam mail to the addresses they collect.

Another field of work is plagiarism detection. That function matters above all when copyrighted material is involved.

How to control a crawler

With meta tags you give the crawler instructions in the head area of a page. Among other things you control whether the crawler indexes a page or follows a link.

This is what an instruction looks like that keeps the address out of the index and stops the crawler from following the links on it:

<meta name="robots" content="noindex,nofollow">

The noindex part of it works exactly like the noindex attribute on any other page. Instructions can be passed to the crawler through the robots.txt file as well.

Crawlers and SEO

Part of search engine optimization is building a website so that the crawler can read it as completely as possible, so that all content ends up in the index of the search engine. A logical URL structure helps here. On top of that, backlinks and clear internal linking lead to the crawler visiting your website more often and staying longer.

Whether crawlers actually get through your pages is one of the questions a technical SEO audit answers. The SEO glossary covers the terms around indexing that go with this one.