What is a user agent?

A user agent is the program that retrieves content from the web on behalf of a user, such as a browser, an app, or a crawler. In the narrower sense, user agent refers to the identifier this program sends to the web server with every request in the HTTP header User-Agent. Web servers use this identifier to recognize who is making the request, and the rules of a robots.txt for search engines and AI crawlers take effect through it.

Term profile

How is a user agent string structured?

A user agent string consists of product names with version numbers, supplemented by details in parentheses about the operating system, device, or purpose. Two examples, a browser and Google’s crawler for smartphones:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
  (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36
  (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
  (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The Mozilla/5.0 at the beginning is a historical leftover. In the 1990s, browsers identified themselves as Mozilla so that web servers would serve them the modern versions of pages, and almost every user agent keeps that opening to this day. Reputable crawlers append their name and a URL with an explanation, in the example Googlebot/2.1 and the link to Google’s documentation.

Since 2022, Chrome has been gradually shortening the user agent string and now provides details such as the exact version or the device model only on request via User-Agent Client Hints. The string is therefore less informative and offers fewer features for recognizing individual devices.

Which user agents do search engines and AI crawlers use?

Every reputable crawler identifies itself with its own name, and the major providers operate several user agents depending on the purpose. The most important identifiers:

The table shows why the distinction matters: One and the same provider can collect training data with one user agent and supply answers with sources through another. If you block an AI crawler, you therefore decide per token whether your website is missing from training, from search, or from both. The entry on Common Crawl explains what is behind the CCBot archive.

How do you control user agents in robots.txt?

In robots.txt, each group of rules sits under a User-agent: line with the token of the crawler it applies to. An example that locks out training crawlers and allows search crawlers:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Allow: /

3 rules apply to the evaluation. A crawler follows the group whose token matches it most specifically and then ignores the group with the asterisk. Upper and lower case in the token do not matter. The robots.txt is a request that reputable crawlers follow. Technically, it stops no one. For individual pages or file types, the X-Robots-Tag also controls indexing. How to specifically allow or block individual AI crawlers is shown in the magazine article Controlling AI crawlers: GPTBot, ClaudeBot, and the others.

How do you check whether a crawler is genuine?

Whether a crawler is genuine is shown by the IP address of the request, because any program can set the user agent string freely. Google explicitly warns in its documentation that the user agent can be spoofed. The check works in 3 steps:

1. Read the log file: The web server logs the user agent and IP address for every request.
2. Run a reverse lookup: For an IP address claiming to be Googlebot, the reverse DNS lookup returns a name under googlebot.com or google.com. The forward lookup of that name must return the same IP address again.
3. Compare with published lists: Google and OpenAI publish the IP ranges of their crawlers as JSON files. A request named GPTBot from an IP address outside these ranges comes from someone else.

Fake Googlebots are often scrapers that copy content, or tools trying to get around blocks. Firewalls that only look at the user agent let them through.

What role does the user agent play in SEO?

For SEO, the user agent determines which version of a page Google sees and evaluates. 3 connections matter:

• Mobile-first indexing: Google crawls most websites with the smartphone Googlebot. With mobile-first indexing, the version this user agent receives is the one that counts.
• Cloaking: If a crawler deliberately receives different content than a human, that violates Google’s spam policies. Cloaking via the user agent is the classic form of it.
• Rendering: Some websites deliver a pre-rendered HTML version to crawlers and a JavaScript application to humans. Google now calls this dynamic rendering a workaround and recommends server-side rendering. Background is in the entry on JavaScript SEO.

Why do firewalls block some user agents?

Firewalls and content delivery networks block user agents they classify as automated or suspicious, and in doing so they also hit requests that are wanted. Many rules block requests without a browser identifier or with the default identifier of programming libraries. taismo measured this with its own data requests: A service behind Cloudflare answered requests without a browser identifier with the HTTP status code 403 and error 1010. The response looked like a missing access right, and with a browser identifier the same request went through immediately.

The same mechanism affects AI crawlers. If a security rule at the host or CDN blocks GPTBot, ClaudeBot, or PerplexityBot, the website is missing from their answers even though robots.txt allows everything. Whether AI crawlers actually reach a website is checked by an AEO audit (also called GEO audit) with requests under the real user agents and a look at the log files.

Frequently asked questions about user agents

Can I change my user agent?
Yes. The developer tools in Chrome, Firefox, and Safari let you simulate a different user agent, for example to see a page as a smartphone or as Googlebot would.

Is the user agent personal data?
On its own, the user agent usually does not identify a person. Together with the IP address, screen size, and other features, it can become part of a fingerprint that allows devices to be recognized.

What does User-agent: * mean in robots.txt?
The asterisk stands for all crawlers that do not have their own group. If a crawler has its own group, only that group applies to it.

What is the difference between the user agent and the IP address?
The user agent is an identifier the program chooses itself. The IP address is the network address the request comes from, which is why it can be used to check where a request originates.