What is indexing?

Indexing is the process of a web page being taken into the Google index, the searchable database of the search engine. Only indexed pages can rank in the search results. Before a page is indexed it passes through two steps: crawling (Google finds and fetches the page) and rendering (Google puts the page together including JavaScript). Only after that does Google decide whether the page goes into the index.

Indexing is the entry ticket to Google Search. A page that is not in the index practically does not exist for Google, however good its content may be. Keeping control of the indexing status is therefore one of the most important jobs on the technical side of SEO.

Indexing at a glance

What does indexing mean at Google?

Indexing means that Google analyzes a page, understands it and files it into its index. The Google index is a huge database in which Google stores billions of web pages together with their content, key terms and metadata. When somebody runs a search, Google does not search the live web but this prepared index, which is what makes results in fractions of a second possible.

An indexed page is therefore qualified for a ranking in the first place. The distinction matters: indexing is the precondition for ranking, not a guarantee of it. Google takes a page into the index and then decides, through a completely separate evaluation process, on which position it appears for which search term. Plenty of indexed pages rank for no relevant keyword on page one at all.

Google does not index every URL it finds by any means. The search engine makes a selection and passes over pages it considers low in value, duplicated or irrelevant. In Google Search Console such URLs show up with the status Discovered, currently not indexed or Crawled, currently not indexed.

Crawling, rendering, indexing: what is the difference?

Crawling, rendering and indexing are three consecutive phases a page goes through before it appears in search. They are often mixed up, but they describe clearly separate processes.

1. Crawling: the crawler (Googlebot) visits a URL and downloads its source code. The precondition: the page must not be excluded from crawling by robots.txt. How much of a website Google fetches in a given period is a question of the crawl budget.
2. Rendering: Google puts the page together the way a browser displays it, including content loaded later through JavaScript. Only after rendering does Google see the full content.
3. Indexing: Google evaluates the rendered content and stores the page in the index, provided it meets the quality criteria and no signal forbids inclusion.

The order is what counts: no rendering without crawling, no complete indexing without rendering. One frequent misunderstanding concerns the interplay of robots.txt and noindex. A page blocked by robots.txt cannot be crawled by Google at all, so Google never reads a noindex in the source code either, a contradiction that causes a great many indexing problems.

Why is my page not indexed?

A page is not indexed when a technical signal prevents inclusion or when Google does not consider the content valuable enough. The most common causes fall into two groups: technical blocks and quality decisions.

Technical blocks are usually either intended or a configuration mistake:

• noindex signal: a noindex attribute in the meta tag or an X-Robots-Tag in the HTTP header tells Google not to take the page into the index.
• Crawl block: an entry in robots.txt blocks access. Google cannot read the page then and never gets to see an index signal.
• Canonical pointing to another URL: if the canonical tag points to a different page, Google treats the current URL as a copy and indexes the original instead.
• Missing internal links: if Google finds a page through no internal link and no sitemap, it often never gets crawled in the first place.

Quality decisions Google makes on its own. Thin content, near identical duplicates or pages without recognizable added value end up with the status Crawled, currently not indexed. No technical switch helps here, only better and more independent content. In practice this diagnosis, technical problem or quality problem, is a core piece of every SEO audit.

How do you check the indexing status?

The most reliable way to check the indexing status is Google Search Console. For every verified website the free tool shows exactly which pages are in the index and why others are missing. There are three proven ways:

1. URL inspection in Search Console: enter a single URL and Google shows either URL is on Google or the concrete reason for exclusion. The most precise route for one page.
2. The Pages report: it shows all indexed and non indexed URLs of the whole website, grouped by cause. Ideal for spotting patterns.
3. The site: query on Google: searching for site:your-domain.com roughly lists the indexed pages. That is only a sample, not an exact figure, but a quick first look without logging in.

For individual important pages you can trigger crawling in Search Console through Request indexing. Google then puts the URL into the queue, but there is no guarantee and no fixed deadline for inclusion.

How do you control indexing (noindex, canonical, robots)?

You control indexing through three instruments that work at different points. Each has a clearly separate job, and mixing them up is the most common source of error on the technical side of SEO.

• robots.txt controls crawling, not indexing. It tells Google which areas it should not visit. A page blocked by robots.txt can still end up in the index if other pages link to it, though without a snippet.
• noindex controls indexing directly. As a meta tag or X-Robots-Tag the signal reliably keeps a page out of the index. Important: the page has to stay crawlable, otherwise Google never reads the noindex.
• Canonical controls which of several identical URLs is indexed. The canonical tag bundles duplicates onto one preferred version and avoids duplicate content that way.

The rule of thumb: robots.txt governs access, noindex governs inclusion, canonical governs the choice among duplicates. Anyone who wants to keep a page dependably out of the index sets noindex and leaves the page crawlable. Anyone who wants to save crawl budget uses robots.txt, but accepts that a noindex has no effect there. More terms around crawling and indexing are collected in the SEO glossary.

Frequently asked questions about indexing

How long does indexing a new page take?
From a few hours to several weeks. On established websites that are crawled often it usually goes fast, on new domains without any links it can take a while. Google guarantees no fixed deadline.

Does indexed mean my page is in first place?
No. Indexing is only the precondition for a page being able to rank at all. Which position it appears on is decided by a separate evaluation process based on relevance and quality.

How do I request indexing for a page?
Through the URL inspection in Google Search Console with the Request indexing function. That puts the URL into the crawl queue, but it does not force inclusion.

Why do indexed pages disappear from the index again?
Google removes pages when they return an error over a longer period, receive a noindex, point to another URL through a canonical, or lose quality and relevance. A regular look at the Pages report in Search Console brings such losses to light.