Technical SEO: How Google Crawls, Renders, and Ranks Your Site
Dominik Breitbach founded taismo GmbH, an SEO and GEO agency from Munich. He runs the technical side of search every day: Crawling, indexing, structured data and visibility inside AI answers.
Technical SEO is the work on a website’s infrastructure that lets search engines and AI systems find, load, understand and index it. It sits underneath content and backlinks: When the technical layer fails, even the strongest page stays invisible. This guide works through every layer in the order that decides the outcome and ends with a checklist for 2026.
👉 Not sure where your site stands? A technical SEO audit surfaces the blockers within days.
What technical SEO covers
Technical SEO is the discipline that makes a website machine readable. On-page SEO decides whether your content answers the question, technical SEO whether a search engine can process the site at all.
Technical SEO covers eight work areas:
- Crawl control: The robots.txt, crawl budget, internal link paths.
- Index control: The noindex attribute, X-Robots-Tag, canonical tags, sitemaps.
- Status codes: 200, 301, 404, 410 and the chains between them.
- Rendering: Whether content survives JavaScript.
- Speed: Server response time and the Core Web Vitals.
- Mobile parity: Phone version equals desktop version.
- Security: Valid certificate, HTTPS everywhere, no mixed content.
- Structured data: JSON-LD that types the page as an entity.
Each area produces defects that never surface in your analytics, because the traffic they cost you never arrives: The damage stays silent until somebody goes looking for it.
Technical vs. on-page and off-page SEO
On-page SEO covers everything a reader sees: Text, heading structure, search intent, meta data and internal linking. It answers one question: Is this page the best available answer?
Off-page SEO covers the external trust signals, mainly backlinks and brand mentions, which our entry on off-page optimization breaks down. It answers a second question: Do others treat this domain as credible?
Technical SEO sits underneath both and answers a third: Can a crawler reach, render, understand and store this page? The areas overlap, and that is the point. A clean URL structure is technical and on-page at once, structured data is technical and changes how your result looks.
Sequence beats labels: A brilliant guide that Googlebot cannot reach ranks nowhere.
The four stages every URL passes before it ranks
Google runs a pipeline with four stages: It crawls the URL, renders the code, indexes the result, and only then evaluates the page for ranking.
This pipeline is why technical findings need a priority order rather than a severity label. A page blocked in robots.txt never gets rendered, so its Core Web Vitals do not matter yet. A page that renders empty gets indexed as empty. Work the pipeline downward and much of what sits below disappears on its own.
Crawling and indexing: Can Google reach your pages?
Crawling and indexing are two separate steps, and confusing them causes more damage than any other misunderstanding in technical SEO. Crawling means a search engine fetches the URL, indexing means it stores the result. A page can be crawled and never indexed, or indexed without being crawled.
Crawl budget decides how much of your site Google sees
The crawler, Googlebot in Google’s case, follows links from page to page and fetches what it finds. How many URLs it fetches in a given period is called crawl budget. On a 40 page site this is never the bottleneck. On a shop with faceted navigation and tracking parameters it becomes the whole game: Every crawl spent on a filter permutation is a crawl not spent on a product page.
Four levers move crawl budget: Exclude parameter and filter URLs, resolve redirect chains, keep the architecture flat, remove internal links that run into 404s. Google also raises crawl capacity for sites that respond quickly, which makes server speed a crawling topic too.
robots.txt and the XML sitemap pull in opposite directions
The robots.txt in the root directory tells crawlers which areas they may enter, and one misplaced Disallow locks out an entire directory. The detail that trips up most teams: Disallow prevents crawling, not indexing. A blocked URL can still end up in the index, without Google ever reading its content or its noindex. To keep a page out of the index, robots.txt is the wrong tool.
The XML sitemap is the counterpart. It lists the URLs that should be indexed and helps Google discover deeply nested pages. A clean sitemap holds only indexable 200 pages: No redirects, no noindex, no blocked URLs. Contradictions between sitemap and page status are the most frequent finding in our audits.
noindex, X-Robots-Tag and canonical tags decide what enters the index
Not every page belongs in the index. Filter results, internal search pages and thin archives stay out. Two clean instruments exist: The noindex attribute in the meta robots tag, and the X-Robots-Tag in the HTTP header, which also works for PDFs. Both need crawl access to be read at all.
Canonical links solve a different problem. When the same content is reachable under several URLs, the canonical tag names the preferred version and consolidates the signals there. The rule we apply on every project: Every page carries a self referencing canonical on its own clean URL. That neutralizes tracking parameters and session IDs before they turn into duplicates.
Status codes and redirect chains leak crawl budget and link value
Every URL answers with an HTTP status code. Five matter for SEO: 200 for a working page, 301 for a permanent move that passes signals on, 302 for a temporary one, 404 for not found and 410 for permanently removed. Three mistakes repeat: A 302 for a permanent move, a bulk 301 redirect onto the home page, three hop chains.
These defects hide in plain sight. In a recent audit of a WordPress site we counted 315 internal links pointing at 301 redirects. The cause was one footer template linking to two legal pages without a trailing slash, repeated sitewide. One fix, 315 findings gone: Few causes, many symptoms.
Mobile-first indexing makes the phone version the master copy
Google indexes with the smartphone Googlebot, so the mobile version forms the basis for indexing and ranking. Mobile-first indexing has one practical consequence: Whatever you hide on mobile, you hide from Google. Collapsed content is fine, it exists in the HTML. Content that a responsive design drops entirely, or structured data that only the desktop template outputs, is content Google never evaluates.
Rendering and JavaScript SEO: Does Google see what users see?
Between crawling and indexing sits a stage most audits skip: Rendering. Modern sites assemble much of their content in the browser through JavaScript, and Google renders those pages in a second, deferred pass. When content appears only after JavaScript runs, Google can index the page empty and pick up the real content much later, or never.
JavaScript SEO makes sure critical content ships reliably. Server-side rendering or pre-rendering is the robust answer, because the server hands over finished HTML. Three rules decide the rest:
- Real links: Navigation uses
<a href>elements. A click handler on a<div>is not a link and does not get followed. - No content behind interaction: Anything that loads only after a click or a tab switch is content a crawler will likely miss.
- Stable URLs: Every state that should rank needs its own crawlable URL.
The test takes two minutes: Open the URL inspection tool in Google Search Console and read the rendered HTML Google stored. Missing there means missing from the index.
Site speed and Core Web Vitals in 2026
Speed is a ranking input and a conversion input at once. Google measures it through the Core Web Vitals, three metrics for loading, responsiveness and visual stability, each with a published threshold.
- Largest Contentful Paint (LCP): Measures when the largest visible element finishes loading. Good is 2.5 seconds or faster.
- Interaction to Next Paint (INP): Measures responsiveness to user input and replaced the older First Input Delay metric in March 2024. Good is 200 milliseconds or faster.
- Cumulative Layout Shift (CLS): Measures how much the layout jumps while loading. Good is 0.1 or lower.
One distinction decides whether your speed work is real: Field data beats lab data. A lab test in PageSpeed Insights runs once on a simulated device. The field data in the same report comes from real visitors on real connections, and that is what Google evaluates. A perfect lab score next to failing field data is common, because the lab run never meets the slow phone.
Six levers move the numbers, strongest first:
- Server response time: A slow time to first byte caps everything downstream. No frontend work repairs a two second server.
- Content delivery network: Serves assets from an edge location near the visitor, which matters most for international traffic.
- Compression: Text resources ship with brotli, which beats gzip.
- Images: Modern image formats such as WebP and AVIF cut file size without visible quality loss.
- Lazy loading: Off screen images load on demand. The LCP element must never be lazy loaded, the most common self inflicted LCP problem we find.
- Caching: Repeat visits reuse stored resources.
Speed is not a project with an end date. Every new plugin, tracking script and uncompressed hero image walks the numbers back.
We measure crawling, indexing and Core Web Vitals across your site and hand you the three levers with the largest effect, each with its fix.
Site architecture and internal linking
Structure decides how well crawlers and readers reach what matters. A flat architecture, where every important page sits within three clicks of the home page, spreads internal link value where it earns something. Three principles carry the effect:
- Flat hierarchy: What sits six clicks deep gets crawled less often and evaluated more weakly.
- Readable URLs: Short, lowercase, descriptive, without special characters. Our rule for German sites: Transliterate umlauts, so ä becomes ae and ß becomes ss.
- Deliberate internal linking: Related pages link to each other inside sentences. This article is the working example: It links into our English technical glossary at every term instead of redefining it.
Breadcrumb navigation serves both goals: It shows where a page sits in the hierarchy and adds internal links with unambiguous context.
HTTPS and security
An encrypted connection through SSL/TLS counts as a light ranking signal, and the trust effect weighs more: Browsers mark unencrypted pages as not secure, and visitors leave. Clean means four things: A valid certificate, HTTP redirected to HTTPS, no mixed content, HTTPS URLs as canonical targets.
Structured data: The bridge to rich results and AI visibility
The largest gains of 2026 sit here, and this is taismo’s home turf. Structured data is machine readable information in the source code that tells a search engine what a page is: An article, a product, an organization, an FAQ, a person. The recommended format is JSON-LD with the schema.org vocabulary. The classic payoff is rich snippets: Star ratings, FAQ accordions or breadcrumb paths, which lift click-through rate without moving the position.
The newer and more valuable payoff is AI visibility. Large language models and AI Overviews rely on clearly typed entities to attribute content. A site that joins its organization, authors, services and definitions into one graph through unambiguous @id references is far easier for a machine to quote correctly. That is the core of generative engine optimization (GEO), and the reason we treat structured data as a GEO discipline instead of a snippet trick.
In practice: One entity model instead of copied snippets. Organization, person, brand and services defined once globally and referenced per page, with types such as Article, FAQPage or DefinedTerm on top.
Duplicate content, canonicals and international sites
Duplicate content, meaning identical content under several URLs, dilutes signals and confuses indexing. It grows out of technical accidents: Variants with and without www or a trailing slash, over HTTP and HTTPS, parameter URLs from filters and tracking. The fix is rarely deletion. It is canonicalization plus consistent redirects to the one valid variant.
As soon as a site serves several languages or countries, hreflang enters the picture. It tells Google which language version belongs to which audience and stops versions from cannibalizing each other. hreflang is unusually error prone: References have to be reciprocal, language and region codes valid, and an x-default should catch everyone else. We moved taismo.de from an automatic translation layer to real translated pages for exactly that reason: The layer served German content on English URLs behind an English hreflang annotation.
Technical SEO for ecommerce: Facets, pagination and index bloat
Shops are where technical SEO stops being hygiene and becomes the ranking question itself. A catalog site generates URLs on its own: Every filter, every sort order and every page of a listing is a new address a crawler can request. Google puts a number on where this starts to hurt. Crawl budget becomes a real topic above roughly one million unique pages, or from around 10,000 pages when the content changes daily. Below that threshold, the reason to control facets is duplicate content, and the cleanup is cheap while the catalog is still small.
Faceted navigation multiplies your URL count
Three filters with five values each already produce 125 combinations per category, and one sort parameter doubles every single one of them. Google’s own guidance is blunter than most teams expect: Disallow crawling of faceted navigation URLs in robots.txt, because there is usually no good reason to let a crawler spend server resources on filtered views. Canonical links and rel="nofollow" are the alternatives, and Google calls both generally less effective in the long term. Nofollow only holds if every single link pointing at the URL carries the attribute, a promise no template keeps for long.
Two details decide whether the facets you do want indexed stay clean. Keep the parameter order deterministic, so the same selection always produces the same URL rather than six spellings of it. And return a genuine 404 for filter combinations with no results, instead of an empty page that answers 200.
Pagination: One URL per page, one canonical per URL
Google’s rules for paginated listings are short, and shops break them constantly. Every page in the sequence gets its own URL, typically through a ?page=n parameter, and its own self referencing canonical. Canonicalizing page two onto page one is the classic mistake, because it hides everything that only exists deeper in the sequence. The tags rel="next" and rel="prev" carry no weight at Google anymore. What still counts is a real <a href> link from each page to the next one, and a URL that differs by more than a fragment, because Googlebot ignores everything after the #.
Infinite scroll and load more buttons inherit the rendering problem from the JavaScript section above: If the deeper items never get their own crawlable URL, they never get indexed, no matter how well the listing works for a human.
Index bloat is the bill that arrives later
Index bloat means a large share of your indexed URLs carries no value of its own: Filter views, sorted duplicates, print versions, internal search results, empty tag archives. Nobody hands out a penalty for it. The damage is dilution and delay: Signals spread across near identical URLs, and the discovery of genuinely new pages slows down, which Google names as a direct consequence of crawling filtered URLs. The fix is one decision per URL type: Index it, canonicalize it, or keep it out of the crawl.
How to run a technical SEO audit
A technical SEO audit is a crawl, a comparison and a decision about order. The crawl produces a list, the comparison turns it into findings, the order turns findings into work that pays.
The four data sources an audit needs
Most audits run on a single source and inherit its blind spot. Four sources answer four different questions, and a serious audit uses all of them:
- Your own crawl: The site as a crawler sees it, with status code, canonical, robots directives and outgoing links per URL.
- Google Search Console: What Google actually did, which is rarely the same thing. The page indexing report names the reason a URL stayed out, the crawl stats report shows response codes and crawl volume over time.
- Field data: Core Web Vitals from real visitors, broken down by template type, because a domain average hides the one slow template that carries your money pages.
- Server log files: The only record of what Googlebot actually requested, and how often.
The audit in five steps
Step 1: Crawl every URL. Fetch the whole site the way a crawler would and record status code, canonical, meta robots, headings, images and outgoing links per URL.
Step 2: Compare the crawl against reality. Put the crawl next to the XML sitemap and next to the index report in Search Console. Three gaps matter: Sitemap URLs that are not indexed, indexed URLs that should not be, and crawled URLs no sitemap knows about.
Step 3: Pull field data. Take Core Web Vitals from real users rather than from a single lab run, and inspect the rendered HTML per template type instead of the home page alone.
Step 4: Sort by blocking power, not by count. A report that lists 3,000 missing alt attributes above 12 pages that dropped out of the index has its priorities inverted. Rank by what a finding blocks, using the pipeline from Fig. 2.
Step 5: Monitor for regressions. A fixed defect stays fixed only until the next plugin update. Recurring crawls plus Search Console alerts catch the relapse while it is cheap.
What server logs show that a crawl cannot
A crawler tells you what exists. Logs tell you what Google spends its time on, and the two lists are never identical. Four questions only the logs answer: Which URL patterns absorb the crawl requests, which sections Googlebot has not touched in weeks, how large the share of non-200 responses is, and whether the crawl rate dropped after a server incident. That last question is measurable. Google reduces the crawl rate in proportion to the number of URLs that return 5xx errors, so a bad deployment weekend shows up weeks later as a dip in the crawl curve.
One precaution before you read anything into the numbers: Verify that the requests came from Google at all. The user agent string is trivial to fake, which is why Google documents a reverse DNS check and publishes the IP ranges of its crawlers. Skip that step and you are measuring scrapers.
The technical SEO issues we find most often
A crawl produces rows. Rows are not issues. Six defects account for most of the real damage across the audits we run, and each one has a symptom you can look up on your own site today.
| Issue | What you see | Usual cause | Fix |
|---|---|---|---|
| Soft 404 | The URL answers 200, Search Console files it as soft 404 | Empty category, discontinued product, a page that prints “no results” and still returns 200 | Return a real 404 or 410, or give the page content worth indexing. Soft 404s keep getting crawled and burn budget |
| Blocked resources | The rendered screenshot in the URL inspection tool looks broken | CSS or JavaScript directories sit behind a Disallow in robots.txt |
Allow the resources the layout needs. Google evaluates the version it manages to render |
| Server errors under load | 5xx responses cluster in the crawl stats report | Server or database limits under concurrent requests, often on the largest listing pages | Fix capacity. Google throttles crawling in proportion to the number of URLs returning errors |
| Orphan pages | The URL sits in the sitemap, no internal link points at it | Pages published outside the navigation, campaign landing pages left behind after the campaign | Link them from a relevant page, or take them out of the sitemap. A page nobody links to is a page nobody finds |
| Canonical into a dead end | Google reports the canonical as ignored, or indexes a different URL than the one you named | The canonical target is itself a redirect, a noindex page or a 404, usually after a URL change | Every canonical target answers 200 and is indexable. Check this after every URL migration |
| Staging noindex in production | Rankings fall away within days of a launch, pages leave the index | The X-Robots-Tag or meta robots of the staging environment survives the deployment | Read the response header of a live URL after every release. This one costs the most and takes the least time to check |
Three of these six are template defects rather than page defects, and that is the normal case. It explains why the number of fixes in a good audit is far smaller than the number of findings: One header rule, one robots.txt line, one canonical in one page type.
Scale decides the order more often than severity does. A soft 404 on one retired category costs almost nothing. The same defect across 4,000 empty filter pages eats the crawl budget of an entire shop. So count the URLs behind a finding before you rank it, and check which stage of the pipeline it stops.
Technical SEO checklist for 2026
Work from top to bottom: What sits higher up blocks everything below it.
- Crawling: Correct robots.txt, sitemap limited to indexable 200 pages.
- Indexing: Unimportant pages excluded through
noindex, never robots.txt. - Canonicals: Self referencing on every page, duplicates consolidated.
- Status codes: No internal links into 404s, permanent moves as direct 301s.
- Mobile: Same content and structured data as desktop.
- Rendering: Critical content without JavaScript, navigation as real
<a href>elements. - Core Web Vitals: LCP 2.5 s, INP 200 ms, CLS 0.1, verified against field data.
- Speed levers: Fast server response, compression, modern image formats, lazy loading except the LCP element, caching.
- HTTPS: Valid certificate, HTTP redirected, no mixed content.
- Structured data: Valid JSON-LD, one entity graph, right type per template.
- International: Reciprocal hreflang with
x-default. - Monitoring: Search Console connected, recurring crawl, regressions tracked.
taismo is an SEO and GEO agency from Munich, ranked fifth in the German SEO Contest 2026 and second among Munich online marketing agencies by Agenturtipp. We audit the technical layer and keep the foundation clean in an SEO retainer.
Technical SEO FAQ
What is technical SEO?
Technical SEO is the part of search engine optimization that makes a website findable and processable for machines. Its job is to remove every technical obstacle between your content and a ranking.
What does technical SEO include?
Technical SEO includes eight work areas: Crawl control, index control, HTTP status codes, rendering and JavaScript delivery, Core Web Vitals, mobile-first parity, HTTPS security and structured data.
How do you perform a technical SEO audit?
You crawl every URL, compare the crawl against what Google indexed in Search Console, pull field data on Core Web Vitals, then sort the findings by blocking power instead of by count.
What is the difference between technical SEO and on-page SEO?
On-page SEO works on the content of a page: Text, headings, search intent and meta data. Technical SEO works on the infrastructure that delivers it to a crawler, which is why content Google cannot crawl ranks nowhere.
How much does a technical SEO audit cost?
Prices range from a few hundred euros for a small site to five figures for an enterprise platform. The useful question is the deliverable: A report without a prioritized order of work is expensive at any price.
What are the most common technical SEO issues?
The six that cause most of the damage are soft 404s, resources blocked in robots.txt, server errors under crawl load, orphan pages, canonical tags pointing at redirected or noindexed URLs, and a staging noindex that survives a deployment. Most of them sit in a template, which is why one fix usually clears many findings.
How often should you run a technical SEO audit?
A full audit once or twice a year is enough for a stable site. What matters more is the cadence in between: A recurring crawl plus Search Console alerts catches a regression while it is still cheap. Every relaunch, URL migration or major plugin update deserves its own check on top.
Does crawl budget matter for a small website?
Rarely. Google names crawl budget as a topic above roughly one million unique pages, or above 10,000 pages when the content changes daily. On smaller sites the same filter and parameter URLs still create duplicate content, and that is the reason to clean them up.
Do you need a technical SEO agency?
That depends on where the work stalls. Finding the issues takes a crawler and a few hours. Fixing them needs access to templates, server configuration and the deployment process, which is where in house projects usually stop. Buy the diagnosis and the priority order, and keep the recurring monitoring wherever it actually gets done.
Sources
- Google Search Central: “In-depth guide to how Google Search works”, Google, 2026.
- Google Search Central: “Introduction to robots.txt”, Google, 2026.
- Google Search Central: “How to specify a canonical URL”, Google, 2026.
- Google Search Central: “Understand JavaScript SEO basics”, Google, 2026.
- Google Search Central: “Mobile-first indexing best practices”, Google, 2026.
- Google Search Central: “Core Web Vitals and page experience”, Google, 2026.
- web.dev: “Web Vitals”, Google, 2026.
- web.dev: “Interaction to Next Paint (INP)”, Google, 2026.
- schema.org: “Organization of Schemas”, Schema.org Community Group, 2026.
- Google Crawling Infrastructure: “Crawl Budget Management”, Google, 2026.
- Google Crawling Infrastructure: “Managing crawling of faceted navigation URLs”, Google, 2026.
- Google Crawling Infrastructure: “How HTTP Status Codes Affect Google’s Crawlers”, Google, 2026.
- Google Crawling Infrastructure: “Verify Requests from Google Crawlers and Fetchers”, Google, 2026.
- Google Search Central: “Pagination Best Practices for Google”, Google, 2026.
