How Search Engines Work

Every search engine, from Google to the crawler feeding an AI chatbot, runs the same basic pipeline: crawl → index → rank. Understanding this pipeline is the single highest-leverage piece of knowledge in SEO, because every problem you'll ever debug lives in one of these stages.

Stage 1: Crawling#

Crawling is discovery. A program called a crawler (or spider, or bot; Google's is Googlebot) fetches pages, parses them, extracts every link, and queues those links to fetch next. Repeat forever.

Crawlers discover URLs from:

• Links on pages they already know about
• XML sitemaps you submit (guide)
• Redirects and canonical hints
• Previous crawls (re-checking known pages for changes)

You control crawler behavior primarily through robots.txt:

User-agent: *
Disallow: /admin/
Disallow: /cart
 
Sitemap: https://example.com/sitemap.xml

How much Googlebot reads#

Googlebot for Search fetches only the first 2MB of an HTML or supported text file (and the first 64MB of a PDF). Each CSS or JavaScript file it fetches has its own 2MB limit. Google's other crawlers default to 15MB, which is where the old "Googlebot reads 15MB" line came from. For almost every page this is plenty, but bloated server-rendered pages with giant inline JSON can hit it. Put your <title>, meta tags, canonical and key structured data near the top of the HTML.

Crawl budget#

Search engines won't crawl your site infinitely. Crawl budget = how many URLs a crawler will fetch in a given window, a function of:

• Crawl capacity limit: how fast your server responds without degrading
• Crawl demand: how popular and frequently updated your pages are

Small sites rarely need to think about it; Google's own crawl budget guide is aimed at sites with very large numbers of URLs or very fast-changing content. Large sites live and die by it: wasting budget on duplicate or junk URLs means real pages go uncrawled.

Rendering#

Google renders JavaScript with a headless Chromium. Its docs describe the order: pages with a 200 status go into a render queue (unless a robots meta tag or header says not to index them), wait there "for a few seconds, but it can take longer", and then the rendered HTML is used for indexing and further link discovery. Two consequences worth remembering:

• A noindex in the initial HTML may stop rendering entirely, so JavaScript can't reliably remove it.
• Pages that return a non-200 status may not be rendered at all.

Google is now confident enough in its rendering that in March 2026 it removed an outdated section of its docs, noting that using JavaScript to load content is not "making it harder for Google Search". Other search engines and most AI crawlers are a different story: many fetch only the raw HTML. This is the subject of JavaScript SEO.

Stage 2: Indexing#

Indexing is comprehension and storage. The engine parses the rendered page and decides:

1. What is this page about? Text, headings, structured data, image alt text, link anchors.
2. Is it the canonical version? Duplicate and near-duplicate pages get clustered; one URL is chosen as canonical and the rest are folded into it.
3. Is it worth storing? Thin, duplicate or low-quality pages may be crawled but never indexed: the dreaded "Crawled - currently not indexed" in Search Console.

The result goes into the inverted index: a giant map from words and entities to the documents containing them, which is what makes searching hundreds of billions of pages in a fraction of a second possible.

Stage 3: Ranking#

When a user searches, the engine retrieves candidate documents from the index and ranks them using many signals. Google's ranking systems guide says they work mainly at the page level, with site-wide signals and classifiers also contributing. The major signal families:

Two important truths about ranking:

• Ranking is per-query, not per-site. You don't "rank" in the abstract; you rank for specific queries, each with its own competitive landscape.
• Intent dominates. Google's systems first decide what kind of result satisfies the query (product pages? tutorials? videos?) and rank within that. Matching search intent is a prerequisite, not an optimization.

Where AI answers fit#

AI Overviews and AI Mode are not a separate web. Google's documentation says they draw on its index and core ranking systems, and to be eligible a page simply has to be indexed and eligible to show with a snippet. They may use a query fan-out technique, running several related searches behind the scenes and assembling an answer from the results, which is why a page can be cited for a query it doesn't rank for directly. SEO for AI Mode & AI Overviews goes deeper.

Third-party AI systems run their own version of the pipeline. OpenAI, Anthropic, Perplexity and others each operate separate bots for training, for search indexing and for fetching a page when a user asks, and they obey robots.txt to different degrees. AI Crawlers & Bot Access has the current list, and the AI crawler checker shows which ones your robots.txt allows.

Debugging with the pipeline#

When a page isn't getting traffic, walk the stages in order:

Not crawled?   → check robots.txt, CDN bot rules, internal links, sitemap, server errors
Not rendered?  → check status code, noindex in raw HTML, blocked JS/CSS
Not indexed?   → check noindex, canonicals, content quality, duplicates
Not ranking?   → check intent match, content depth, links, competition
No clicks?     → check title/description, SERP features stealing attention

Google Search Console's URL Inspection tool tells you exactly where a URL sits in this pipeline: make it your first stop. Setting it up takes ten minutes and is covered in Your SEO Toolkit.

Next: Anatomy of a SERP, what the results page actually contains in the age of AI.