Crawling & Indexing

Technical SEO starts here: deliberately controlling what search engines crawl and what they index. The two are different systems with different controls, and confusing them causes most indexing bugs in the wild.

robots.txt#

Lives at the root of each host (https://example.com/robots.txt) and controls crawler access by path prefix:

User-agent: *
Disallow: /api/
Disallow: /admin/
Disallow: /*?sort=
 
# A crawler obeys only the most specific group that names it,
# so repeat the shared rules inside every named group.
User-agent: GPTBot
Disallow: /api/
Disallow: /admin/
 
Sitemap: https://example.com/sitemap.xml

That comment is the most common robots.txt bug in 2026. People add a block for an AI crawler with Allow: /, not realising that the bot now ignores the * group entirely and can fetch /admin/. Google also ignores crawl-delay (use Search Console or slow the crawl with 503/429 responses instead), while Sitemap: lines are supported by every major engine.

In Next.js you can generate the file from code:

import type { MetadataRoute } from "next";
 
const privatePaths = ["/api/", "/admin/"];
 
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "*", disallow: privatePaths },
      { userAgent: "GPTBot", disallow: privatePaths },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}

Which AI bots to allow, and how to separate training from search and user-triggered fetches, is a topic of its own: see AI Crawlers & Bot Access, and paste your file into the AI crawler checker to see what each bot is allowed to do.

How much Googlebot actually reads#

Googlebot for Search fetches the first 2MB of an HTML or other supported text file, and the first 64MB of a PDF. Anything after the cutoff is ignored. Each CSS and JavaScript file is fetched separately with its own limit. The 15MB figure you may remember is now the documented default for Google's other crawlers and fetchers, not for Googlebot on Search (documented 2026-02-03, explained in a March 2026 post).

Most pages are nowhere near 2MB. The ones that get close are server-rendered apps that inline huge JSON state, base64 images or SVG sprites. Practical rules:

• Put <title>, meta robots, the canonical and essential structured data high in the document.
• Keep inline data payloads small; load the rest from separate files.
• Check the uncompressed HTML size of your heaviest templates (curl -s URL | wc -c).

Google's crawlers and the AI question#

Google runs many crawlers and fetchers from one infrastructure. The ones that matter for Search and AI:

• Googlebot (smartphone and desktop) crawls for Search, including Discover, Images, Video and News. AI Overviews and AI Mode are built from this same index, so blocking Googlebot removes you from both classic results and AI features.
• Google-Extended is a robots.txt product token, not a separate crawler. Per the common crawlers page (updated 2026-07-14) it controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Search Console's help page adds that it also limits training of the models behind Search's generative features. The two pages differ in scope, so don't promise more than either says.
• User-triggered fetchers (Google-Agent, added 2026-03-20 for agents acting on a user's behalf, plus Google-GeminiNotebook, Google-Read-Aloud and others) fetch a page because a person asked. Google states these "generally ignore robots.txt rules".

Controlling how Google's AI features use your content#

You now have three levers, from blunt to precise:

1. Search generative AI setting (Search Console > Settings > Search generative AI: Include, Exclude or Inherit from parent). Excluding a property removes its links and content from AI Overviews, AI Mode and Discover's generative AI features. Google says it "isn't used as a ranking or inclusion signal" for the rest of Search and takes effect in about one to two days. It launched for UK sites on 2026-06-03 and globally on 2026-08-31. It works per property, so a URL-prefix property can scope it to a section.
2. nosnippet stops snippets entirely, and Google's robots meta docs say it also prevents the content being used as a direct input for AI Overviews and AI Mode. max-snippet:[n] limits how much text may be used. Both also shrink your classic snippet.
3. data-nosnippet on an element keeps just that passage out of snippets and AI answers.

Opting out has a cost: the measuring AI visibility guide shows how to check what AI features actually send you before you decide.

XML sitemaps#

A sitemap is a machine-readable list of URLs you want crawled: a discovery aid, not a ranking factor. Rules:

• Include only canonical, indexable, 200-status URLs. A sitemap full of redirects and noindexed pages erodes trust in the file.
• 50,000 URLs or 50MB (uncompressed) per file; use a sitemap index beyond that.
• lastmod should be honest: Google uses it when it is consistently accurate and learns to ignore it when it lies. Google ignores priority and changefreq.

import type { MetadataRoute } from "next";
import { getAllPosts } from "@/lib/posts";
 
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getAllPosts();
  return [
    { url: "https://example.com", lastModified: new Date() },
    ...posts.map((post) => ({
      url: `https://example.com/blog/${post.slug}`,
      lastModified: post.updatedAt,
    })),
  ];
}

Submit it once in Search Console (and Bing Webmaster Tools); afterwards engines refetch it on their own schedule. Bing, Yandex and others also accept instant pings via IndexNow; Google does not support IndexNow.

Static vs dynamic sitemaps#

• Static: a plain public/sitemap.xml you write or generate at build time. Fine for small or rarely changing sites, trivial to inspect, but stale the moment you add a page and forget it.
• Dynamic: an app/sitemap.ts route (above) that builds the list from your data. What you want once URLs come from a CMS or database. It can never drift out of sync with your content.

A useful rule: if a non-developer can publish a page, you need a dynamic sitemap. If every page ships in a Git commit, a static file is fine.

Sharding past 50,000 URLs#

Large sites split into multiple files behind a sitemap index. Next.js does this with generateSitemaps, which produces numbered shards and the index automatically:

import type { MetadataRoute } from "next";
import { getProductCount, getProducts } from "@/lib/products";
 
const PER_SITEMAP = 50_000;
 
export async function generateSitemaps() {
  const count = await getProductCount();
  const pages = Math.ceil(count / PER_SITEMAP);
  return Array.from({ length: pages }, (_, id) => ({ id }));
}
 
export default async function sitemap({
  id,
}: {
  id: number;
}): Promise<MetadataRoute.Sitemap> {
  const start = id * PER_SITEMAP;
  const products = await getProducts(start, start + PER_SITEMAP);
  return products.map((product) => ({
    url: `https://example.com/product/${product.slug}`,
    lastModified: product.updatedAt,
  }));
}

Index control: noindex#

noindex removes a page from results while letting users access it: right for internal search results, thin tag pages, thank-you pages and login screens.

export const metadata = {
  robots: { index: false, follow: true },
};

For non-HTML resources (PDFs, images), send it as an HTTP header instead: X-Robots-Tag: noindex.

Two details documented in 2025-2026: Google respects a robots meta tag even when it sits in the <body> (so a stray tag in a component still counts), and a noindex in the initial HTML may stop Google rendering the page at all, so removing it later with JavaScript is unreliable. See JavaScript SEO.

Canonicalization#

When several URLs serve the same content (parameters, casing, trailing slashes, www/non-www, http/https), engines pick one canonical and fold the rest into it. Influence the choice with, in order of strength:

1. Redirects (301/308): the strongest statement; use whenever the duplicates have no reason to exist
2. rel=canonical: when duplicates must stay accessible (tracking parameters, print views)
3. Sitemap inclusion: list only canonicals
4. Consistent internal linking: always link to the canonical form

Audit reality against intent with Search Console's URL Inspection: "Google-selected canonical" disagreeing with "user-declared canonical" is a top cause of "indexed, but not the URL I wanted". After a fix, be patient: Google's troubleshooting doc says it may hold pages in a duplicate cluster for up to two weeks. The full playbook is in Duplicate Content.

Status codes engines care about#

Soft 404s (pages returning 200 with "not found" content) confuse all of this. Return real status codes; in Next.js, call notFound() rather than rendering an empty state with a 200. Google's HTTP status code reference has the full list.

Debugging indexing issues#

Work through Search Console's Indexing → Pages report. The frequent culprits:

"Excluded by 'noindex' tag"            → intended? if not, remove the tag
"Alternate page with proper canonical" → engine merged it; check that's correct
"Crawled - currently not indexed"      → usually quality or duplication, not technical
"Discovered - currently not indexed"   → crawl capacity or weak internal links
"Blocked by robots.txt"                → unblock (or stop listing it in the sitemap)

Then confirm with URL Inspection → "Test live URL", which shows the rendered HTML, the detected canonical and any robots directives exactly as Googlebot saw them.

Next: Site Architecture, structuring URLs and navigation so crawlers and link equity flow where you want.