A technical audit is the systematic health check that finds what is silently costing a site visibility. Done professionally, it is not a 200-row dump of crawler warnings. It is a short, prioritized list of issues with evidence, impact estimates and owners. This is the playbook, and it is the same method you apply step by step in the Capstone: Audit Your Own Site.
Before touching a tool#
Establish context, or the audit optimizes the wrong things:
- Which pages make money or matter? Findings get weighted by the pages they affect.
- What changed recently? Migrations, redesigns and CMS swaps explain most sudden problems. Check deploy history and Search Console annotations.
- Baseline the numbers: Search Console clicks and impressions by section, indexed page count, Core Web Vitals status, and (new in 2026) impressions in the Search Console Generative AI performance report. You measure the audit's impact against these.
- Get access: Search Console (ideally a Domain property), Bing Webmaster Tools, analytics, the CDN dashboard and at least 30 days of server or CDN logs.
Sample by template, not by URL#
A site with a million URLs usually has fewer than 20 templates: home, category, product, article, author, tag, search, location and so on. Bugs live in templates, so audit templates.
- List the templates and map each to a URL pattern (
/p/*,/c/*,/blog/*). - Pull a sample per template: 50 to 200 URLs each, weighted toward high-traffic and high-revenue pages, plus a random slice from the long tail.
- Audit the sample deeply (rendered HTML, canonicals, structured data, internal links, CWV).
- Size the blast radius with full-site data: Search Console's Page indexing report filtered by sitemap, or a full crawl if the site allows it.
A finding written as "every product page canonicalizes to the parent category (118 of 118 sampled; about 410,000 URLs)" gets fixed. "Some canonicals look wrong" does not.
Phase 1: Crawl the site#
Run a crawl (Screaming Frog, Sitebulb or similar), ideally twice: with and without JavaScript rendering, then diff. Flag:
| Check | Issue signal |
|---|---|
| Status codes | 4xx from internal links, redirect chains or loops, 5xx clusters |
| Titles and meta | Missing, duplicate or truncated at scale (preview with the SERP preview tool) |
| Canonicals | Missing, pointing off-site by accident, canonical to a redirect or a 404 |
| Robots directives | noindex on templates that should rank; noindex in raw HTML that JavaScript later removes |
| Depth | Money pages more than 3 clicks deep; orphans (crawl vs sitemap diff) |
| Duplicates | Parameter URLs, http/https, trailing-slash variants all returning 200 |
| JS dependency | Links or content present only in the rendered crawl (JavaScript SEO) |
| HTML size | Key tags far down very large HTML documents |
Two 2025-2026 details matter here. Google documented in December 2025 that when it sees noindex it may skip rendering, so JavaScript that removes a noindex may never run. And Googlebot for Search fetches only the first 2MB of an HTML file (each CSS and JS file has its own 2MB limit), so put the title, meta robots, canonical and essential structured data near the top of the document.
Phase 2: Indexation reality#
Compare three numbers that should roughly agree: pages you intend to be indexed, pages in your sitemaps, and pages Google reports indexed (Search Console > Indexing > Pages).
- Indexed far below intended: dig into the exclusion reasons. The crawling and indexing guide covers each one. "Crawled - currently not indexed" at scale is usually a quality verdict, not a technical bug.
- Indexed far above intended: index bloat from parameters, tags, internal search results or staging leaks. Fix with
noindex, canonicals, robots.txt and by removing internal links to junk URLs. - Spot-check 10 to 20 critical URLs in URL Inspection: Google-selected canonical, last crawl date, and the rendered HTML.
- Segment sitemaps by template (
/sitemaps/products.xml,/sitemaps/articles.xml) so the Page indexing report can be filtered per template from then on.
Phase 3: Log-file analysis#
Crawlers from the outside see what could be crawled. Logs show what is crawled. For any site above a few thousand URLs, this is where the most valuable findings come from.
grep -i "googlebot" access.log \
| awk '{print $7}' \
| cut -d'?' -f1 \
| awk -F/ '{print "/"$2}' \
| sort | uniq -c | sort -rn | head -20What to look for:
- Crawl share by template vs value by template. If 60% of Googlebot hits go to filter URLs that earn 2% of clicks, that is your crawl-budget finding.
- Status codes served to bots: 5xx and 429 spikes slow crawling; long 3xx chains waste it.
- Pages never crawled in 30 days that you want indexed: usually a linking or sitemap problem.
- AI crawlers: count hits from
GPTBot,OAI-SearchBot,ChatGPT-User,ClaudeBot,Claude-SearchBot,PerplexityBotand others. Zero hits from a bot you allow is a finding in itself (see Phase 4).
Phase 4: The specialized passes#
Performance. Core Web Vitals status per URL group from Search Console, field data from PageSpeed Insights or CrUX Vis (the CrUX Looker Studio dashboard was shut down at the end of November 2025). Thresholds are unchanged: LCP 2.5 s, INP 200 ms, CLS 0.1 at the 75th percentile. Lighthouse 13 (October 2025) replaced the old audits with "insights" such as LCP breakdown and render-blocking requests, so update any checklist that still says "Defer offscreen images". Diagnose with the Core Web Vitals guide.
Structured data. Run sample URLs per template through the Rich Results Test (Google features) and the Schema Markup Validator (schema.org validity). Check three things:
- Markup matches visible content and is valid on every template that should carry it.
- No effort is going into retired features. FAQ rich results stopped showing on 2026-05-07, HowTo went in 2023, and Course Info, Estimated Salary, Learning Video, Special Announcement and Vehicle Listing were dropped in 2025. Leaving the markup is harmless; promising rich results from it is wrong.
- Search Console's Enhancements reports show errors at template scale.
See the structured data guide for what still earns a rich result. Structured data is not required to appear in AI Overviews or AI Mode, according to Google.
AI crawler access. This check did not exist in older audits, and it now fails often. Test three layers:
- robots.txt. Paste it into the AI crawler checker to see which bots are allowed. Remember the three tiers: training bots (
GPTBot,ClaudeBot), search bots (OAI-SearchBot,Claude-SearchBot,PerplexityBot) and user-triggered fetchers (ChatGPT-User,Perplexity-User,Google-Agent), most of which do not follow robots.txt. - CDN and WAF. Since 2025-07-01 Cloudflare blocks AI crawlers by default on new domains, and from 2026-09-15 it also blocks training and agent bots by default on pages that show ads for new domains. Many owners do not know their site blocks AI search bots. Compare the dashboard settings with your log counts.
- Search Console. Check Settings > Search generative AI. If someone set it to Exclude, the property is removed from AI Overviews, AI Mode and Discover generative AI features. Google-Extended in robots.txt does not control AI Overviews or AI Mode.
The AI Crawlers & Bot Access guide explains the policy decisions behind each setting.
International. hreflang reciprocity and self-references, if applicable (international SEO).
Security and hygiene. HTTPS everywhere, no mixed content, www/non-www and slash consistency enforced by single redirects. Also check third-party scripts for back button hijacking: Google made it an explicit spam policy, enforced from 2026-06-15, and it can come from ad or recommendation libraries rather than your own code.
Severity definitions#
Agree on these with stakeholders before the report, so "critical" means the same thing to everyone:
| Severity | Definition | Example | Expected response |
|---|---|---|---|
| Critical | Active harm at scale right now | noindex or Disallow on money templates, 5xx clusters, canonical to the homepage site-wide | Fix this sprint, hotfix if needed |
| High | Meaningful loss of traffic or crawl efficiency | Index bloat, money pages 5+ clicks deep, AI search bots blocked at the CDN | Next 1 to 2 sprints |
| Medium | Compounding hygiene | Redirect chains, duplicate titles on a template, invalid markup on eligible pages | Roadmap this quarter |
| Low | Polish | Missing alt text on decorative images, minor title truncation | Batch when convenient |
Then score within each band: impact × pages affected ÷ effort. The classic audit failure is shipping 80 equal-weight recommendations and watching none get implemented.
The deliverable#
A good audit report fits in three layers:
- One-page summary for decision makers: the 3 to 5 findings that matter, expected effect, and what you need (developer time, approvals).
- Findings table for the team:
id,severity,template,finding,evidence,pages_affected,fix,owner,effort,status
T-01,Critical,category,"noindex in category template","12/12 sampled; GSC: 3,140 excluded by noindex","3,140","Remove robots meta from layout; add CI check",web,S,open
T-02,High,all,"AI search bots blocked at CDN","0 OAI-SearchBot hits in 30d; Cloudflare AI bot block on","site-wide","Allow search-tier bots in CDN rules",infra,S,open- Appendix with crawl exports, log extracts and screenshots.
Phase 5: Verify the fixes#
An audit is not done when the report ships:
- Re-crawl after fixes deploy and confirm the issue class is gone across the sample.
- Use URL Inspection to request indexing for a few critical changed URLs, and resubmit sitemaps.
- Add a Search Console annotation for each major fix, so the effect is visible on the chart later.
- Compare against the baselines at +30 and +90 days. That is how you prove value and calibrate your own impact estimates (SEO Strategy & Stakeholder Reporting covers how to present it).
Run the full cycle quarterly for active sites, and always after a migration. Next: Programmatic SEO, then Site Migrations.