Technical SEO basics: how search engines find and index your pages

Executive overview

If search engines can't access or index your pages, nothing else in SEO matters. Five foundational elements control whether your pages make it into Google's index.

Getting indexing right is the prerequisite for all other SEO work.

Noindex tag

• Tells search engines to exclude a page from their index
• Often left on by accident after a site redesign or agency migration
• Check for stray noindex tags any time a site is rebuilt or migrated
• Some pages (e.g. author archive pages) are legitimately noindexed when they add user value but little SEO value

Robots.txt

• Lives at yourdomain.com/robots.txt; controls which crawlers can access which parts of your site
• Key directives: User-agent (which crawler) and Disallow (which paths to block)
• Setting User-agent: * and Disallow: / blocks all crawlers from the entire site — a common misconfiguration
• Subdomains each need their own robots file
• If you hit indexing issues, robots.txt is the first place to check

Sitemaps

• XML files listing the important URLs on your site (pages, images, videos)
• Help Google crawl your site more intelligently
• WordPress plugins (Yoast, RankMath) generate and maintain sitemaps automatically
• Submit via Google Search Console and reference in your robots.txt

Redirects

• Move visitors and crawlers from one URL to another
• Consolidate link signals from an old URL to a current one
• Use when merging similar pages (e.g. a 2018 best-of post redirecting to the current version)

Canonical tag

• HTML snippet that declares the preferred URL for a page
• Resolves duplicate content issues (e.g. HTTP vs HTTPS versions of the same page)
• Passes link signals to the canonical URL rather than splitting them across duplicates
• Google may override your canonical choice if it disagrees — e.g. preferring HTTPS over HTTP regardless of what you set