Search engines such as Google and Bing work in three broad stages: crawling, indexing and serving (ranking). Every page has to get through all three before it can appear for a search. Knowing the stages is the single most useful mental model in SEO, because almost every visibility problem can be traced to one of them.

Why this matters

Imagine a Melbourne physiotherapy clinic launches a new page for "sports physio Richmond". A month later it still doesn't appear in Google. The owner assumes the page "isn't optimised enough" and pays for more content. In reality, the page may never have been crawled because nothing links to it, or it may have been crawled but not indexed because it's almost identical to the clinic's Hawthorn page. More content won't fix either problem. Working through the stages in order saves time and money.

Stage 1: Crawling (discovering and fetching pages)

A crawler is software that requests web pages, much like a browser does. Google's is Googlebot; Bing's is Bingbot. Crawlers don't have a list of every page on the web. They find URLs in a few ways:

  • Following links from pages they already know. This is the main route, which is why internal links and links from other sites matter.
  • XML sitemaps submitted through Google Search Console or Bing Webmaster Tools, or listed in robots.txt.
  • Direct notifications, such as Bing's support for the IndexNow protocol, which lets a site tell participating engines that URLs have changed.

What stops or slows crawling

  • robots.txt disallow rules. The robots.txt file at the root of a domain tells compliant crawlers which paths not to fetch. A single misplaced line such as Disallow: / blocks the whole site. This happens surprisingly often after a website redesign, when a staging site's robots.txt is copied to production.
  • Server errors and slow responses. If a site often returns 5xx errors or times out, Google reduces how fast it crawls so it doesn't overload the server. Google calls this balance between demand and server capacity the crawl budget; it mainly matters for very large sites (tens of thousands of URLs and up), but server health matters for everyone.
  • No way in. An orphan page has no internal links pointing to it. If it also isn't in a sitemap, crawlers may never find it.
  • Login walls and forms. Crawlers don't sign in or submit search forms, so content that only appears after a login or a search box is effectively invisible.
  • Links that aren't real links. Crawlers reliably follow standard HTML anchor elements with an href attribute. Navigation that only works through JavaScript click handlers may not be followed.

Important nuance: robots.txt controls crawling, not indexing. A blocked URL can still appear in results (usually with no description) if other pages link to it. To keep a page out of the index, it must be crawlable and carry a noindex directive.

Stage 2: Indexing (understanding and storing pages)

After fetching a page, the search engine processes it. It reads the text and important elements such as the title, headings and links; it renders the page (runs its JavaScript, much like Chrome would) so content added by scripts can be seen; and it looks at images, video and structured data. Then it decides whether to store the page in its index, the enormous database results are drawn from.

Duplicates and canonicals

Many sites have several URLs showing the same content: with and without a trailing slash, with tracking parameters, printer versions, or the same product in several categories. Search engines group these duplicates and choose one as the canonical (representative) URL, which is the one shown in results. You can suggest your preferred version with a rel="canonical" link, consistent internal links and redirects, but the engine makes the final choice.

Why a crawled page may not be indexed

  • A noindex directive in a meta robots tag or an X-Robots-Tag HTTP header.
  • The page is a duplicate or near-duplicate, so another URL was chosen as canonical.
  • The page returns an error status (404, 410, 5xx) or redirects elsewhere.
  • Quality: the content is thin, auto-generated or adds little, so the engine decides it isn't worth storing. Google states plainly that it doesn't guarantee to crawl, index or serve any page, even one that follows its guidelines.

In Google Search Console, the Page indexing report and the URL Inspection tool show which of these applies, with statuses such as "Crawled – currently not indexed", "Duplicate without user-selected canonical" or "Excluded by 'noindex' tag".

Stage 3: Serving and ranking (answering a query)

When someone searches, the engine finds indexed pages that match and orders them. Google describes many signals feeding into this, which group into a few themes:

  • Meaning and relevance: does the page cover what the query is asking, including synonyms and related concepts, not just the exact words?
  • Quality: does the content appear helpful, reliable and trustworthy? Signals such as links from other reputable sites contribute here.
  • Usability: does the page work well on mobile, load reasonably quickly and avoid intrusive interstitials?
  • Context: the searcher's location, language, device and settings.

That last point is why there is no single ranking position. A search for "accountant" in Perth shows different results from the same search in Auckland or London. Results differ between mobile and desktop, and change from day to day as the web and the systems change. When someone sends a screenshot saying "we've dropped to #7", the first question is: searched from where, on what device?

A practical diagnostic routine

When an important page isn't performing, work through the stages in order:

  1. Can it be crawled? Check robots.txt, confirm the page returns a 200 status, and make sure at least one crawlable internal link points to it. Search Console's URL Inspection shows whether Google could fetch it.
  2. Is it indexed? URL Inspection reports the index status and which URL Google chose as canonical. A site: search (for example site:example.com.au/services/) is a rough check but not authoritative.
  3. Why isn't it ranking? Only now look at relevance, content quality, intent match, internal links and competition. Compare it honestly with the pages that do rank.

Worked example: an online homewares store in Brisbane finds its new "linen sheets" category missing from Google. URL Inspection shows "Excluded by 'noindex' tag". The developer had added noindex to all filtered category URLs, and the new category was built using a filter. Removing the tag from that template and requesting indexing fixed the problem within days. No content changes were needed.

Different engines, same principles

Google holds by far the largest search share in Australia, the UK, the US and most of Europe, but the stages are the same for Bing (which also powers several other search services), Yandex in Russia, Naver in South Korea and Baidu in China. If you work in markets where another engine matters, check its own webmaster documentation; the fundamentals of crawlable links, indexable content and helpful pages carry across.

Common mistakes and how to avoid them

Launching with the staging robots.txt. Development sites are usually blocked with Disallow: / so they don't appear in search. When the site goes live, that file is sometimes copied across. Add "check robots.txt and noindex" to every launch checklist, and look at Search Console's Page indexing report a few days after launch.

Relying on JavaScript-only links. Menus and filters built with buttons rather than links may look fine to visitors but give crawlers no URLs to follow. Ask developers to use real anchor elements with href attributes for anything that should be discoverable.

Blocking resources Google needs to render the page. If robots.txt blocks the CSS or JavaScript files, Google may not be able to render the page properly and may misjudge its content or mobile-friendliness. Allow crawling of the assets that build the page.

Using robots.txt to remove pages from results. Blocking a URL stops Google re-crawling it, so it can't see a noindex tag you add later. To remove a page, either return a 404 or 410 status, redirect it to a relevant replacement, or allow crawling with noindex. For urgent cases, Search Console's Removals tool hides a URL temporarily while you fix it properly.

Assuming "indexed" means "ranking". A page can be indexed and still receive no traffic because it doesn't match what people search for or competing pages are more helpful. Indexing is necessary, not sufficient.

Quick reference

Stage Question Main tools and signals
Crawl Can the engine fetch it? robots.txt, status codes, internal links, sitemaps
Index Will the engine store it? noindex, canonicals, duplicates, content quality
Rank Where does it appear? relevance, quality, usability, context

Key takeaways

  • Every page must be crawled, indexed and then ranked; problems at an earlier stage block everything after it.
  • robots.txt controls crawling; noindex controls indexing. Don't confuse them.
  • Search engines choose a canonical URL among duplicates; you can only suggest your preference.
  • Indexing is never guaranteed, even for good pages.
  • Rankings depend on context (location, device, language), so there's no single position.
  • Diagnose in order: crawl, then index, then rank. Search Console's URL Inspection is the fastest tool for the first two.