The process of automatically requesting web pages and other resources. Search engines discover URLs through links, sitemaps, and previously known pages.

CrawlingDiscover a URLRequest theresourceReceive aresponse
Links and sitemaps help crawlers find URLs to request; a fetch does not guarantee indexing.

Crawling is the automated retrieval of web pages and other resources. A search-engine crawler requests a URL and receives a response from the server, such as page content, a redirect or an error. That request gives the engine material to process; it does not by itself prove the page will be indexed or shown in search results.

A crawler needs a way to discover a URL. Links from other pages, a sitemap and URLs already known to the engine can help. A new page that is only reachable through a hard-to-find interaction may be less straightforward to discover than one linked from a relevant public page. Discovery and access are separate questions: an engine can know a URL exists but be unable or not permitted to retrieve its contents.

Access can be affected by server errors, authentication requirements or crawler rules. A robots.txt rule can restrict crawling, but it is not a reliable way to remove a known URL from Google’s results. Indexing controls serve a different purpose. Blocking a crawler can also prevent it from seeing a noindex instruction on the page.

Takeaway: Make important public pages reachable through useful links and check unexpected access problems. Do not assume a page has been crawled just because it opens in your own logged-in browser.

Example

A business publishes a new service page but leaves it disconnected from its services menu and sitemap. Adding a relevant internal link gives visitors and crawlers a route to it. If the server then returns an error, that is a separate problem to fix; adding more links will not repair the failed response.

What to check

Google Search Console’s URL Inspection tool can show information about Google’s last indexed version and can run a separate live test. Use the distinction when checking a recent fix: an old report may describe the earlier page, while a successful live test still does not guarantee indexing or rankings.

For more detail, see How Google Search works.