The process of automatically requesting web pages and other resources. Search engines discover URLs through links, sitemaps, and previously known pages.
Crawling is the automated retrieval of web pages and other resources. A search-engine crawler requests a URL and receives a response from the server, such as page content, a redirect or an error. That request gives the engine material to process; it does not by itself prove the page will be indexed or shown in search results.
A crawler needs a way to discover a URL. Links from other pages, a sitemap and URLs already known to the engine can help. A new page that is only reachable through a hard-to-find interaction may be less straightforward to discover than one linked from a relevant public page. Discovery and access are separate questions: an engine can know a URL exists but be unable or not permitted to retrieve its contents.
Access can be affected by server errors, authentication requirements or crawler rules. A robots.txt rule can restrict crawling, but it is not a reliable way to remove a known URL from Google’s results. Indexing controls serve a different purpose. Blocking a crawler can also prevent it from seeing a noindex instruction on the page.
Takeaway: Make important public pages reachable through useful links and check unexpected access problems. Do not assume a page has been crawled just because it opens in your own logged-in browser.
Example
A business publishes a new service page but leaves it disconnected from its services menu and sitemap. Adding a relevant internal link gives visitors and crawlers a route to it. If the server then returns an error, that is a separate problem to fix; adding more links will not repair the failed response.
What to check
Google Search Console’s URL Inspection tool can show information about Google’s last indexed version and can run a separate live test. Use the distinction when checking a recent fix: an old report may describe the earlier page, while a successful live test still does not guarantee indexing or rankings.
For more detail, see How Google Search works.