A file containing crawl instructions for compliant bots. Blocking crawling does not reliably prevent a URL from appearing in search results.
A robots.txt file gives participating crawlers instructions about which URL paths they may request. It is commonly used to manage crawling of parts of a site. These instructions are different from rules about whether a page should appear in a search index.
For Google, blocking a page in robots.txt can prevent the crawler from seeing its content, but the URL may still appear in search based on information found elsewhere. A crawl block can also prevent Google from reading a noindex directive on the page. Decide whether your aim is to limit requests, remove a page from search or protect private information before choosing a control.
Robots.txt is publicly accessible and depends on crawler cooperation. It does not keep a determined visitor out of a private file or replace a login. Be equally careful with public content: a broad restriction copied from a staging site can unintentionally stop compliant crawlers from reaching important pages after launch.
Takeaway: Review crawl instructions during launches and migrations. Protect public access to important content, while using proper authentication rather than robots.txt to secure customer or staff information.
Example
A designer launches a site with the broad crawl block used during development still in place. Customers can browse the pages, but compliant crawlers are instructed not to request them. The team should correct the unintended crawl restriction and inspect the outcome, not promise that rankings will disappear or recover on a fixed overnight schedule.
What to check
Test the exact path and the crawler group a rule applies to before saving a change. Keep a copy of the prior file and review the intended scope. Do not combine a crawl block with noindex while expecting the crawler to read the blocked page.
For more detail, see Introduction to robots.txt.