Information sent with a request identifying the requesting software. User-agent strings can be imitated and do not independently verify a crawler’s identity.
A user agent is software acting on a user’s behalf, such as a web browser or crawler. In HTTP requests, the User-Agent header is a text field describing the requesting software. People often use “user agent” to mean that header value when discussing server logs or crawler behavior.
The header can help diagnose how different clients are accessing a site. For example, a developer might compare requests associated with a mobile browser and a crawler while investigating an error. The value can also be used in rules, which is why overly broad matching deserves careful review.
A user-agent string is self-reported. It is not an identity certificate, a password or evidence that a visitor has permission to access private material. A request that includes “Googlebot” or “bingbot” can be sent by software that does not belong to either organization. Use the crawler operator’s verification guidance when identity matters, and use proper authentication for protected resources.
Takeaway: User-agent data is useful for troubleshooting, but it should not be the sole basis for granting access or deciding that traffic is trustworthy. Treat it as one part of the evidence.
Example
A site owner notices many requests labeled as a major search crawler and considers bypassing a security rule for that name. Before making the change, the administrator verifies where those requests originate. The same text could have been copied into an unrelated script, so a name-only exception would be a weak control.
What to check
When sharing a crawl issue with a developer, include the URL, response status, time and relevant request details. Keep logs containing personal data private. Distinguish a diagnostic request that imitates a user agent from a verified visit by the real crawler.
For more detail, see User-Agent header.