OpenAI’s crawler for collecting web content that may be used to train its generative AI foundation models; distinct from OAI-SearchBot.
GPTBot is OpenAI’s crawler for content that may be used to train its generative AI foundation models. The phrase may be used is important: an observed request does not prove that a particular passage entered a training dataset or changed a model’s answers.
OpenAI distinguishes GPTBot from OAI-SearchBot, which supports discovery for ChatGPT search. Their robots.txt settings are independent. A site owner’s decision about training-related crawling therefore should not be treated as the same decision as search discovery or as a promise about recommendations.
For a business, the practical question is what use it intends to permit and which public content is involved. That decision should be documented rather than changed casually by an SEO plugin or a blanket bot-blocking setting. Existing content rights, sensitive information and the company’s publishing policy may require separate consideration.
Takeaway: Keep training-crawler choices distinct from search visibility choices. Verify the actual rules and do not claim that allowing GPTBot guarantees inclusion in answers or future recommendations.
Example
A publisher allows access to public product guides for search discovery but chooses to disallow GPTBot for training-related collection. Its developer implements the documented settings separately and checks that unrelated customer-facing access remains intact.
What to check
Use OpenAI’s current crawler documentation when checking identifiers and supported controls. If you inspect logs, verify the request identity rather than trusting a name alone. Do not assume a settings change erases prior information from trained models or removes every reference to the business across the web.
For more detail, see OpenAI crawler documentation.