An HTTP response header carrying crawler directives. It can apply to HTML pages and other resources, including PDFs.
X-Robots-Tag is an HTTP response header used to deliver indexing and search-presentation rules. It can apply to an HTML page or to a resource such as a PDF that does not have an ordinary HTML head section. Its effect depends on the rule supplied and the search engine’s support for that rule.
The header is not part of the visible page text. Looking only at a page’s HTML source can therefore miss a restriction delivered by the server or another response-handling layer. This is an important distinction when a document opens normally but is unexpectedly excluded from search.
Use response-level rules with a clear scope. A rule meant for one class of files can affect a much larger set if its matching pattern is too broad. Before changing it, identify which responses receive the header and why. Public access, search visibility and confidentiality are separate questions, so an indexing header should not be treated as a security boundary.
Takeaway: Include HTTP headers in indexing investigations, especially for PDFs and other downloadable files. A clean HTML robots tag does not establish that the full response contains no restriction.
Example
A manufacturer wants an older public brochure to remain available through direct links but not appear independently in search. A scoped noindex response header may fit that search-visibility goal. Applying the same rule to all PDFs without review could also exclude current product documents that customers need to discover.
What to check
Inspect the actual final response for the exact file, not just a general server configuration screen. Check for multiple or conflicting rules and confirm the intended scope before saving a change. Private downloads still need proper access controls.
For more detail, see Robots meta tag and X-Robots-Tag specifications.