Technical SEOPublished26.05.2026

X-Robots-Tag (HTTP header for robots)

X-Robots-Tag is an HTTP header for controlling indexing and crawling of pages and files. It works like the robots meta tag but applies to any file type: PDFs, images, videos.

In brief

X-Robots-Tag is an HTTP response header that controls search robot behavior regarding indexing and crawling of a resource. It supports the same directives as the robots meta tag (noindex, nofollow, noarchive, etc.), but is applied at the server level.

What is X-Robots-Tag

X-Robots-Tag is an HTTP header that the server adds to a resource response. It controls search robot behavior: allowing or prohibiting indexing, link following, page caching, and other actions.

The main advantage of X-Robots-Tag over the <meta name="robots"> tag is that it works at the HTTP response level and can be applied to any resource type: HTML pages, PDF documents, images, video files, and any resource that has no HTML <head> tag.

X-Robots-Tag and the robots meta tag use identical directives. If both are present, the more restrictive directive applies. Since robots must download the page to read the header, X-Robots-Tag doesn't replace robots.txt for blocking crawling.

Supported directives

noindex: Prevents the page from being added to the search index.
nofollow: Prevents the robot from following links on the page.
noarchive: Prevents Google from caching the page and showing a cache link.
nosnippet: Prevents display of a text or video snippet in search results.
none: Equivalent to noindex + nofollow — fully blocks indexing and link following.
max-snippet:N: Limits the text snippet length to N characters.
max-image-preview:[none|standard|large]: Controls the size of image previews in search results.

X-Robots-Tag vs. robots meta tag

Parameter	X-Robots-Tag	Robots meta tag
Application level	HTTP header (server)	HTML tag (page)
Resource types	Any (HTML, PDF, images)	HTML pages only
How to add	Server config / .htaccess / code	Tag in <head>
Read by robot	Before page body is loaded	After HTML is loaded and parsed
Scope	Can be applied globally via URL patterns	Individually per page

When to use X-Robots-Tag

PDFs and documents: block technical PDFs, price lists, or contracts from indexing via server header
Images: restrict preview display via max-image-preview:none for copyright protection
Staging builds: block test environment from indexing with X-Robots-Tag: noindex at server level
API responses: add noindex to JSON endpoints that Google may attempt to index
Archival pages: block caching with noarchive without removing the page from the index

How to add X-Robots-Tag

Apache (.htaccess): Header set X-Robots-Tag "noindex, nofollow" — globally or conditionally
Nginx: add_header X-Robots-Tag "noindex"; in the location or server block
PHP: header('X-Robots-Tag: noindex'); before any output
Node.js/Next.js: via middleware or in the headers configuration in next.config.ts
CDN: Cloudflare Workers or Rules allow adding headers without modifying the server

FAQ

Common questions

robots.txt blocks crawling — the robot doesn't download the page at all. X-Robots-Tag blocks indexing — the robot downloads the page, reads the header, and doesn't add it to the index. For PDFs, X-Robots-Tag is more effective since robots.txt cannot control specific content types.

Yes. In Nginx and Apache, set the header via location/Directory blocks or RewriteRule with URL conditions. In application code (PHP, Node.js) — programmatically, based on routing logic.

Google and Bing fully support it. Yandex also supports X-Robots-Tag with the same directives. Other smaller crawlers may not respect it. Always verify behavior in the respective webmaster tool.

Both are considered, and the more restrictive directive applies. If X-Robots-Tag says noindex but the meta tag says index — the page will not be indexed.

Direct contacts

Discuss your project?

Share your goals and website context — I will suggest a practical next step.

[email protected]@pavelbarushka LinkedIn