Indexing
The process of adding website pages to the search engine index. Without indexing, a page does not exist for search.
In brief
Indexing is the stage of search engine operation where a crawled page is analysed and its data is stored in the index. You can control indexing using sitemap.xml, the noindex tag, canonical, and Google Search Console tools.
What is indexing
Indexing is the process that follows crawling. If crawling is simply reading a page, indexing involves parsing the content, extracting key signals (headings, links, structured data), and storing them in the index. Only indexed pages can rank.
How to control indexing
- Sitemap.xml — tells the bot which pages are important for indexing.
- Meta robots (noindex) — prevents indexing of a specific page (useful for carts, filters, admin sections).
- Canonical — points to the main version among duplicates, consolidating indexing signals.
- X‑Robots-Tag — controls indexing via HTTP headers (e.g., for PDFs or images).
- GSC → URL Inspection — manual request for indexing or re‑indexing.
Indexing control tools
- Google Search Console: Coverage report, URL Inspection tool, submit to index.
- Yandex.Webmaster: similar features for Yandex.
- Screaming Frog SEO Spider (with GSC API integration) — bulk indexing status checks.
Indexing is not instantaneous. Even after submitting via GSC, it can take from a few hours to several days. For new sites, the first pages may take weeks to be indexed.
FAQ
Common questions
Crawling is the process of a robot visiting pages to read content. Indexing is analysing and adding the page to the index. A page can be crawled but not indexed (e.g., due to noindex or low quality).
Use GSC for manual URL submission, add pages to sitemap.xml, create internal links from already indexed pages, and build external links.
The noindex tag does not block crawling. The bot may still visit the page, see noindex, and drop it from the index. To completely block access, use robots.txt (but then noindex won‘t be read).
Direct contacts
Discuss your project?
Share your goals and website context — I will suggest a practical next step.