Indexing
What is indexing?
Indexing is the moment a search engine stores your page in its index, after crawling it and rendering it. Until that happens the page cannot appear in results, however good it is. Crawling and indexing are not the same thing: Google can crawl a page and still decide to leave it out — because it repeats another page, for example, or has too little on it.
Example
A haulage company launches a site with 40 pages. A month later the page indexing report in Search Console shows 12 indexed and 28 left out, for three different reasons:
- “Excluded by ‘noindex’ tag” — 9 pages. The tag is left over from the staging site, where it was there on purpose.
- “Blocked by robots.txt” — 6 pages. A rule written for the old site forbids a folder that now holds the services.
- “Crawled - currently not indexed” — 13 pages. Nothing technical is in the way here. Google has seen the pages and decided they are not worth storing: three sentences each, and often the same three sentences.
The first two are a day’s work. The third is not fixed with a button: those pages have to say something that is not already elsewhere.
Why it matters for a business
A page that is not indexed is not ranking fiftieth; it does not exist. It is also the cheapest problem in search to solve, because the cause is usually one line in one file.
Two details are worth knowing. Blocking an address in robots.txt stops crawling, but does not guarantee the address stays out of Google — and once a page is blocked, nobody can read a noindex tag inside it; the two do not combine. Second, Google crawls with its smartphone agent, so anything missing from the mobile version is effectively missing: mobile optimisation and indexing stopped being separate subjects long ago.
How to improve it
- A sitemap that lists only the addresses you want indexed, submitted in Search Console.
- A look at the page indexing report after every release; new exclusion reasons show up there first.
- One address per page, with the canonical pointing at it.
- Enough on the pages you want indexed, and a deliberate noindex on the ones you do not.
- On large sites, fewer pointless addresses from filters and sorting, so crawling is not spent on them.
What is blocking indexing is the first question in the SEO work we run.