Crawling, indexing and structured data: where sites actually break

What this is about. Three things have to happen before any of the clever work matters: a crawler has to fetch the page, an index has to keep it, and a machine has to be able to read what it says. This page covers that layer, with the documented limits and thresholds in one place.

On this page

• Where do sites actually break: crawl, index or serve?
• Sitemaps: the limits, and what lastmod is worth
• Does structured data still do anything?
• Speed is a threshold, not a score
• Push protocols, and who listens

Crawl, index, serve — and where sites break

Crawling is a fetch. Indexing is a decision to keep what was fetched. Serving is a decision to show it for a given query. They fail independently, and the fix differs at each stage, which is why “we are not ranking” is not a diagnosis.

In practice the failures cluster in a short list. A page blocked in robots.txt that also carries a noindex tag: the crawler never reads the tag, so the page can linger as a URL-only entry. A canonical pointing somewhere the site owner did not intend, quietly consolidating the wrong page. Body text that arrives only after JavaScript executes. Parameters generating an unbounded set of URLs that consume crawl capacity and index nothing.

None of that is exotic, and all of it is visible in server logs and in Search Console’s page indexing report. Read those two before changing anything else.

Sitemaps: the limits, and what lastmod is worth

The format is old and the rules are stable. One file, at most 50 000 URLs, at most 50 MB before compression. Past either limit, split into several files and list them in a sitemap index. Dates go in W3C Datetime format, which means a timezone, not a bare date, if you want the time to be read.

What a sitemap does is narrow: it tells a crawler which URLs you consider canonical and roughly when they last changed. It does not push, it does not rank, and inclusion is not a promise of indexing. Its value is highest on large sites and on sites with weak internal linking, which is another way of saying that a good sitemap is a partial patch for a navigation problem.

The lastmod field is worth stating carefully because it is widely misused. It is a hint. Google’s documentation says an inaccurate value is ignored rather than penalised, and that a demonstrably unreliable value can make the field untrusted for the whole file. So the failure mode is not a penalty — it is being tuned out. Reserve the field for changes to the page’s own content, and never let a site-wide template edit rewrite every date at once.

The news sitemap

A Google News sitemap uses an extra namespace and carries publication name, language, publication date and title. It should list only articles from the last two days; older entries are simply ignored. It sits alongside the normal sitemap rather than replacing it, and it matters only if you publish on a news cadence.

Does structured data still do anything?

Structured data earns specific, documented search features. It is not a ranking factor and it is not, on any published evidence, a citation factor. The useful question is therefore narrow: what does this type actually produce today?

Type	What it is for	Worth the effort?
Article / NewsArticle	Describes the piece, its author, its dates	Yes. Cheap, and the author link is the part that carries weight.
BreadcrumbList	Breadcrumb trail in results	Yes. Visible, and trivial to generate.
FAQPage	Question and answer pairs	Rich results are now limited to a narrow set of authoritative sites. Keep it for the machine-readable pairs, not for the display.
Product, Recipe, Event, JobPosting	Category-specific rich results	Yes, where the content genuinely is that thing.
Dataset	Describes a published data set	Underused. If you publish your own measurements, this is the type that says so.
Person / Organization with sameAs	Ties an author or brand to identities elsewhere	Yes. This is the practical half of the authorship story.

Two rules keep this from going wrong. Mark up what is actually on the page, visible to the reader. And validate, because a malformed block earns nothing and a wrong one can earn a manual action.

Speed is a threshold, not a score

Core Web Vitals are assessed against fixed thresholds on field data, at the 75th percentile of page loads. You pass or you do not. Getting well under a threshold buys you nothing in ranking terms, which is worth saying to anyone chasing a synthetic score of 100.

Metric	Good	Needs improvement	Poor
LCP — largest contentful paint	≤ 2.5 s	2.5–4.0 s	> 4.0 s
INP — interaction to next paint	≤ 200 ms	200–500 ms	> 500 ms
CLS — cumulative layout shift	≤ 0.1	0.1–0.25	> 0.25

For a content site the three fixes that move these are unglamorous: serve the main image at the size it is displayed and do not lazy-load the one above the fold, reserve space for anything that arrives late, and ship less JavaScript. A content delivery network helps latency, which helps LCP, and it does nothing for the other two.

Push protocols, and who listens

A sitemap waits to be read. IndexNow is a push: you send a URL and the engine is told immediately. It is supported by Bing, Yandex, Seznam and Naver. Google does not consume it. So the answer for a publishing site is both — IndexNow for near-instant pickup where it is supported, a sitemap for everyone else.

Push only what changed. Sending your whole URL set on every deploy is the same mistake as rewriting every lastmod: it turns a signal into noise, and noise gets discounted.

FAQ