Technical hygiene keeps improving on its own: HTTPS is near 91%, title tags near 99%. The interesting numbers are elsewhere. Blocking rates for AI crawlers rose sharply in a single year. And llms.txt turned up on 2% of sites, most of it generated by a plugin rather than chosen.
Table of contents
- What this is about
- What the source says
- What the data shows
- What it does not mean
- What to do
- Numbers at a glance
- FAQ
- Key takeaways
Key takeaways
- HTTPS hit 91%+ and title tags came in at nearly 99% across the Web Almanac’s crawl sample in 2025.
- GPTBot blocks climbed from 2.9% of desktop sites in 2024 to 4.5% in 2025, roughly 55% year-on-year. ClaudeBot blocks came close to doubling.
- A little over 2% of sites in the 2025 crawl had a valid llms.txt file, and 39.6% of those files trace back to All in One SEO.
- FAQPage schema adoption is rising even after Google reduced FAQ rich results. The Web Almanac authors read that as structured data being optimised for AI retrieval rather than SERP features.
- Nearly 33% of pages still lack canonical tags, and over 91% of iframes have no loading attribute set.
What this is about
The Web Almanac is an annual HTTP Archive project that crawls millions of pages and measures what is actually in the markup, not what practitioners say they do. The 2025 SEO chapter is the clearest population-level picture yet of how bot access decisions and structured data choices are shifting as LLM crawlers become routine visitors. If you manage a site and have not revisited your robots.txt since the AI crawlers arrived, this data is for you.
What the source says
Chris Green contributed to the 2025 Web Almanac SEO chapter and wrote up his analysis separately. His central argument is that the web is getting more technically sound almost by accident: CMS platforms and SEO plugins now enforce baseline hygiene without site owners having to think about it.
On robots.txt, his position is that the file has moved from crawl housekeeping to something closer to a policy document. He writes that the question “Which bots should we allow?” now carries downstream effects on budgets, revenue and users, not just on crawl budget.
On llms.txt, Green is careful to frame adoption as a signal of intent rather than proof of efficacy. He notes that at the start of 2025 he manually crawled the Majestic Million and found only 15 sites (0.015%) with the file. The Web Almanac’s broader crawl later found just over 2% adoption. A result he describes as surprising given his earlier baseline. He does not claim the file works; he treats its spread as evidence that site owners are making a new category of decision.
On FAQPage schema, he flags that adoption continued to rise even after Google said it was limiting FAQ rich results in the SERPs. His reading is that structured, extractable answers have value in AI retrieval contexts, though he acknowledges this could be correlation rather than causation.
What the data shows
The Web Almanac draws its crawl sample from the HTTP Archive. The 2025 edition covers millions of URLs across desktop and mobile. Green’s article doesn’t specify the exact page count for every metric, but the methodology is publicly documented and consistent year-on-year, so the directional comparisons hold up even where I’m not quoting absolute totals.
The bot-blocking figures are the sharpest numbers in the dataset, and they break down like this:
- GPTBot was blocked on 4.5% of desktop and 4.2% of mobile sites in 2025, up from 2.9% and 2.7% respectively in 2024.
- ClaudeBot was blocked on 3.6% of desktop and 3.4% of mobile sites in 2025, up from 1.9% and 1.6% in 2024.
- CCBot reached 3.5% desktop and 3.2% mobile in 2025, up from 2.7% and 2.4%.
- PetalBot reached 4.0% desktop and 4.4% mobile; it was not separately tracked in 2024.
All of these are robots.txt block rates, not crawl rates. A site blocking GPTBot in robots.txt is making a policy choice. Whether GPTBot honours it is a separate question, and this dataset does not answer it.
The llms.txt figures carry a methodological caveat worth stating plainly. Green compared two different crawl populations: his own January 2025 crawl of the Majestic Million (one million sites, backlink-ranked) against the Web Almanac’s crawl of roughly 16 million sites. He flags this himself. The jump from 0.015% to 2% is not a clean before-and-after on the same population, so the acceleration story is suggestive, not proven. The plugin attribution is a straight count from the Web Almanac data, which makes it the more reliable number: 39.6% of llms.txt files trace to All in One SEO, 3.6% to Yoast SEO. It tells you that a meaningful share of llms.txt files exist because a plugin generated them rather than because a developer made a considered choice.
The FAQPage schema trend is described directionally in the source but the specific year-on-year percentages are referenced as visible in Web Almanac charts rather than quoted as figures in Green’s article. I’m not going to invent those numbers. The directional claim. Rising adoption across three consecutive Web Almanac publications. Is what the source supports.
What it does not mean
The most common overreach I expect to see from this data is the claim that blocking LLM crawlers in robots.txt is a meaningful content-protection strategy, or conversely that allowing them guarantees citation.
Robots.txt is a request, not an enforcement mechanism. A crawler that ignores the file will still fetch your content. One that honours it will not. The Web Almanac data shows that site owners are increasingly making the policy choice, not that the choice has a measurable effect on how often their content appears in AI-generated answers. Those are different questions, and the dataset answers only the first one.
Similarly, the llms.txt adoption numbers tell you that plugins are generating the file at scale. They do not tell you that any answer engine reads it, which is the same gap we found when testing the three headline GEO claims, weights it, or changes retrieval behaviour because of it. Green says so directly. If you’re planning to implement llms.txt because 2% of sites now have one, you are following a crowd that is itself mostly following a plugin default, not a body of evidence.
The FAQPage correlation is the most tempting overreach of all. Rising schema adoption alongside rising AI citation of FAQ-style content is interesting. It is not evidence that adding FAQPage markup causes AI citation. The causal mechanism would require showing that answer engines retrieve structured data from the markup rather than from the text, and that the schema is what triggers the citation. The Web Almanac data does not show that.
What to do
These are ordered by effort, lowest first.
1. Audit your robots.txt for legacy bot entries. The Web Almanac data shows “msnbot” still appearing in the top five bot declarations long after it was replaced by 16 years ago. A 20-minute audit will tell you whether your file is managing real crawlers or managing ghosts. While you are there, make a deliberate decision about GPTBot, ClaudeBot and CCBot rather than living with a default. The AI Search section of this site covers what each crawler actually does with access once it has it.
2. Fix canonical coverage before adding new schema. Nearly 33% of pages lack canonical implementation. That gap has more certain downstream effects on how search engines (traditional and AI-backed) handle duplicate signals than any llms.txt file does. Close it first.
3. Review whether your llms.txt file was generated by a plugin default. If you have one and didn’t consciously create it, check what it says. A file generated without review may be pointing crawlers at content you’d rather they didn’t prioritise, or omitting content you’d want them to find.
4. Implement FAQPage schema where you have genuine Q&A content. It is the lowest-risk structured data available right now. The cost is small, and the downside if AI retrieval ignores it is nothing worse than a few lines of markup. The markup itself is what matters, not the hope of a SERP feature.
Numbers at a glance
Source: 2025 Web Almanac SEO chapter. Desktop figures shown; mobile figures in parentheses where they differ.
| Metric | 2024 | 2025 | Change |
|---|---|---|---|
| HTTPS adoption | — | 91%+ | — |
| Title tag adoption | — | ~99% | — |
| Canonical tag adoption | 65% | 67%+ | +2pp |
| Pages missing canonical | ~35% | ~33% | −2pp |
| Meta robots usage | 45.5% | 46.2% | +0.7pp |
| Robots.txt 404 rate | 14% | 13% | −1pp |
| GPTBot blocked (desktop) | 2.9% | 4.5% | +55% |
| ClaudeBot blocked (desktop) | 1.9% | 3.6% | +89% |
| CCBot blocked (desktop) | 2.7% | 3.5% | +30% |
| PetalBot blocked (desktop) | not tracked | 4.0% | — |
| Sites with valid llms.txt | ~0.015%† | ~2% | — |
| Images without loading attribute | — | ~67% | — |
| Iframes without loading attribute | — | 91%+ | — |
The llms.txt comparison is across different crawl populations (Majestic Million vs. ~16 million sites) and should be read as directional only. †The 0.015% figure is from the author’s own crawl of the Majestic Million at the start of 2025, not a 2024 Web Almanac measurement.
FAQ
Does blocking GPTBot in robots.txt stop OpenAI from using my content?
Robots.txt works by convention, not enforcement. Whether a crawler honours a disallow line is a separate question from whether the line exists, and this dataset only measures the line. Content already indexed before you added the block may remain in training data. The Web Almanac data shows 4.5% of desktop sites now block GPTBot. It measures the policy decision, not its enforcement outcome.
If 2% of sites have llms.txt, should I add one?
Only if you understand what the file does and have reviewed its contents. The Web Almanac data shows 39.6% of llms.txt files tied to All in One SEO and 3.6% to Yoast SEO. Plugins, in other words, are the main driver of adoption. The Web Almanac data does not show whether any answer engine reads the file or changes retrieval behaviour because of it. Implement it as a structured signalling attempt if you want to, but don’t treat adoption rate as evidence of effect.
