Research

Bot Appétit State of the Web & AI Readiness

How many of the sites we scan let our scanner in, and what they get right and wrong, check by check. Percentages only, refreshed every hour.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Headlines

  • 100% of BigCommerce sites pass the “Search Console / Bing verification” check, against 0% of Carrd sites. See the detail

  • The check sites fail most often is “llms.txt is present and valid”: 77% of the sites we could read fail it. See the detail

Can AI read the web?

In November 2026, our scanner could read 91% of the sites we scanned. The other 9% turned it away.

Sites our scanner could read 91% All sites
Clear all six readiness gates 9% All sites
Publish structured data (JSON-LD) 45% All sites

This is our scanner, not every AI. A site that turns it away may still let other visitors in; how sites treat AI crawlers is in AI access, below.

How sites land

Every page we read gets a score out of 100, the number at the top of its report (how scoring works). Each row is a range of scores. Where too few sites score in the lowest ranges, they are joined into one row. A site our scanner could not read has no score.

  • 91–100 21%
  • 81–90 38%
  • 71–80 19%
  • 61–70 8%
  • 0–60 4%
  • Turned our scanner away 9%
Show as a table
Share of sites in each score range
ScoreShare of sites
91–10021%
81–9038%
71–8019%
61–708%
0–604%
Turned our scanner away9%

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Embed or share this figure : sites our scanner could read
In November 2026, our scanner could read 91% of the sites we scanned. The other 9% turned it away.
Embed style

AI access

In November 2026, 14% of the sites we scanned disallowed some AI crawlers in robots.txt, and 9% turned our scanner away at the server.

  • robots.txt disallows some AI crawlers 14%
  • robots.txt disallows major AI crawlers 4%
  • The server turned our scanner away 9%
Show as a table
How sites treat AI crawlers
What we foundShare of sites
robots.txt disallows some AI crawlers14%
robots.txt disallows major AI crawlers4%
The server turned our scanner away9%
robots.txt allows AI crawlers, the server turned our scanner away3%
Embed or share this figure : AI access
In November 2026, 14% of the sites we scanned disallowed some AI crawlers in robots.txt, and 9% turned our scanner away at the server.
Embed style
3% robots.txt allows AI crawlers, the server turned our scanner away
Embed or share this figure : robots.txt allows AI crawlers but the server turned our scanner away
In November 2026, 3% of the sites we scanned allow AI crawlers in robots.txt and still turned our scanner away at the server.
Embed style

These shares are of every site we scanned, including the ones that turned our scanner away. The shares in Every check are of the sites we could read, so the same crawler check can read differently there.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Where sites differ

In November 2026, here is where the sites we scanned differ most.

By platform

In November 2026, the “AI bot rules are explicit and intentional” check showed the widest gap between platforms: 100% of BigCommerce sites passed it, against 0% of Carrd sites.

How sites on each platform treat AI crawlers in robots.txt

  • Astro 32%
  • BigCommerce 5%
  • Carrd 0%
  • Eleventy 13%
  • Framer 5%
  • Ghost 14%
  • Jekyll 7%
  • Netlify 6%
  • Next.js 19%
  • Shopify 5%
  • SvelteKit 35%
  • Vercel 2%
  • WordPress 29%
Show as a table
How sites on each platform treat AI crawlers in robots.txt
PlatformDisallow some AI crawlers in robots.txt
Astro32%
BigCommerce5%
Carrd0%
Eleventy13%
Framer5%
Ghost14%
Jekyll7%
Netlify6%
Next.js19%
Shopify5%
SvelteKit35%
Vercel2%
WordPress29%

A platform is only known for pages our scanner could read, so this table cannot say how many sites on each platform turned our scanner away.

The checks that differ most between platforms, by the share of sites that pass. Each share is over the sites the check applies to.

AI bot rules are explicit and intentional Astro41% BigCommerce100% Carrd0% Eleventy18% Framer10% Ghost22% Jekyll7% Netlify6% Next.js— Shopify15% SvelteKit38% Vercel7% WordPress33%
Search Console / Bing verification Astro92% BigCommerce100% Carrd0% Eleventy51% Framer80% Ghost83% Jekyll51% Netlify67% Next.js— Shopify98% SvelteKit81% Vercel53% WordPress69%
Structured data is present Astro62% BigCommerce35% Carrd0% Eleventy25% Framer37% Ghost97% Jekyll86% Netlify24% Next.js— Shopify80% SvelteKit55% Vercel41% WordPress79%
Clean content for agents (Markdown) Astro14% BigCommerce97% Carrd0% Eleventy4% Framer85% Ghost0% Jekyll8% Netlify4% Next.js— Shopify5% SvelteKit0% Vercel3% WordPress5%
Sitemap dates tell a machine what changed Astro— BigCommerce— Carrd100% Eleventy64% Framer3% Ghost94% Jekyll— Netlify— Next.js— Shopify3% SvelteKit— Vercel— WordPress—

Which industries are most AI-ready

In November 2026, Shopping & retail sites were the most AI-ready of the industries we could compare: 22% of them cleared every readiness gate, against 6% of Education & research sites.

The share of sites in each industry that clear every readiness gate. Each industry has its own page.

  • Shopping & retail 22%
  • Food & drink 18%
  • Software & SaaS 15%
  • Agencies & professional services 13%
  • Finance & insurance 12%
  • News & media 9%
  • Entertainment & games 8%
  • Government & nonprofit 8%
  • Community & personal 7%
  • Education & research 6%
Show as a table
Share of sites in each industry that clear every readiness gate
IndustryClear every readiness gate
Shopping & retail22%
Food & drink18%
Software & SaaS15%
Agencies & professional services13%
Finance & insurance12%
News & media9%
Entertainment & games8%
Government & nonprofit8%
Community & personal7%
Education & research6%

An industry is only known for pages our scanner could read, so this table cannot say how many sites in each industry turned our scanner away.

Visitors versus research lists

In November 2026, 13% of the sites visitors scanned turned our scanner away, against 9% of the sites on our research lists.

Lists we scan for research are not a random sample of the web either.

  • 91–100 Scanned by visitors: Not enough sites yet Research lists: 21%
  • 81–90 Scanned by visitors: 35% Research lists: 38%
  • 0–80 Scanned by visitors: 30% Research lists: 32%
  • Turned our scanner away Scanned by visitors: 13% Research lists: 9%
Show as a table
Share of sites in each score range, scanned by visitors and on research lists
ScoreScanned by visitorsResearch lists
91–100—21%
81–9035%38%
0–8030%32%
Turned our scanner away13%9%

The checks where the two groups differ most, by the share of sites that pass. Each share is over the sites the check applies to.

Share of sites passing, scanned by visitors and on research lists, for the checks that differ most
CheckScanned by visitorsResearch lists
Has a meta description61%84%
The site settles on one host (www vs non-www)69%87%
Right schema type for the page84%66%
Organization schema with sameAs32%15%
Title fits a search snippet58%41%
Search Console / Bing verification83%67%
Author / publisher is clear51%37%
security.txt is published18%5%
llms.txt is present and valid24%14%
This page declares its canonical URL72%63%

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Every check

In November 2026, several checks tie as the ones sites passed most often, each at 100%. “llms.txt is present and valid” was the one they failed most often: 77% of the sites we could read failed it.

A dash means not enough sites yet. Each check links to what it is and what good looks like.

Applies to is the share of the sites we could read that the check ran on, because some checks have nothing to judge on many pages.

The other columns are shares of those sites, not of every site. Newer checks only cover sites scanned since the check was added.

“Automated visitors can reach your pages” shows a dash: a site passes it whenever our scanner could read the page, so it only repeats whether we got in. See how many turned us away.

Fixed and Broke compare re-scanned sites, and those columns are left out until there are enough of them. Not enough history yet.

AEO / content

AEO / content: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Your content survives without JavaScript Applies to 94% Pass 89% Worth a look 2% To fix 4% Couldn't judge 4%
Your answer is in the HTML Applies to 100% Pass 86% Worth a look 7% To fix 7% Couldn't judge 0%
Semantic HTML landmarks Applies to 100% Pass 53% Worth a look 47% To fix 0% Couldn't judge 0%
Content is scannable in self-contained chunks Applies to 100% Pass 78% Worth a look 22% To fix 0% Couldn't judge 0%
Author / publisher is clear Applies to 100% Pass 38% Worth a look 62% To fix 0% Couldn't judge 0%
A machine-readable published / updated date Applies to 100% Pass 13% Worth a look 87% To fix 0% Couldn't judge 0%
llms.txt is present and valid Applies to 100% Pass 15% Worth a look 5% To fix 77% Couldn't judge 3%
Clean content for agents (Markdown) Applies to 100% Pass 9% Worth a look 88% To fix 0% Couldn't judge 3%

AI access

AI access: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Buttons, links and fields have names Applies to 86% Pass 50% Worth a look 45% To fix 4% Couldn't judge 0%
AI crawlers aren't blocked Applies to 100% Pass 83% Worth a look 9% To fix 3% Couldn't judge 5%
Automated visitors can reach your pages See how many turned us away Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Engines are allowed to quote the page Applies to 93% Pass 99% Worth a look 1% To fix <1% Couldn't judge 0%
Your page arrives quickly Applies to 93% Pass 94% Worth a look 4% To fix 2% Couldn't judge 0%

AI signals

AI signals: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
AI bot rules are explicit and intentional Applies to 100% Pass 19% Worth a look 77% To fix 0% Couldn't judge 5%

Structured data

Structured data: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Structured data is present Applies to 100% Pass 45% Worth a look 0% To fix 55% Couldn't judge 0%
Structured data is valid Applies to 43% Pass 97% Worth a look 1% To fix 2% Couldn't judge 0%
FAQ / Q&A schema Applies to 8% Pass 36% Worth a look 64% To fix 0% Couldn't judge 0%
Organization schema with sameAs Applies to 100% Pass 16% Worth a look 16% To fix 68% Couldn't judge 0%
Right schema type for the page Applies to 42% Pass 69% Worth a look 31% To fix 0% Couldn't judge 0%
Schema facts also appear in the page text Applies to 10% Pass 53% Worth a look 47% To fix 0% Couldn't judge 0%
Breadcrumb structured data Applies to 5% Pass 13% Worth a look 87% To fix 0% Couldn't judge 0%

SEO

SEO: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Reached without a redirect chain Applies to 100% Pass 98% Worth a look 2% To fix 0% Couldn't judge 0%
Caching or validation headers are set Applies to 100% Pass 92% Worth a look 8% To fix 0% Couldn't judge 0%
The site settles on one host (www vs non-www) Applies to 100% Pass 86% Worth a look 6% To fix 4% Couldn't judge 4%
robots.txt is present and not over-blocking Applies to 100% Pass 79% Worth a look <1% To fix 20% Couldn't judge 1%
Not accidentally set to noindex Applies to 100% Pass 99% Worth a look 0% To fix 1% Couldn't judge 0%
This page declares its canonical URL Applies to 100% Pass 64% Worth a look 8% To fix 28% Couldn't judge 0%
Has a title tag Applies to 100% Pass 99% Worth a look 0% To fix 1% Couldn't judge 0%
Has a meta description Applies to 100% Pass 82% Worth a look 2% To fix 16% Couldn't judge 0%
The page has an H1 Applies to 100% Pass 63% Worth a look 1% To fix 36% Couldn't judge 0%
Logical heading order Applies to 100% Pass 60% Worth a look 27% To fix 1% Couldn't judge 12%
Topic is consistent across title, H1 and description Applies to 100% Pass 47% Worth a look 13% To fix 0% Couldn't judge 40%
Images have descriptive alt text Applies to 83% Pass 82% Worth a look 12% To fix 6% Couldn't judge 0%
Your page's HTML isn't oversized Applies to 93% Pass 97% Worth a look 1% To fix 1% Couldn't judge 0%
Open Graph tags for sharing Applies to 100% Pass 48% Worth a look 28% To fix 24% Couldn't judge 0%
XML sitemap is present and referenced Applies to 100% Pass 60% Worth a look 10% To fix 21% Couldn't judge 8%
Missing pages return a real 404 Applies to 100% Pass 85% Worth a look 0% To fix 12% Couldn't judge 3%
Language and charset declared Applies to 100% Pass 94% Worth a look 7% To fix 0% Couldn't judge 0%
hreflang for language variants Applies to 18% Pass 99% Worth a look 1% To fix 0% Couldn't judge 0%
No meta-refresh redirect Applies to 100% Pass 99% Worth a look 1% To fix 0% Couldn't judge 0%

Links

Links: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
No broken links Applies to 90% Pass 89% Worth a look 9% To fix 1% Couldn't judge 1%
Links don't point at redirects Applies to 83% Pass 85% Worth a look 14% To fix 0% Couldn't judge 2%
Descriptive anchor text Applies to 90% Pass >99% Worth a look <1% To fix 0% Couldn't judge 0%
Links are real links Applies to 84% Pass 81% Worth a look 18% To fix 1% Couldn't judge 0%

Performance (not scored)

Not enough sites yet

Performance (not scored): share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Largest Contentful Paint (LCP) Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Cumulative Layout Shift (CLS) Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Interaction to Next Paint (INP) Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Page weight and render-blocking Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —

Security

Security: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Served over a secure HTTPS connection Applies to 100% Pass >99% Worth a look 0% To fix <1% Couldn't judge <1%
HTTP redirects to HTTPS Applies to 100% Pass 88% Worth a look 4% To fix 7% Couldn't judge 2%
No mixed (insecure) content Applies to 100% Pass 99% Worth a look <1% To fix <1% Couldn't judge <1%
Not flagged for malware or phishing Applies to 96% Pass 100% Worth a look 0% To fix 0% Couldn't judge 0%

Advanced (not scored)

Advanced (not scored): share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Content usage preferences declared Applies to 100% Pass 5% Worth a look 90% To fix 0% Couldn't judge 5%
Resource-discovery Link headers Applies to 100% Pass 2% Worth a look 98% To fix 0% Couldn't judge 0%
MCP Server Card Applies to 100% Pass 1% Worth a look 92% To fix 0% Couldn't judge 7%
AI catalog Applies to 94% Pass 1% Worth a look 93% To fix 0% Couldn't judge 6%
A2A Agent Card Applies to 93% Pass 1% Worth a look 93% To fix 0% Couldn't judge 6%
WebMCP tools in the page Applies to 93% Pass 2% Worth a look 98% To fix 0% Couldn't judge 0%
AI usage preferences (Content-Usage) Applies to 93% Pass <1% Worth a look 95% To fix 0% Couldn't judge 5%
Licensing terms for AI (RSL) Applies to 93% Pass <1% Worth a look 95% To fix 0% Couldn't judge 5%
Text-and-data-mining reservation (TDMRep) Applies to 93% Pass <1% Worth a look 94% To fix 0% Couldn't judge 6%
Markdown version linked from the page Applies to 93% Pass 1% Worth a look 99% To fix 0% Couldn't judge 0%
Content signals agree everywhere Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Agent Skills index Applies to 100% Pass 1% Worth a look 92% To fix 0% Couldn't judge 6%
API catalog Applies to 100% Pass 1% Worth a look 93% To fix 0% Couldn't judge 6%
OAuth authorization-server metadata Applies to 100% Pass 4% Worth a look 89% To fix 0% Couldn't judge 7%
OAuth protected-resource metadata Applies to 100% Pass 4% Worth a look 90% To fix 0% Couldn't judge 7%
OpenID Connect discovery Applies to 100% Pass 4% Worth a look 89% To fix 0% Couldn't judge 7%
Web Bot Auth (operates a signed bot) Applies to 100% Pass <1% Worth a look 93% To fix 0% Couldn't judge 7%
Universal Commerce Protocol manifest Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Auth.md agent registration Applies to 100% Pass 1% Worth a look 94% To fix 0% Couldn't judge 6%
AI discovery via DNS Applies to 100% Pass <1% Worth a look 99% To fix 0% Couldn't judge 1%

Good practice (not scored)

Good practice (not scored): share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Core security headers are set Applies to 100% Pass 15% Worth a look 57% To fix 28% Couldn't judge 0%
No server version leakage Applies to 100% Pass 90% Worth a look 10% To fix 0% Couldn't judge 0%
security.txt is published Applies to 100% Pass 6% Worth a look 89% To fix 0% Couldn't judge 5%
Mobile viewport is set Applies to 100% Pass 85% Worth a look 12% To fix 2% Couldn't judge 0%
Title fits a search snippet Applies to 99% Pass 42% Worth a look 58% To fix 0% Couldn't judge 0%
Description fits a search snippet Applies to 78% Pass 55% Worth a look 45% To fix 0% Couldn't judge 0%
Twitter/X card tags Applies to 100% Pass 62% Worth a look 38% To fix 0% Couldn't judge 0%
Images are optimised Applies to 83% Pass 22% Worth a look 78% To fix 0% Couldn't judge 0%
Analytics is installed Applies to 100% Pass 61% Worth a look 39% To fix 0% Couldn't judge 0%
Search Console / Bing verification Applies to 100% Pass 68% Worth a look 32% To fix 0% Couldn't judge <1%
A favicon is set Applies to 100% Pass 89% Worth a look 12% To fix 0% Couldn't judge 0%
Claims come with figures, quotes or sources Applies to 53% Pass 51% Worth a look 49% To fix 0% Couldn't judge 0%
Section headings are phrased as questions Applies to 47% Pass 22% Worth a look 78% To fix 0% Couldn't judge 0%
Reading the page is affordable for a machine Applies to 88% Pass 84% Worth a look 16% To fix 0% Couldn't judge 0%
No hidden text written for an AI model Applies to 88% Pass 100% Worth a look 0% To fix 0% Couldn't judge 0%
Sitemap dates tell a machine what changed Applies to 53% Pass 57% Worth a look 43% To fix 0% Couldn't judge 0%
The links in llms.txt resolve Applies to 12% Pass 93% Worth a look 6% To fix 0% Couldn't judge 1%
Sensible internal/external link mix Applies to 90% Pass 88% Worth a look 12% To fix 0% Couldn't judge 0%

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

What goes together

In November 2026, these were the pairs of checks that sites tended to pass together.

This is an association between checks, not a cause: passing one does not make a site pass the other. It shows what tends to sit together on a site that has been looked after.

  • Sites that pass the “Structured data is present” check are 8 times as likely to pass the “Author / publisher is clear” check too. 74% of the sites that pass the first check pass this one, against 9% of the sites that do not.

  • Sites that pass the “Core security headers are set” check are 6.5 times as likely to pass the “security.txt is published” check too. 23% of the sites that pass the first check pass this one, against 3% of the sites that do not.

  • Sites that pass the “Structured data is present” check are 6 times as likely to pass the “A machine-readable published / updated date” check too. 24% of the sites that pass the first check pass this one, against 4% of the sites that do not.

  • Sites that pass the “Has a meta description” check are 5.5 times as likely to pass the “Open Graph tags for sharing” check too. 56% of the sites that pass the first check pass this one, against 11% of the sites that do not.

  • Sites that pass the “Content usage preferences declared” check are 5 times as likely to pass the “AI bot rules are explicit and intentional” check too. 84% of the sites that pass the first check pass this one, against 16% of the sites that do not.

  • Sites that pass the “llms.txt is present and valid” check are 5 times as likely to pass the “Organization schema with sameAs” check too. 49% of the sites that pass the first check pass this one, against 10% of the sites that do not.

  • Sites that pass the “Author / publisher is clear” check are 4.5 times as likely to pass the “A machine-readable published / updated date” check too. 24% of the sites that pass the first check pass this one, against 6% of the sites that do not.

  • Sites that pass the “This page declares its canonical URL” check are 4.5 times as likely to pass the “llms.txt is present and valid” check too. 21% of the sites that pass the first check pass this one, against 5% of the sites that do not.

  • Sites that pass the “Content is scannable in self-contained chunks” check are 4.5 times as likely to pass the “Semantic HTML landmarks” check too. 64% of the sites that pass the first check pass this one, against 15% of the sites that do not.

  • Sites that pass the “Your answer is in the HTML” check are 4.5 times as likely to pass the “Semantic HTML landmarks” check too. 59% of the sites that pass the first check pass this one, against 13% of the sites that do not.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

By category

In November 2026, here is how the sites we could read did in each category we score.

Advanced and Good practice are on every report but do not count towards the score, so they are marked not scored.

AEO / content

Of the AEO / content results, 47% are passes, 40% are worth a look and 11% need fixing.

  • Pass 47%
  • Worth a look 40%
  • To fix 11%
  • Couldn't judge 1%

AI access

Of the AI access results, 82% are passes, 14% are worth a look and 2% need fixing.

  • Pass 82%
  • Worth a look 14%
  • To fix 2%
  • Couldn't judge 1%

AI signals

Of the AI signals results, 19% are passes, 77% are worth a look and 0% need fixing.

  • Pass 19%
  • Worth a look 77%
  • To fix 0%
  • Couldn't judge 5%

Structured data

Of the Structured data results, 46% are passes, 14% are worth a look and 40% need fixing.

  • Pass 46%
  • Worth a look 14%
  • To fix 40%
  • Couldn't judge 0%

SEO

Of the SEO results, 80% are passes, 7% are worth a look and 10% need fixing.

  • Pass 80%
  • Worth a look 7%
  • To fix 10%
  • Couldn't judge 4%

Links

Of the Links results, 89% are passes, 10% are worth a look and 1% need fixing.

  • Pass 89%
  • Worth a look 10%
  • To fix 1%
  • Couldn't judge 1%

Security

Of the Security results, 97% are passes, 1% are worth a look and 2% need fixing.

  • Pass 97%
  • Worth a look 1%
  • To fix 2%
  • Couldn't judge <1%

Advanced (not scored)

Of the Advanced results, 2% are passes, 94% are worth a look and 0% need fixing.

  • Pass 2%
  • Worth a look 94%
  • To fix 0%
  • Couldn't judge 5%

Good practice (not scored)

Of the Good practice results, 60% are passes, 37% are worth a look and 2% need fixing.

  • Pass 60%
  • Worth a look 37%
  • To fix 2%
  • Couldn't judge <1%
Show as a table
Share of results by category
CategoryPassWorth a lookTo fixCouldn't judge
AEO / content47%40%11%1%
AI access82%14%2%1%
AI signals19%77%0%5%
Structured data46%14%40%0%
SEO80%7%10%4%
Links89%10%1%1%
Security97%1%2%<1%
Advanced (not scored)2%94%0%5%
Good practice (not scored)60%37%2%<1%

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Structured data

In November 2026, the most common type of structured data was WebSite, on 30% of the sites we could read.

  • WebSite 30%
  • Organization 20%
  • WebPage 8%
  • BreadcrumbList 5%
  • FAQPage 3%
  • LocalBusiness 3%
  • SoftwareApplication 2%
  • ItemList 2%
  • WebApplication 1%
  • NewsMediaOrganization 1%
  • Person 1%
  • Article 1%
  • ImageObject 1%
  • CollectionPage 1%
Show as a table
Schema.org types in use
TypeShare of sites
WebSite30%
Organization20%
WebPage8%
BreadcrumbList5%
FAQPage3%
LocalBusiness3%
SoftwareApplication2%
ItemList2%
WebApplication1%
NewsMediaOrganization1%
Person1%
Article1%
ImageObject1%
CollectionPage1%

Each type is counted as the type a page publishes in its own right, not when it sits nested inside other markup. The “Organization schema with sameAs” check also counts subtypes and nested markup, so its figure in Every check is a different count. See Every check.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Emerging standards

In November 2026, “llms.txt is present and valid” was the emerging standard that sites passed most often: 15% of the sites we could read passed it.

Embed or share this figure : llms.txt
In November 2026, 15% of the sites we could read published a valid llms.txt file.
Embed style
Show as a table
Emerging standards
StandardShare of sites
llms.txt is present and valid15%
Clean content for agents (Markdown)9%
Content usage preferences declared5%
Resource-discovery Link headers2%

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Change over time

These lines count the sites scanned in each calendar month, so a month's figure can differ from the headline figures above, which use each site's latest scan from the last three months. The “All sites” column is every site scanned that month, including sites added since, so a change there can be a different crowd being scanned.

“Same sites” is every month for the one group of sites scanned in both the first month with a figure and the latest month, so a change there is the sites themselves changing.

Agent Skills index

Counting every site scanned in November 2026, 1% of the sites we could read passed the “Agent Skills index” check.

Show as a table
Agent Skills index, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%7%—
November 2026100%1%1%

API catalog

Counting every site scanned in November 2026, 1% of the sites we could read passed the “API catalog” check.

Show as a table
API catalog, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%3%—
November 2026100%1%1%

Auth.md agent registration

Counting every site scanned in November 2026, <1% of the sites we could read passed the “Auth.md agent registration” check.

Show as a table
Auth.md agent registration, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%3%—
November 2026100%<1%<1%

AI discovery via DNS

Counting every site scanned in November 2026, <1% of the sites we could read passed the “AI discovery via DNS” check.

Show as a table
AI discovery via DNS, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%0%—
November 2026100%<1%<1%

MCP Server Card

Counting every site scanned in November 2026, 1% of the sites we could read passed the “MCP Server Card” check.

Show as a table
MCP Server Card, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%8%—
November 2026100%1%1%

OAuth authorization-server metadata

Counting every site scanned in November 2026, 4% of the sites we could read passed the “OAuth authorization-server metadata” check.

Show as a table
OAuth authorization-server metadata, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%3%—
November 2026100%4%4%

OAuth protected-resource metadata

Counting every site scanned in November 2026, 4% of the sites we could read passed the “OAuth protected-resource metadata” check.

Show as a table
OAuth protected-resource metadata, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%3%—
November 2026100%4%4%

OpenID Connect discovery

Counting every site scanned in November 2026, 4% of the sites we could read passed the “OpenID Connect discovery” check.

Show as a table
OpenID Connect discovery, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%3%—
November 2026100%4%4%

Web Bot Auth (operates a signed bot)

Counting every site scanned in November 2026, <1% of the sites we could read passed the “Web Bot Auth (operates a signed bot)” check.

Show as a table
Web Bot Auth (operates a signed bot), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%0%—
November 2026100%<1%<1%

AI bot rules are explicit and intentional

Counting every site scanned in November 2026, 18% of the sites we could read passed the “AI bot rules are explicit and intentional” check.

Show as a table
AI bot rules are explicit and intentional, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%25%—
November 2026100%18%18%

AI crawlers aren't blocked

Counting every site scanned in November 2026, 84% of the sites we could read passed the “AI crawlers aren't blocked” check.

Show as a table
AI crawlers aren't blocked, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
August 2026100%76%—
October 2026100%74%—
November 2026100%84%84%

Analytics is installed

Counting every site scanned in November 2026, 60% of the sites we could read passed the “Analytics is installed” check.

Show as a table
Analytics is installed, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%61%—
November 2026100%60%60%

Descriptive anchor text

Counting every site scanned in November 2026, >99% of the sites where it applies (90% of the sites we could read) passed the “Descriptive anchor text” check.

Show as a table
Descriptive anchor text, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202695%100%—
November 202690%>99%>99%

Author / publisher is clear

Counting every site scanned in November 2026, 37% of the sites we could read passed the “Author / publisher is clear” check.

Show as a table
Author / publisher is clear, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%56%—
November 2026100%37%37%

No broken links

Counting every site scanned in November 2026, 89% of the sites where it applies (90% of the sites we could read) passed the “No broken links” check.

Show as a table
No broken links, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202695%88%—
November 202690%89%89%

Caching or validation headers are set

Counting every site scanned in November 2026, 91% of the sites we could read passed the “Caching or validation headers are set” check.

Show as a table
Caching or validation headers are set, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%93%—
November 2026100%91%91%

This page declares its canonical URL

Counting every site scanned in November 2026, 63% of the sites we could read passed the “This page declares its canonical URL” check.

Show as a table
This page declares its canonical URL, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%72%—
November 2026100%63%63%

The site settles on one host (www vs non-www)

Counting every site scanned in November 2026, 87% of the sites we could read passed the “The site settles on one host (www vs non-www)” check.

Show as a table
The site settles on one host (www vs non-www), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%77%—
November 2026100%87%87%

Content is scannable in self-contained chunks

Counting every site scanned in November 2026, 78% of the sites we could read passed the “Content is scannable in self-contained chunks” check.

Show as a table
Content is scannable in self-contained chunks, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%90%—
November 2026100%78%78%

Your answer is in the HTML

Counting every site scanned in November 2026, 86% of the sites we could read passed the “Your answer is in the HTML” check.

Show as a table
Your answer is in the HTML, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%97%—
November 2026100%86%86%

Content usage preferences declared

Counting every site scanned in November 2026, 5% of the sites we could read passed the “Content usage preferences declared” check.

Show as a table
Content usage preferences declared, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%10%—
November 2026100%5%5%

Missing pages return a real 404

Counting every site scanned in November 2026, 84% of the sites we could read passed the “Missing pages return a real 404” check.

Show as a table
Missing pages return a real 404, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%89%—
November 2026100%84%84%

A favicon is set

Counting every site scanned in November 2026, 88% of the sites we could read passed the “A favicon is set” check.

Show as a table
A favicon is set, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%93%—
November 2026100%88%88%

A machine-readable published / updated date

Counting every site scanned in November 2026, 12% of the sites we could read passed the “A machine-readable published / updated date” check.

Show as a table
A machine-readable published / updated date, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%16%—
November 2026100%12%12%

Logical heading order

Counting every site scanned in November 2026, 60% of the sites we could read passed the “Logical heading order” check.

Show as a table
Logical heading order, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%67%—
November 2026100%60%60%

Reached without a redirect chain

Counting every site scanned in November 2026, 98% of the sites we could read passed the “Reached without a redirect chain” check.

Show as a table
Reached without a redirect chain, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%98%—
November 2026100%98%98%

Served over a secure HTTPS connection

Counting every site scanned in November 2026, >99% of the sites we could read passed the “Served over a secure HTTPS connection” check.

Show as a table
Served over a secure HTTPS connection, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%98%—
November 2026100%>99%>99%

HTTP redirects to HTTPS

Counting every site scanned in November 2026, 88% of the sites we could read passed the “HTTP redirects to HTTPS” check.

Show as a table
HTTP redirects to HTTPS, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%82%—
November 2026100%88%88%

Images have descriptive alt text

Counting every site scanned in November 2026, 81% of the sites where it applies (83% of the sites we could read) passed the “Images have descriptive alt text” check.

Show as a table
Images have descriptive alt text, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202687%89%—
November 202683%81%81%

Images are optimised

Counting every site scanned in November 2026, 21% of the sites where it applies (83% of the sites we could read) passed the “Images are optimised” check.

Show as a table
Images are optimised, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202687%28%—
November 202683%21%21%

Structured data is present

Counting every site scanned in November 2026, 44% of the sites we could read passed the “Structured data is present” check.

Show as a table
Structured data is present, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
August 2026100%54%—
October 2026100%64%—
November 2026100%44%44%

Structured data is valid

Counting every site scanned in November 2026, 97% of the sites where it applies (41% of the sites we could read) passed the “Structured data is valid” check.

Show as a table
Structured data is valid, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202657%100%—
November 202641%97%97%

Topic is consistent across title, H1 and description

Counting every site scanned in November 2026, 47% of the sites we could read passed the “Topic is consistent across title, H1 and description” check.

Show as a table
Topic is consistent across title, H1 and description, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%54%—
November 2026100%47%47%

Language and charset declared

Counting every site scanned in November 2026, 94% of the sites we could read passed the “Language and charset declared” check.

Show as a table
Language and charset declared, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%93%—
November 2026100%94%94%

Resource-discovery Link headers

Counting every site scanned in November 2026, 2% of the sites we could read passed the “Resource-discovery Link headers” check.

Show as a table
Resource-discovery Link headers, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%10%—
November 2026100%2%2%

Sensible internal/external link mix

Counting every site scanned in November 2026, 87% of the sites where it applies (90% of the sites we could read) passed the “Sensible internal/external link mix” check.

Show as a table
Sensible internal/external link mix, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202695%95%—
November 202690%87%87%

llms.txt is present and valid

Counting every site scanned in November 2026, 14% of the sites we could read passed the “llms.txt is present and valid” check.

Show as a table
llms.txt is present and valid, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
August 2026100%25%—
October 2026100%26%—
November 2026100%14%14%

Clean content for agents (Markdown)

Counting every site scanned in November 2026, 9% of the sites we could read passed the “Clean content for agents (Markdown)” check.

Show as a table
Clean content for agents (Markdown), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%7%—
November 2026100%9%9%

Has a meta description

Counting every site scanned in November 2026, 84% of the sites we could read passed the “Has a meta description” check.

Show as a table
Has a meta description, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%89%—
November 2026100%84%84%

No meta-refresh redirect

Counting every site scanned in November 2026, 99% of the sites we could read passed the “No meta-refresh redirect” check.

Show as a table
No meta-refresh redirect, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%100%—
November 2026100%99%99%

Not accidentally set to noindex

Counting every site scanned in November 2026, 99% of the sites we could read passed the “Not accidentally set to noindex” check.

Show as a table
Not accidentally set to noindex, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%100%—
November 2026100%99%99%

No mixed (insecure) content

Counting every site scanned in November 2026, 99% of the sites we could read passed the “No mixed (insecure) content” check.

Show as a table
No mixed (insecure) content, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%98%—
November 2026100%99%99%

Open Graph tags for sharing

Counting every site scanned in November 2026, 48% of the sites we could read passed the “Open Graph tags for sharing” check.

Show as a table
Open Graph tags for sharing, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%48%—
November 2026100%48%48%

Organization schema with sameAs

Counting every site scanned in November 2026, 14% of the sites we could read passed the “Organization schema with sameAs” check.

Show as a table
Organization schema with sameAs, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
August 2026100%29%—
October 2026100%31%—
November 2026100%14%14%

Links don't point at redirects

Counting every site scanned in November 2026, 84% of the sites where it applies (82% of the sites we could read) passed the “Links don't point at redirects” check.

Show as a table
Links don't point at redirects, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202692%93%—
November 202682%84%84%

robots.txt is present and not over-blocking

Counting every site scanned in November 2026, 78% of the sites we could read passed the “robots.txt is present and not over-blocking” check.

Show as a table
robots.txt is present and not over-blocking, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%85%—
November 2026100%78%78%

Not flagged for malware or phishing

Counting every site scanned in November 2026, 100% of the sites we could read passed the “Not flagged for malware or phishing” check.

Show as a table
Not flagged for malware or phishing, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%100%—
November 2026>99%100%100%

Search Console / Bing verification

Counting every site scanned in November 2026, 66% of the sites we could read passed the “Search Console / Bing verification” check.

Show as a table
Search Console / Bing verification, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%80%—
November 2026100%66%66%

Core security headers are set

Counting every site scanned in November 2026, 14% of the sites we could read passed the “Core security headers are set” check.

Show as a table
Core security headers are set, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%23%—
November 2026100%14%14%

security.txt is published

Counting every site scanned in November 2026, 4% of the sites we could read passed the “security.txt is published” check.

Show as a table
security.txt is published, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%20%—
November 2026100%4%4%

Semantic HTML landmarks

Counting every site scanned in November 2026, 52% of the sites we could read passed the “Semantic HTML landmarks” check.

Show as a table
Semantic HTML landmarks, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%67%—
November 2026100%52%52%

No server version leakage

Counting every site scanned in November 2026, 90% of the sites we could read passed the “No server version leakage” check.

Show as a table
No server version leakage, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%89%—
November 2026100%90%90%

The page has an H1

Counting every site scanned in November 2026, 63% of the sites we could read passed the “The page has an H1” check.

Show as a table
The page has an H1, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%79%—
November 2026100%63%63%

XML sitemap is present and referenced

Counting every site scanned in November 2026, 59% of the sites we could read passed the “XML sitemap is present and referenced” check.

Show as a table
XML sitemap is present and referenced, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
August 2026100%71%—
October 2026100%69%—
November 2026100%59%59%

Has a title tag

Counting every site scanned in November 2026, 99% of the sites we could read passed the “Has a title tag” check.

Show as a table
Has a title tag, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%98%—
November 2026100%99%99%

Title fits a search snippet

Counting every site scanned in November 2026, 40% of the sites where it applies (99% of the sites we could read) passed the “Title fits a search snippet” check.

Show as a table
Title fits a search snippet, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%52%—
November 202699%40%40%

Twitter/X card tags

Counting every site scanned in November 2026, 60% of the sites we could read passed the “Twitter/X card tags” check.

Show as a table
Twitter/X card tags, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%74%—
November 2026100%60%60%

Right schema type for the page

Counting every site scanned in November 2026, 66% of the sites where it applies (41% of the sites we could read) passed the “Right schema type for the page” check.

Show as a table
Right schema type for the page, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202657%80%—
November 202641%66%66%

Mobile viewport is set

Counting every site scanned in November 2026, 86% of the sites we could read passed the “Mobile viewport is set” check.

Show as a table
Mobile viewport is set, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%90%—
November 2026100%86%86%

AI catalog

Counting every site scanned in November 2026, 1% of the sites we could read passed the “AI catalog” check.

Show as a table
AI catalog, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%5%—
November 2026100%1%1%

Your content survives without JavaScript

Counting every site scanned in November 2026, 89% of the sites we could read passed the “Your content survives without JavaScript” check.

Show as a table
Your content survives without JavaScript, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%93%—
November 2026100%89%89%

A2A Agent Card

Counting every site scanned in November 2026, <1% of the sites we could read passed the “A2A Agent Card” check.

Show as a table
A2A Agent Card, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%5%—
November 2026100%<1%<1%

AI usage preferences (Content-Usage)

Counting every site scanned in November 2026, 0% of the sites we could read passed the “AI usage preferences (Content-Usage)” check.

Show as a table
AI usage preferences (Content-Usage), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%2%—
November 2026100%0%0%

Markdown version linked from the page

Counting every site scanned in November 2026, 1% of the sites we could read passed the “Markdown version linked from the page” check.

Show as a table
Markdown version linked from the page, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%8%—
November 2026100%1%1%

Licensing terms for AI (RSL)

Counting every site scanned in November 2026, <1% of the sites we could read passed the “Licensing terms for AI (RSL)” check.

Show as a table
Licensing terms for AI (RSL), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%2%—
November 2026100%<1%<1%

Text-and-data-mining reservation (TDMRep)

Counting every site scanned in November 2026, <1% of the sites we could read passed the “Text-and-data-mining reservation (TDMRep)” check.

Show as a table
Text-and-data-mining reservation (TDMRep), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%2%—
November 2026100%<1%<1%

WebMCP tools in the page

Counting every site scanned in November 2026, 2% of the sites we could read passed the “WebMCP tools in the page” check.

Show as a table
WebMCP tools in the page, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%5%—
November 2026100%2%2%

Buttons, links and fields have names

Counting every site scanned in November 2026, 51% of the sites where it applies (92% of the sites we could read) passed the “Buttons, links and fields have names” check.

Show as a table
Buttons, links and fields have names, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202695%48%—
November 202692%51%51%

Your page's HTML isn't oversized

Counting every site scanned in November 2026, 98% of the sites we could read passed the “Your page's HTML isn't oversized” check.

Show as a table
Your page's HTML isn't oversized, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%95%—
November 2026100%98%98%

Description fits a search snippet

Counting every site scanned in November 2026, 54% of the sites where it applies (84% of the sites we could read) passed the “Description fits a search snippet” check.

Show as a table
Description fits a search snippet, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202687%60%—
November 202684%54%54%

Your page arrives quickly

Counting every site scanned in November 2026, 94% of the sites we could read passed the “Your page arrives quickly” check.

Show as a table
Your page arrives quickly, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%90%—
November 2026100%94%94%

Engines are allowed to quote the page

Counting every site scanned in November 2026, >99% of the sites we could read passed the “Engines are allowed to quote the page” check.

Show as a table
Engines are allowed to quote the page, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%92%—
November 2026100%>99%>99%

Links are real links

Counting every site scanned in November 2026, 81% of the sites where it applies (90% of the sites we could read) passed the “Links are real links” check.

Show as a table
Links are real links, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202693%88%—
November 202690%81%81%

Claims come with figures, quotes or sources

Counting every site scanned in November 2026, 51% of the sites where it applies (56% of the sites we could read) passed the “Claims come with figures, quotes or sources” check.

Show as a table
Claims come with figures, quotes or sources, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202674%53%—
November 202656%51%51%

Section headings are phrased as questions

Counting every site scanned in November 2026, 22% of the sites where it applies (50% of the sites we could read) passed the “Section headings are phrased as questions” check.

Show as a table
Section headings are phrased as questions, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202661%30%—
November 202650%22%22%

Webpage markup

Counting every site scanned in November 2026, 0% of the sites we could read published Webpage structured data as a type of its own.

Show as a table
Webpage markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20260%—
November 20260%0%

Brand markup

Counting every site scanned in November 2026, <1% of the sites we could read published Brand structured data as a type of its own.

Show as a table
Brand markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20262%—
November 2026<1%<1%

Organization markup

Counting every site scanned in November 2026, 19% of the sites we could read published Organization structured data as a type of its own.

Show as a table
Organization markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 202626%—
October 202631%—
November 202619%19%

SoftwareApplication markup

Counting every site scanned in November 2026, 2% of the sites we could read published SoftwareApplication structured data as a type of its own.

Show as a table
SoftwareApplication markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20268%—
October 20265%—
November 20262%2%

BreadcrumbList markup

Counting every site scanned in November 2026, 5% of the sites we could read published BreadcrumbList structured data as a type of its own.

Show as a table
BreadcrumbList markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20266%—
October 20265%—
November 20265%5%

WebSite markup

Counting every site scanned in November 2026, 30% of the sites we could read published WebSite structured data as a type of its own.

Show as a table
WebSite markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 202624%—
October 202626%—
November 202630%30%

WebPage markup

Counting every site scanned in November 2026, 7% of the sites we could read published WebPage structured data as a type of its own.

Show as a table
WebPage markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 202614%—
October 202626%—
November 20267%7%

ItemList markup

Counting every site scanned in November 2026, 2% of the sites we could read published ItemList structured data as a type of its own.

Show as a table
ItemList markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20263%—
October 20260%—
November 20262%2%

Article markup

Counting every site scanned in November 2026, 1% of the sites we could read published Article structured data as a type of its own.

Show as a table
Article markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20263%—
October 20267%—
November 20261%1%

NewsMediaOrganization markup

Counting every site scanned in November 2026, 1% of the sites we could read published NewsMediaOrganization structured data as a type of its own.

Show as a table
NewsMediaOrganization markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20262%—
November 20261%1%

NewsArticle markup

Counting every site scanned in November 2026, 0% of the sites we could read published NewsArticle structured data as a type of its own.

Show as a table
NewsArticle markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20265%—
October 20260%—
November 20260%0%

ImageObject markup

Counting every site scanned in November 2026, 1% of the sites we could read published ImageObject structured data as a type of its own.

Show as a table
ImageObject markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20263%—
November 20261%1%

FAQPage markup

Counting every site scanned in November 2026, 3% of the sites we could read published FAQPage structured data as a type of its own.

Show as a table
FAQPage markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20265%—
October 20265%—
November 20263%3%

CollectionPage markup

Counting every site scanned in November 2026, 1% of the sites we could read published CollectionPage structured data as a type of its own.

Show as a table
CollectionPage markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20265%—
October 20262%—
November 20261%1%

Corporation markup

Counting every site scanned in November 2026, 1% of the sites we could read published Corporation structured data as a type of its own.

Show as a table
Corporation markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20263%—
October 20262%—
November 20261%1%

WebApplication markup

Counting every site scanned in November 2026, 1% of the sites we could read published WebApplication structured data as a type of its own.

Show as a table
WebApplication markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20260%—
November 20261%1%

BlogPosting markup

Counting every site scanned in November 2026, <1% of the sites we could read published BlogPosting structured data as a type of its own.

Show as a table
BlogPosting markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20260%—
November 2026<1%<1%

SocialMediaPosting markup

Counting every site scanned in November 2026, 0% of the sites we could read published SocialMediaPosting structured data as a type of its own.

Show as a table
SocialMediaPosting markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20260%—
November 20260%0%

Review markup

Counting every site scanned in November 2026, 0% of the sites we could read published Review structured data as a type of its own.

Show as a table
Review markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20260%—
November 20260%0%

Service markup

Counting every site scanned in November 2026, <1% of the sites we could read published Service structured data as a type of its own.

Show as a table
Service markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20263%—
November 2026<1%<1%

HowTo markup

Counting every site scanned in November 2026, <1% of the sites we could read published HowTo structured data as a type of its own.

Show as a table
HowTo markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20262%—
November 2026<1%<1%

TechArticle markup

Counting every site scanned in November 2026, 0% of the sites we could read published TechArticle structured data as a type of its own.

Show as a table
TechArticle markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20262%—
November 20260%0%

Person markup

Counting every site scanned in November 2026, 1% of the sites we could read published Person structured data as a type of its own.

Show as a table
Person markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20265%—
November 20261%1%

ProfessionalService markup

Counting every site scanned in November 2026, <1% of the sites we could read published ProfessionalService structured data as a type of its own.

Show as a table
ProfessionalService markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20262%—
November 2026<1%<1%

LocalBusiness markup

Counting every site scanned in November 2026, 3% of the sites we could read published LocalBusiness structured data as a type of its own.

Show as a table
LocalBusiness markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20263%—
November 20263%3%

ProfilePage markup

Counting every site scanned in November 2026, 0% of the sites we could read published ProfilePage structured data as a type of its own.

Show as a table
ProfilePage markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20262%—
November 20260%0%

Sites scoring 81–90

Counting every site scanned in November 2026, 37% of the sites we scanned scored 81–90.

Show as a table
Sites scoring 81–90, share of sites by month
MonthAll sites (new ones included)Same sites
October 202648%—
November 202637%37%

Turned our scanner away

Counting every site scanned in November 2026, 9% of the sites we scanned turned our scanner away.

Show as a table
Turned our scanner away, share of sites by month
MonthAll sites (new ones included)Same sites
October 20265%—
November 20269%9%

Lines that start in November 2026

These lines have one month of figures so far. Each becomes a month-by-month line once a second month of figures arrives. Score ranges are counted from November 2026, when we began recording them, so they have no earlier months. A range with too few sites behind it is left out, so the lines can add up to less than all the sites.

Show as a table
Lines that start in November 2026
SeriesAll sites
Sites scoring 91–10022%
Sites scoring 71–8020%
Sites scoring 61–708%
Sites scoring 51–604%

Checks that have only one month of figures are not listed here again: their latest figure is in Every check.

When we changed how we judge

  1. Broken links, Safe Browsing and a first group of agent readiness checks
  2. Four new checks, and a caching check
  3. A site that turns our scanner away now gets a report, not an error screen
  4. The verdict now follows how well AI can read you, not just the total
  5. Eight checks were tightened after we tested them on real sites
  6. The AI crawler list was refreshed, and page weight is measured properly
  7. "We could not look" is no longer reported as "there is nothing there"
  8. A site that blocks AI crawlers can no longer look healthy
  9. A site that blocks our scanner gets no score at all
  10. Site scans no longer count the same page more than once
  11. A missing llms.txt is reported as a fail again
  12. Rebalanced one check that was counting for more than it proved
  13. A category we could not measure no longer shows as a zero
  14. Every scan now records which version of our scoring produced it
  15. We look in more than one place before telling you something is missing
  16. Your site score now reflects how much of your site is actually right
  17. Your score now counts only what decides whether a machine can read your page
  18. "AI access" now means access, and what you tell AI has its own place
  19. Site scans now use all the pages they are allowed to read
  20. Pages that build themselves in the browser are now read the way a browser reads them
  21. A "Robot or human?" screen is no longer scored as your site
  22. A paywall or a rate limit is no longer called a firewall
  23. A page withheld for legal reasons is no longer called bot protection
  24. A directory of AI crawlers, with who runs each one and what it is for
  25. A site scan now names the pages your robots.txt disallows for us
  26. New checks, and fewer signals only a person can see
  27. Short brand names are compared, and checks we could not run say why
  28. Five checks stopped misreading a few kinds of page
  29. One contact-file check moves to shown, not scored; two others are re-balanced
  30. Analytics look-alikes no longer count, and page titles read the way a browser reads them
  31. When we can't reach a page, we record why
  32. Site scans no longer test a "#" or "back to top" link as a link
  33. A Content-Signal mentioned in a robots.txt comment no longer counts
  34. Placeholder links no longer count as internal links
  35. Pages with search boxes and lazy-loaded images are read in full
  36. Link checks no longer test a link to the page itself
  37. The sitemap your robots.txt points to is now checked
  38. Structured data about a page's main subject is now read
  39. A site whose secure version answers slowly is no longer reported as not using https
  40. A slow secure version no longer marks a site down for its canonical or HSTS
  41. Analytics is found wherever it sits on the page, and more tools are recognised
  42. The www and non-www check no longer skips when the other name does not exist
  43. A link inside a button now names the button
  44. A button inside a link now names the link
  45. Structured data in menus, headers and footers is read where it belongs
  46. Titles, headings and redirects are read the way a browser reads them
  47. Organization markup is recognised for every kind of organization
  48. Fewer false passes on sites that answer missing files with a styled error page
  49. Service catalogs in your structured data are now read
  50. A site that asks us to slow down now gets a patient second look before a check is skipped
  51. The OAuth protected-resource check now reads the file instead of only finding it
  52. Your site scan now compares your pages with each other
  53. Very large pages now finish scanning instead of timing out
  54. Hidden template content no longer counts toward your headings
  55. A site scan no longer spends its pages on the same page twice

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

July 2026, August 2026, September 2026, October 2026, November 2026

Better or worse

Not enough history yet.

Fixed
Fixed means a site failed a check, or only got a “worth a look”, the first time we scanned it and passes it now.
Broke
Broke means the reverse.
Includes our changes
Includes our changes means we changed how a check is judged between the two scans. We count those sites apart and never as a site improving.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Performance

Not enough performance data yet.

Core Web Vitals are Google's three measures of loading, responsiveness and visual stability. Field results come from real Chrome visitors over the past four weeks. Lab results come from one test run.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Readiness ladder

In November 2026, 9% of the sites we scanned cleared every one of the gates.

The gates are “Automated visitors can reach your pages”, “AI crawlers aren't blocked”, “Reached without a redirect chain”, “Your answer is in the HTML”, “Structured data is present”, “llms.txt is present and valid”: the basics a machine needs before it can use a page.

  1. 6% No gates cleared
  2. 3% One of six gates cleared
  3. 4% Two of six gates cleared
  4. 12% Three of six gates cleared
  5. 40% Four of six gates cleared
  6. 26% Five of six gates cleared
  7. 9% All six gates cleared
Show as a table
How many readiness gates sites clear
Gates clearedShare of sites
No gates cleared6%
One of six gates cleared3%
Two of six gates cleared4%
Three of six gates cleared12%
Four of six gates cleared40%
Five of six gates cleared26%
All six gates cleared9%

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Embed or share this figure : readiness gates
In November 2026, 9% of the sites we scanned cleared every one of the gates.
Embed style

Method

  • A site is a host exactly as we scanned it: the www version and the bare domain are different sites.
  • Each site counts once, on its latest scan of the homepage (or its latest scan of any page if it has no homepage scan). A scan that failed outright is never counted.
  • "Turned our scanner away" means the site answered our scanner with a refusal, a bot check or an empty page we could not read. A site that was down or timed out is left out, not counted as turning us away.
  • Figures about a single check count only the sites we could read: a site that turned our scanner away has no check results, so it is left out of them. Within those, a check that found nothing to judge on a page is counted only on the sites it applies to, and the "Applies to" column says how many that is.
  • The headline figures use scans from the last three months. The month-by-month lines use calendar months in UTC, so a month's figure can differ from the headline one.
  • Figures are whole percentages. A share above zero is never rounded down to nothing (it reads "<1%"), and one short of all is never rounded up to all (it reads ">99%"). The JSON file keeps numbers: 0.1 means under 1% and 99.9 means over 99%.
  • A figure appears only when enough different sites sit behind it. Below that it reads "not enough sites yet", and we never merge views to fill a gap.
  • We show statuses and sites counted in ten-point score ranges, never individual scores. A site we could not judge on a check is counted as "couldn't judge", not as a failure.
  • When we change how a check is judged, it is marked on the change-over-time lines and listed in the changelog.
  • Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Figures generated 2026-11-15 from build 9b4aa1e7cd74f60e. Download the data as JSON or CSV; both hold percentages only.

These figures never name a site. To see which homepages score best, by category, open the Leaderboard.

How to cite this

Milk Moon Studio, "Bot Appétit State of the Web & AI Readiness", botappetit.ai/state-of-the-web/, figures from November 2026.

Figures change every hour, so say which month you took them from.

How does your site compare?

A free scan checks all of this on your page in seconds, with no signup.

Scan your site free

Want it handled? We do that too. No pitch, just a plain-English chat.

Book a call