Research

Bot Appétit State of the Web & AI Readiness

How many of the sites we scan let our scanner in, and what they get right and wrong, check by check. Percentages only, refreshed every hour.

Based on the sites visitors have scanned with Bot Appétit. Not a random sample of the web.

This group is small, so one site moves a figure by several percentage points. Read each percentage as a handful of sites, not a trend.

Headlines

Headlines cover all sites, not just the sites visitors scanned. Switch to “All sites” above to see them.

Can AI read the web?

In November 2026, our scanner could read 87% of the sites visitors scanned. The other 13% turned it away.

Sites our scanner could read 87% Scanned by visitors
Clear all six readiness gates 16% Scanned by visitors
Publish structured data (JSON-LD) 53% Scanned by visitors

This is our scanner, not every AI. A site that turns it away may still let other visitors in; how sites treat AI crawlers is in AI access, below.

How sites land

Every page we read gets a score out of 100, the number at the top of its report (how scoring works). Each row is a range of scores. Where too few sites score in the lowest ranges, they are joined into one row. A site our scanner could not read has no score.

  • 81–90 35%
  • 0–80 30%
  • Turned our scanner away 13%
Show as a table
Share of sites in each score range
ScoreShare of sites
81–9035%
0–8030%
Turned our scanner away13%

Based on the sites visitors have scanned with Bot Appétit. Not a random sample of the web.

AI access

In November 2026, 22% of the sites visitors scanned disallowed some AI crawlers in robots.txt, and 13% turned our scanner away at the server.

  • robots.txt disallows some AI crawlers 22%
  • robots.txt disallows major AI crawlers 12%
  • The server turned our scanner away 13%
Show as a table
How sites treat AI crawlers
What we foundShare of sites
robots.txt disallows some AI crawlers22%
robots.txt disallows major AI crawlers12%
The server turned our scanner away13%
robots.txt allows AI crawlers, the server turned our scanner away3%
3% robots.txt allows AI crawlers, the server turned our scanner away

These shares are of every site visitors scanned, including the ones that turned our scanner away. The shares in Every check are of the sites we could read, so the same crawler check can read differently there.

Based on the sites visitors have scanned with Bot Appétit. Not a random sample of the web.

Where sites differ

Where sites differ covers all sites, not just the sites visitors scanned.

In November 2026, here is where the sites we scanned differ most.

By platform

In November 2026, the “AI bot rules are explicit and intentional” check showed the widest gap between platforms: 100% of BigCommerce sites passed it, against 0% of Carrd sites.

How sites on each platform treat AI crawlers in robots.txt

  • Astro 32%
  • BigCommerce 5%
  • Carrd 0%
  • Eleventy 13%
  • Framer 5%
  • Ghost 14%
  • Jekyll 7%
  • Netlify 6%
  • Next.js 19%
  • Shopify 5%
  • SvelteKit 35%
  • Vercel 2%
  • WordPress 29%
Show as a table
How sites on each platform treat AI crawlers in robots.txt
PlatformDisallow some AI crawlers in robots.txt
Astro32%
BigCommerce5%
Carrd0%
Eleventy13%
Framer5%
Ghost14%
Jekyll7%
Netlify6%
Next.js19%
Shopify5%
SvelteKit35%
Vercel2%
WordPress29%

A platform is only known for pages our scanner could read, so this table cannot say how many sites on each platform turned our scanner away.

The checks that differ most between platforms, by the share of sites that pass. Each share is over the sites the check applies to.

AI bot rules are explicit and intentional Astro41% BigCommerce100% Carrd0% Eleventy18% Framer10% Ghost22% Jekyll7% Netlify6% Next.js— Shopify15% SvelteKit38% Vercel7% WordPress33%
Search Console / Bing verification Astro92% BigCommerce100% Carrd0% Eleventy51% Framer80% Ghost83% Jekyll51% Netlify67% Next.js— Shopify98% SvelteKit81% Vercel53% WordPress69%
Structured data is present Astro62% BigCommerce35% Carrd0% Eleventy25% Framer37% Ghost97% Jekyll86% Netlify24% Next.js— Shopify80% SvelteKit55% Vercel41% WordPress79%
Clean content for agents (Markdown) Astro14% BigCommerce97% Carrd0% Eleventy4% Framer85% Ghost0% Jekyll8% Netlify4% Next.js— Shopify5% SvelteKit0% Vercel3% WordPress5%
Sitemap dates tell a machine what changed Astro— BigCommerce— Carrd100% Eleventy64% Framer3% Ghost94% Jekyll— Netlify— Next.js— Shopify3% SvelteKit— Vercel— WordPress—

Which industries are most AI-ready

In November 2026, Shopping & retail sites were the most AI-ready of the industries we could compare: 22% of them cleared every readiness gate, against 6% of Education & research sites.

The share of sites in each industry that clear every readiness gate. Each industry has its own page.

  • Shopping & retail 22%
  • Food & drink 18%
  • Software & SaaS 15%
  • Agencies & professional services 13%
  • Finance & insurance 12%
  • News & media 9%
  • Entertainment & games 8%
  • Government & nonprofit 8%
  • Community & personal 7%
  • Education & research 6%
Show as a table
Share of sites in each industry that clear every readiness gate
IndustryClear every readiness gate
Shopping & retail22%
Food & drink18%
Software & SaaS15%
Agencies & professional services13%
Finance & insurance12%
News & media9%
Entertainment & games8%
Government & nonprofit8%
Community & personal7%
Education & research6%

An industry is only known for pages our scanner could read, so this table cannot say how many sites in each industry turned our scanner away.

Visitors versus research lists

In November 2026, 13% of the sites visitors scanned turned our scanner away, against 9% of the sites on our research lists.

Lists we scan for research are not a random sample of the web either.

  • 91–100 Scanned by visitors: Not enough sites yet Research lists: 21%
  • 81–90 Scanned by visitors: 35% Research lists: 38%
  • 0–80 Scanned by visitors: 30% Research lists: 32%
  • Turned our scanner away Scanned by visitors: 13% Research lists: 9%
Show as a table
Share of sites in each score range, scanned by visitors and on research lists
ScoreScanned by visitorsResearch lists
91–100—21%
81–9035%38%
0–8030%32%
Turned our scanner away13%9%

The checks where the two groups differ most, by the share of sites that pass. Each share is over the sites the check applies to.

Share of sites passing, scanned by visitors and on research lists, for the checks that differ most
CheckScanned by visitorsResearch lists
Has a meta description61%84%
The site settles on one host (www vs non-www)69%87%
Right schema type for the page84%66%
Organization schema with sameAs32%15%
Title fits a search snippet58%41%
Search Console / Bing verification83%67%
Author / publisher is clear51%37%
security.txt is published18%5%
llms.txt is present and valid24%14%
This page declares its canonical URL72%63%

Based on all the sites Bot Appétit has scanned. Not a random sample of the web.

Every check

In November 2026, several checks tie as the ones sites passed most often, each at 100%. “Organization schema with sameAs” was the one they failed most often: 55% of the sites we could read failed it.

A dash means not enough sites yet. Each check links to what it is and what good looks like.

Applies to is the share of the sites we could read that the check ran on, because some checks have nothing to judge on many pages.

The other columns are shares of those sites, not of every site. Newer checks only cover sites scanned since the check was added.

“Automated visitors can reach your pages” shows a dash: a site passes it whenever our scanner could read the page, so it only repeats whether we got in. See how many turned us away.

Fixed and Broke compare re-scanned sites, and those columns are left out until there are enough of them. Not enough history yet.

AEO / content

AEO / content: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Your content survives without JavaScript Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Your answer is in the HTML Applies to 100% Pass 85% Worth a look 7% To fix 8% Couldn't judge 0%
Semantic HTML landmarks Applies to 100% Pass 59% Worth a look 41% To fix 0% Couldn't judge 0%
Content is scannable in self-contained chunks Applies to 100% Pass 77% Worth a look 23% To fix 0% Couldn't judge 0%
Author / publisher is clear Applies to 100% Pass 51% Worth a look 49% To fix 0% Couldn't judge 0%
A machine-readable published / updated date Applies to 100% Pass 20% Worth a look 80% To fix 0% Couldn't judge 0%
llms.txt is present and valid Applies to 100% Pass 24% Worth a look 60% To fix 15% Couldn't judge 1%
Clean content for agents (Markdown) Applies to 100% Pass 16% Worth a look 81% To fix 0% Couldn't judge 3%

AI access

AI access: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Buttons, links and fields have names Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
AI crawlers aren't blocked Applies to 100% Pass 77% Worth a look 8% To fix 13% Couldn't judge 2%
Automated visitors can reach your pages See how many turned us away Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Engines are allowed to quote the page Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Your page arrives quickly Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —

AI signals

AI signals: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
AI bot rules are explicit and intentional Applies to 100% Pass 23% Worth a look 75% To fix 0% Couldn't judge 2%

Structured data

Structured data: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Structured data is present Applies to 100% Pass 53% Worth a look 0% To fix 47% Couldn't judge 0%
Structured data is valid Applies to 51% Pass 100% Worth a look 0% To fix 0% Couldn't judge 0%
FAQ / Q&A schema Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Organization schema with sameAs Applies to 100% Pass 32% Worth a look 14% To fix 55% Couldn't judge 0%
Right schema type for the page Applies to 51% Pass 84% Worth a look 16% To fix 0% Couldn't judge 0%
Schema facts also appear in the page text Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Breadcrumb structured data Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —

SEO

SEO: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Reached without a redirect chain Applies to 100% Pass 98% Worth a look 2% To fix 0% Couldn't judge 0%
Caching or validation headers are set Applies to 100% Pass 94% Worth a look 6% To fix 0% Couldn't judge 0%
The site settles on one host (www vs non-www) Applies to 100% Pass 69% Worth a look 7% To fix 8% Couldn't judge 16%
robots.txt is present and not over-blocking Applies to 100% Pass 83% Worth a look 0% To fix 17% Couldn't judge 0%
Not accidentally set to noindex Applies to 100% Pass 99% Worth a look 0% To fix 1% Couldn't judge 0%
This page declares its canonical URL Applies to 100% Pass 72% Worth a look 3% To fix 25% Couldn't judge 0%
Has a title tag Applies to 100% Pass 97% Worth a look 0% To fix 3% Couldn't judge 0%
Has a meta description Applies to 100% Pass 61% Worth a look 20% To fix 18% Couldn't judge 0%
The page has an H1 Applies to 100% Pass 56% Worth a look 7% To fix 38% Couldn't judge 0%
Logical heading order Applies to 100% Pass 55% Worth a look 26% To fix 15% Couldn't judge 5%
Topic is consistent across title, H1 and description Applies to 100% Pass 49% Worth a look 8% To fix 0% Couldn't judge 43%
Images have descriptive alt text Applies to 76% Pass 90% Worth a look 7% To fix 3% Couldn't judge 0%
Your page's HTML isn't oversized Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Open Graph tags for sharing Applies to 100% Pass 47% Worth a look 31% To fix 23% Couldn't judge 0%
XML sitemap is present and referenced Applies to 100% Pass 68% Worth a look 8% To fix 17% Couldn't judge 7%
Missing pages return a real 404 Applies to 100% Pass 84% Worth a look 0% To fix 15% Couldn't judge 1%
Language and charset declared Applies to 100% Pass 89% Worth a look 11% To fix 0% Couldn't judge 0%
hreflang for language variants Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
No meta-refresh redirect Applies to 100% Pass 100% Worth a look 0% To fix 0% Couldn't judge 0%

Links

Links: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
No broken links Applies to 85% Pass 91% Worth a look 8% To fix 0% Couldn't judge 1%
Links don't point at redirects Applies to 80% Pass 89% Worth a look 10% To fix 0% Couldn't judge 1%
Descriptive anchor text Applies to 85% Pass 100% Worth a look 0% To fix 0% Couldn't judge 0%
Links are real links Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —

Performance (not scored)

Not enough sites yet

Performance (not scored): share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Largest Contentful Paint (LCP) Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Cumulative Layout Shift (CLS) Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Interaction to Next Paint (INP) Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Page weight and render-blocking Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —

Security

Security: share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Served over a secure HTTPS connection Applies to 100% Pass 100% Worth a look 0% To fix 0% Couldn't judge 0%
HTTP redirects to HTTPS Applies to 100% Pass 86% Worth a look 5% To fix 8% Couldn't judge 1%
No mixed (insecure) content Applies to 100% Pass 100% Worth a look 0% To fix 0% Couldn't judge 0%
Not flagged for malware or phishing Applies to 43% Pass 100% Worth a look 0% To fix 0% Couldn't judge 0%

Advanced (not scored)

Advanced (not scored): share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Content usage preferences declared Applies to 100% Pass 7% Worth a look 91% To fix 0% Couldn't judge 2%
Resource-discovery Link headers Applies to 100% Pass 5% Worth a look 95% To fix 0% Couldn't judge 0%
MCP Server Card Applies to 100% Pass 5% Worth a look 90% To fix 0% Couldn't judge 6%
AI catalog Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
A2A Agent Card Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
WebMCP tools in the page Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
AI usage preferences (Content-Usage) Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Licensing terms for AI (RSL) Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Text-and-data-mining reservation (TDMRep) Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Markdown version linked from the page Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Content signals agree everywhere Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Agent Skills index Applies to 100% Pass 5% Worth a look 89% To fix 0% Couldn't judge 7%
API catalog Applies to 100% Pass 3% Worth a look 91% To fix 0% Couldn't judge 6%
OAuth authorization-server metadata Applies to 100% Pass 5% Worth a look 89% To fix 0% Couldn't judge 7%
OAuth protected-resource metadata Applies to 100% Pass 2% Worth a look 91% To fix 0% Couldn't judge 7%
OpenID Connect discovery Applies to 100% Pass 5% Worth a look 89% To fix 0% Couldn't judge 7%
Web Bot Auth (operates a signed bot) Applies to 100% Pass 0% Worth a look 92% To fix 0% Couldn't judge 8%
Universal Commerce Protocol manifest Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Auth.md agent registration Applies to 100% Pass 0% Worth a look 95% To fix 0% Couldn't judge 5%
AI discovery via DNS Applies to 100% Pass 0% Worth a look 98% To fix 0% Couldn't judge 2%

Good practice (not scored)

Good practice (not scored): share of sites by result
CheckApplies toResultPassWorth a lookTo fixCouldn't judge
Core security headers are set Applies to 100% Pass 18% Worth a look 65% To fix 17% Couldn't judge 0%
No server version leakage Applies to 100% Pass 86% Worth a look 14% To fix 0% Couldn't judge 0%
security.txt is published Applies to 100% Pass 18% Worth a look 78% To fix 0% Couldn't judge 3%
Mobile viewport is set Applies to 100% Pass 83% Worth a look 9% To fix 8% Couldn't judge 0%
Title fits a search snippet Applies to 97% Pass 58% Worth a look 42% To fix 0% Couldn't judge 0%
Description fits a search snippet Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Twitter/X card tags Applies to 100% Pass 69% Worth a look 31% To fix 0% Couldn't judge 0%
Images are optimised Applies to 76% Pass 27% Worth a look 73% To fix 0% Couldn't judge 0%
Analytics is installed Applies to 100% Pass 66% Worth a look 34% To fix 0% Couldn't judge 0%
Search Console / Bing verification Applies to 100% Pass 83% Worth a look 17% To fix 0% Couldn't judge 0%
A favicon is set Applies to 100% Pass 95% Worth a look 5% To fix 0% Couldn't judge 0%
Claims come with figures, quotes or sources Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Section headings are phrased as questions Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Reading the page is affordable for a machine Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
No hidden text written for an AI model Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Sitemap dates tell a machine what changed Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
The links in llms.txt resolve Applies to —

Not enough sites yet

Pass — Worth a look — To fix — Couldn't judge —
Sensible internal/external link mix Applies to 85% Pass 91% Worth a look 9% To fix 0% Couldn't judge 0%

Based on the sites visitors have scanned with Bot Appétit. Not a random sample of the web.

What goes together covers all sites, not just the sites visitors scanned.

What goes together

In November 2026, these were the pairs of checks that sites tended to pass together.

This is an association between checks, not a cause: passing one does not make a site pass the other. It shows what tends to sit together on a site that has been looked after.

  • Sites that pass the “Structured data is present” check are 8 times as likely to pass the “Author / publisher is clear” check too. 74% of the sites that pass the first check pass this one, against 9% of the sites that do not.

  • Sites that pass the “Core security headers are set” check are 6.5 times as likely to pass the “security.txt is published” check too. 23% of the sites that pass the first check pass this one, against 3% of the sites that do not.

  • Sites that pass the “Structured data is present” check are 6 times as likely to pass the “A machine-readable published / updated date” check too. 24% of the sites that pass the first check pass this one, against 4% of the sites that do not.

  • Sites that pass the “Has a meta description” check are 5.5 times as likely to pass the “Open Graph tags for sharing” check too. 56% of the sites that pass the first check pass this one, against 11% of the sites that do not.

  • Sites that pass the “Content usage preferences declared” check are 5 times as likely to pass the “AI bot rules are explicit and intentional” check too. 84% of the sites that pass the first check pass this one, against 16% of the sites that do not.

  • Sites that pass the “llms.txt is present and valid” check are 5 times as likely to pass the “Organization schema with sameAs” check too. 49% of the sites that pass the first check pass this one, against 10% of the sites that do not.

  • Sites that pass the “Author / publisher is clear” check are 4.5 times as likely to pass the “A machine-readable published / updated date” check too. 24% of the sites that pass the first check pass this one, against 6% of the sites that do not.

  • Sites that pass the “This page declares its canonical URL” check are 4.5 times as likely to pass the “llms.txt is present and valid” check too. 21% of the sites that pass the first check pass this one, against 5% of the sites that do not.

  • Sites that pass the “Content is scannable in self-contained chunks” check are 4.5 times as likely to pass the “Semantic HTML landmarks” check too. 64% of the sites that pass the first check pass this one, against 15% of the sites that do not.

  • Sites that pass the “Your answer is in the HTML” check are 4.5 times as likely to pass the “Semantic HTML landmarks” check too. 59% of the sites that pass the first check pass this one, against 13% of the sites that do not.

Based on all the sites Bot Appétit has scanned. Not a random sample of the web.

By category

In November 2026, here is how the sites we could read did in each category we score.

Advanced and Good practice are on every report but do not count towards the score, so they are marked not scored.

AEO / content

Of the AEO / content results, 49% are passes, 47% are worth a look and 3% need fixing.

  • Pass 49%
  • Worth a look 47%
  • To fix 3%
  • Couldn't judge 1%

AI access

Of the AI access results, 77% are passes, 13% are worth a look and 9% need fixing.

  • Pass 77%
  • Worth a look 13%
  • To fix 9%
  • Couldn't judge 1%

AI signals

Of the AI signals results, 23% are passes, 75% are worth a look and 0% need fixing.

  • Pass 23%
  • Worth a look 75%
  • To fix 0%
  • Couldn't judge 2%

Structured data

Of the Structured data results, 56% are passes, 16% are worth a look and 29% need fixing.

  • Pass 56%
  • Worth a look 16%
  • To fix 29%
  • Couldn't judge 0%

SEO

Of the SEO results, 77% are passes, 8% are worth a look and 11% need fixing.

  • Pass 77%
  • Worth a look 8%
  • To fix 11%
  • Couldn't judge 4%

Links

Of the Links results, 92% are passes, 7% are worth a look and 0% need fixing.

  • Pass 92%
  • Worth a look 7%
  • To fix 0%
  • Couldn't judge 1%

Security

Of the Security results, 96% are passes, 1% are worth a look and 2% need fixing.

  • Pass 96%
  • Worth a look 1%
  • To fix 2%
  • Couldn't judge <1%

Advanced (not scored)

Of the Advanced results, 4% are passes, 92% are worth a look and 0% need fixing.

  • Pass 4%
  • Worth a look 92%
  • To fix 0%
  • Couldn't judge 5%

Good practice (not scored)

Of the Good practice results, 63% are passes, 34% are worth a look and 2% need fixing.

  • Pass 63%
  • Worth a look 34%
  • To fix 2%
  • Couldn't judge <1%
Show as a table
Share of results by category
CategoryPassWorth a lookTo fixCouldn't judge
AEO / content49%47%3%1%
AI access77%13%9%1%
AI signals23%75%0%2%
Structured data56%16%29%0%
SEO77%8%11%4%
Links92%7%0%1%
Security96%1%2%<1%
Advanced (not scored)4%92%0%5%
Good practice (not scored)63%34%2%<1%

Based on the sites visitors have scanned with Bot Appétit. Not a random sample of the web.

Structured data

In November 2026, the most common type of structured data was Organization, on 27% of the sites we could read.

  • Organization 27%
  • WebSite 24%
  • WebPage 14%
  • SoftwareApplication 8%
  • BreadcrumbList 6%
  • Article 5%
  • FAQPage 5%
  • NewsArticle 3%
  • CollectionPage 3%
  • NewsMediaOrganization 2%
  • ImageObject 2%
  • Corporation 2%
  • Webpage 1%
  • Brand 1%
  • TechArticle 1%
  • ItemList 1%
  • LocalBusiness 1%
  • Service 1%
  • WebApplication 1%
  • BlogPosting 1%
  • Person 1%
  • SocialMediaPosting 1%
  • Review 1%
Show as a table
Schema.org types in use
TypeShare of sites
Organization27%
WebSite24%
WebPage14%
SoftwareApplication8%
BreadcrumbList6%
Article5%
FAQPage5%
NewsArticle3%
CollectionPage3%
NewsMediaOrganization2%
ImageObject2%
Corporation2%
Webpage1%
Brand1%
TechArticle1%
ItemList1%
LocalBusiness1%
Service1%
WebApplication1%
BlogPosting1%
Person1%
SocialMediaPosting1%
Review1%

Each type is counted as the type a page publishes in its own right, not when it sits nested inside other markup. The “Organization schema with sameAs” check also counts subtypes and nested markup, so its figure in Every check is a different count. See Every check.

Based on the sites visitors have scanned with Bot Appétit. Not a random sample of the web.

Emerging standards

In November 2026, “llms.txt is present and valid” was the emerging standard that sites passed most often: 24% of the sites we could read passed it.

Show as a table
Emerging standards
StandardShare of sites
llms.txt is present and valid24%
Clean content for agents (Markdown)16%
Content usage preferences declared7%
Resource-discovery Link headers5%

Based on the sites visitors have scanned with Bot Appétit. Not a random sample of the web.

Change over time

Change over time covers all sites, not just the sites visitors scanned.

These lines count the sites scanned in each calendar month, so a month's figure can differ from the headline figures above, which use each site's latest scan from the last three months. The “All sites” column is every site scanned that month, including sites added since, so a change there can be a different crowd being scanned.

“Same sites” is every month for the one group of sites scanned in both the first month with a figure and the latest month, so a change there is the sites themselves changing.

Agent Skills index

Counting every site scanned in November 2026, 1% of the sites we could read passed the “Agent Skills index” check.

Show as a table
Agent Skills index, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%7%—
November 2026100%1%1%

API catalog

Counting every site scanned in November 2026, 1% of the sites we could read passed the “API catalog” check.

Show as a table
API catalog, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%3%—
November 2026100%1%1%

Auth.md agent registration

Counting every site scanned in November 2026, <1% of the sites we could read passed the “Auth.md agent registration” check.

Show as a table
Auth.md agent registration, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%3%—
November 2026100%<1%<1%

AI discovery via DNS

Counting every site scanned in November 2026, <1% of the sites we could read passed the “AI discovery via DNS” check.

Show as a table
AI discovery via DNS, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%0%—
November 2026100%<1%<1%

MCP Server Card

Counting every site scanned in November 2026, 1% of the sites we could read passed the “MCP Server Card” check.

Show as a table
MCP Server Card, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%8%—
November 2026100%1%1%

OAuth authorization-server metadata

Counting every site scanned in November 2026, 4% of the sites we could read passed the “OAuth authorization-server metadata” check.

Show as a table
OAuth authorization-server metadata, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%3%—
November 2026100%4%4%

OAuth protected-resource metadata

Counting every site scanned in November 2026, 4% of the sites we could read passed the “OAuth protected-resource metadata” check.

Show as a table
OAuth protected-resource metadata, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%3%—
November 2026100%4%4%

OpenID Connect discovery

Counting every site scanned in November 2026, 4% of the sites we could read passed the “OpenID Connect discovery” check.

Show as a table
OpenID Connect discovery, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%3%—
November 2026100%4%4%

Web Bot Auth (operates a signed bot)

Counting every site scanned in November 2026, <1% of the sites we could read passed the “Web Bot Auth (operates a signed bot)” check.

Show as a table
Web Bot Auth (operates a signed bot), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%0%—
November 2026100%<1%<1%

AI bot rules are explicit and intentional

Counting every site scanned in November 2026, 18% of the sites we could read passed the “AI bot rules are explicit and intentional” check.

Show as a table
AI bot rules are explicit and intentional, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%25%—
November 2026100%18%18%

AI crawlers aren't blocked

Counting every site scanned in November 2026, 84% of the sites we could read passed the “AI crawlers aren't blocked” check.

Show as a table
AI crawlers aren't blocked, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
August 2026100%76%—
October 2026100%74%—
November 2026100%84%84%

Analytics is installed

Counting every site scanned in November 2026, 60% of the sites we could read passed the “Analytics is installed” check.

Show as a table
Analytics is installed, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%61%—
November 2026100%60%60%

Descriptive anchor text

Counting every site scanned in November 2026, >99% of the sites where it applies (90% of the sites we could read) passed the “Descriptive anchor text” check.

Show as a table
Descriptive anchor text, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202695%100%—
November 202690%>99%>99%

Author / publisher is clear

Counting every site scanned in November 2026, 37% of the sites we could read passed the “Author / publisher is clear” check.

Show as a table
Author / publisher is clear, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%56%—
November 2026100%37%37%

No broken links

Counting every site scanned in November 2026, 89% of the sites where it applies (90% of the sites we could read) passed the “No broken links” check.

Show as a table
No broken links, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202695%88%—
November 202690%89%89%

Caching or validation headers are set

Counting every site scanned in November 2026, 91% of the sites we could read passed the “Caching or validation headers are set” check.

Show as a table
Caching or validation headers are set, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%93%—
November 2026100%91%91%

This page declares its canonical URL

Counting every site scanned in November 2026, 63% of the sites we could read passed the “This page declares its canonical URL” check.

Show as a table
This page declares its canonical URL, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%72%—
November 2026100%63%63%

The site settles on one host (www vs non-www)

Counting every site scanned in November 2026, 87% of the sites we could read passed the “The site settles on one host (www vs non-www)” check.

Show as a table
The site settles on one host (www vs non-www), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%77%—
November 2026100%87%87%

Content is scannable in self-contained chunks

Counting every site scanned in November 2026, 78% of the sites we could read passed the “Content is scannable in self-contained chunks” check.

Show as a table
Content is scannable in self-contained chunks, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%90%—
November 2026100%78%78%

Your answer is in the HTML

Counting every site scanned in November 2026, 86% of the sites we could read passed the “Your answer is in the HTML” check.

Show as a table
Your answer is in the HTML, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%97%—
November 2026100%86%86%

Content usage preferences declared

Counting every site scanned in November 2026, 5% of the sites we could read passed the “Content usage preferences declared” check.

Show as a table
Content usage preferences declared, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%10%—
November 2026100%5%5%

Missing pages return a real 404

Counting every site scanned in November 2026, 84% of the sites we could read passed the “Missing pages return a real 404” check.

Show as a table
Missing pages return a real 404, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%89%—
November 2026100%84%84%

A favicon is set

Counting every site scanned in November 2026, 88% of the sites we could read passed the “A favicon is set” check.

Show as a table
A favicon is set, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%93%—
November 2026100%88%88%

A machine-readable published / updated date

Counting every site scanned in November 2026, 12% of the sites we could read passed the “A machine-readable published / updated date” check.

Show as a table
A machine-readable published / updated date, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%16%—
November 2026100%12%12%

Logical heading order

Counting every site scanned in November 2026, 60% of the sites we could read passed the “Logical heading order” check.

Show as a table
Logical heading order, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%67%—
November 2026100%60%60%

Reached without a redirect chain

Counting every site scanned in November 2026, 98% of the sites we could read passed the “Reached without a redirect chain” check.

Show as a table
Reached without a redirect chain, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%98%—
November 2026100%98%98%

Served over a secure HTTPS connection

Counting every site scanned in November 2026, >99% of the sites we could read passed the “Served over a secure HTTPS connection” check.

Show as a table
Served over a secure HTTPS connection, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%98%—
November 2026100%>99%>99%

HTTP redirects to HTTPS

Counting every site scanned in November 2026, 88% of the sites we could read passed the “HTTP redirects to HTTPS” check.

Show as a table
HTTP redirects to HTTPS, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%82%—
November 2026100%88%88%

Images have descriptive alt text

Counting every site scanned in November 2026, 81% of the sites where it applies (83% of the sites we could read) passed the “Images have descriptive alt text” check.

Show as a table
Images have descriptive alt text, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202687%89%—
November 202683%81%81%

Images are optimised

Counting every site scanned in November 2026, 21% of the sites where it applies (83% of the sites we could read) passed the “Images are optimised” check.

Show as a table
Images are optimised, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202687%28%—
November 202683%21%21%

Structured data is present

Counting every site scanned in November 2026, 44% of the sites we could read passed the “Structured data is present” check.

Show as a table
Structured data is present, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
August 2026100%54%—
October 2026100%64%—
November 2026100%44%44%

Structured data is valid

Counting every site scanned in November 2026, 97% of the sites where it applies (41% of the sites we could read) passed the “Structured data is valid” check.

Show as a table
Structured data is valid, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202657%100%—
November 202641%97%97%

Topic is consistent across title, H1 and description

Counting every site scanned in November 2026, 47% of the sites we could read passed the “Topic is consistent across title, H1 and description” check.

Show as a table
Topic is consistent across title, H1 and description, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%54%—
November 2026100%47%47%

Language and charset declared

Counting every site scanned in November 2026, 94% of the sites we could read passed the “Language and charset declared” check.

Show as a table
Language and charset declared, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%93%—
November 2026100%94%94%

Resource-discovery Link headers

Counting every site scanned in November 2026, 2% of the sites we could read passed the “Resource-discovery Link headers” check.

Show as a table
Resource-discovery Link headers, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%10%—
November 2026100%2%2%

Sensible internal/external link mix

Counting every site scanned in November 2026, 87% of the sites where it applies (90% of the sites we could read) passed the “Sensible internal/external link mix” check.

Show as a table
Sensible internal/external link mix, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202695%95%—
November 202690%87%87%

llms.txt is present and valid

Counting every site scanned in November 2026, 14% of the sites we could read passed the “llms.txt is present and valid” check.

Show as a table
llms.txt is present and valid, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
August 2026100%25%—
October 2026100%26%—
November 2026100%14%14%

Clean content for agents (Markdown)

Counting every site scanned in November 2026, 9% of the sites we could read passed the “Clean content for agents (Markdown)” check.

Show as a table
Clean content for agents (Markdown), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%7%—
November 2026100%9%9%

Has a meta description

Counting every site scanned in November 2026, 84% of the sites we could read passed the “Has a meta description” check.

Show as a table
Has a meta description, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%89%—
November 2026100%84%84%

No meta-refresh redirect

Counting every site scanned in November 2026, 99% of the sites we could read passed the “No meta-refresh redirect” check.

Show as a table
No meta-refresh redirect, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%100%—
November 2026100%99%99%

Not accidentally set to noindex

Counting every site scanned in November 2026, 99% of the sites we could read passed the “Not accidentally set to noindex” check.

Show as a table
Not accidentally set to noindex, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%100%—
November 2026100%99%99%

No mixed (insecure) content

Counting every site scanned in November 2026, 99% of the sites we could read passed the “No mixed (insecure) content” check.

Show as a table
No mixed (insecure) content, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%98%—
November 2026100%99%99%

Open Graph tags for sharing

Counting every site scanned in November 2026, 48% of the sites we could read passed the “Open Graph tags for sharing” check.

Show as a table
Open Graph tags for sharing, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%48%—
November 2026100%48%48%

Organization schema with sameAs

Counting every site scanned in November 2026, 14% of the sites we could read passed the “Organization schema with sameAs” check.

Show as a table
Organization schema with sameAs, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
August 2026100%29%—
October 2026100%31%—
November 2026100%14%14%

Links don't point at redirects

Counting every site scanned in November 2026, 84% of the sites where it applies (82% of the sites we could read) passed the “Links don't point at redirects” check.

Show as a table
Links don't point at redirects, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202692%93%—
November 202682%84%84%

robots.txt is present and not over-blocking

Counting every site scanned in November 2026, 78% of the sites we could read passed the “robots.txt is present and not over-blocking” check.

Show as a table
robots.txt is present and not over-blocking, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%85%—
November 2026100%78%78%

Not flagged for malware or phishing

Counting every site scanned in November 2026, 100% of the sites we could read passed the “Not flagged for malware or phishing” check.

Show as a table
Not flagged for malware or phishing, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%100%—
November 2026>99%100%100%

Search Console / Bing verification

Counting every site scanned in November 2026, 66% of the sites we could read passed the “Search Console / Bing verification” check.

Show as a table
Search Console / Bing verification, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%80%—
November 2026100%66%66%

Core security headers are set

Counting every site scanned in November 2026, 14% of the sites we could read passed the “Core security headers are set” check.

Show as a table
Core security headers are set, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%23%—
November 2026100%14%14%

security.txt is published

Counting every site scanned in November 2026, 4% of the sites we could read passed the “security.txt is published” check.

Show as a table
security.txt is published, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%20%—
November 2026100%4%4%

Semantic HTML landmarks

Counting every site scanned in November 2026, 52% of the sites we could read passed the “Semantic HTML landmarks” check.

Show as a table
Semantic HTML landmarks, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%67%—
November 2026100%52%52%

No server version leakage

Counting every site scanned in November 2026, 90% of the sites we could read passed the “No server version leakage” check.

Show as a table
No server version leakage, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%89%—
November 2026100%90%90%

The page has an H1

Counting every site scanned in November 2026, 63% of the sites we could read passed the “The page has an H1” check.

Show as a table
The page has an H1, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%79%—
November 2026100%63%63%

XML sitemap is present and referenced

Counting every site scanned in November 2026, 59% of the sites we could read passed the “XML sitemap is present and referenced” check.

Show as a table
XML sitemap is present and referenced, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
August 2026100%71%—
October 2026100%69%—
November 2026100%59%59%

Has a title tag

Counting every site scanned in November 2026, 99% of the sites we could read passed the “Has a title tag” check.

Show as a table
Has a title tag, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%98%—
November 2026100%99%99%

Title fits a search snippet

Counting every site scanned in November 2026, 40% of the sites where it applies (99% of the sites we could read) passed the “Title fits a search snippet” check.

Show as a table
Title fits a search snippet, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%52%—
November 202699%40%40%

Twitter/X card tags

Counting every site scanned in November 2026, 60% of the sites we could read passed the “Twitter/X card tags” check.

Show as a table
Twitter/X card tags, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%74%—
November 2026100%60%60%

Right schema type for the page

Counting every site scanned in November 2026, 66% of the sites where it applies (41% of the sites we could read) passed the “Right schema type for the page” check.

Show as a table
Right schema type for the page, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202657%80%—
November 202641%66%66%

Mobile viewport is set

Counting every site scanned in November 2026, 86% of the sites we could read passed the “Mobile viewport is set” check.

Show as a table
Mobile viewport is set, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%90%—
November 2026100%86%86%

AI catalog

Counting every site scanned in November 2026, 1% of the sites we could read passed the “AI catalog” check.

Show as a table
AI catalog, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%5%—
November 2026100%1%1%

Your content survives without JavaScript

Counting every site scanned in November 2026, 89% of the sites we could read passed the “Your content survives without JavaScript” check.

Show as a table
Your content survives without JavaScript, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 2026100%93%—
November 2026100%89%89%

A2A Agent Card

Counting every site scanned in November 2026, <1% of the sites we could read passed the “A2A Agent Card” check.

Show as a table
A2A Agent Card, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%5%—
November 2026100%<1%<1%

AI usage preferences (Content-Usage)

Counting every site scanned in November 2026, 0% of the sites we could read passed the “AI usage preferences (Content-Usage)” check.

Show as a table
AI usage preferences (Content-Usage), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%2%—
November 2026100%0%0%

Markdown version linked from the page

Counting every site scanned in November 2026, 1% of the sites we could read passed the “Markdown version linked from the page” check.

Show as a table
Markdown version linked from the page, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%8%—
November 2026100%1%1%

Licensing terms for AI (RSL)

Counting every site scanned in November 2026, <1% of the sites we could read passed the “Licensing terms for AI (RSL)” check.

Show as a table
Licensing terms for AI (RSL), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%2%—
November 2026100%<1%<1%

Text-and-data-mining reservation (TDMRep)

Counting every site scanned in November 2026, <1% of the sites we could read passed the “Text-and-data-mining reservation (TDMRep)” check.

Show as a table
Text-and-data-mining reservation (TDMRep), share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%2%—
November 2026100%<1%<1%

WebMCP tools in the page

Counting every site scanned in November 2026, 2% of the sites we could read passed the “WebMCP tools in the page” check.

Show as a table
WebMCP tools in the page, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%5%—
November 2026100%2%2%

Buttons, links and fields have names

Counting every site scanned in November 2026, 51% of the sites where it applies (92% of the sites we could read) passed the “Buttons, links and fields have names” check.

Show as a table
Buttons, links and fields have names, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202695%48%—
November 202692%51%51%

Your page's HTML isn't oversized

Counting every site scanned in November 2026, 98% of the sites we could read passed the “Your page's HTML isn't oversized” check.

Show as a table
Your page's HTML isn't oversized, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%95%—
November 2026100%98%98%

Description fits a search snippet

Counting every site scanned in November 2026, 54% of the sites where it applies (84% of the sites we could read) passed the “Description fits a search snippet” check.

Show as a table
Description fits a search snippet, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202687%60%—
November 202684%54%54%

Your page arrives quickly

Counting every site scanned in November 2026, 94% of the sites we could read passed the “Your page arrives quickly” check.

Show as a table
Your page arrives quickly, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%90%—
November 2026100%94%94%

Engines are allowed to quote the page

Counting every site scanned in November 2026, >99% of the sites we could read passed the “Engines are allowed to quote the page” check.

Show as a table
Engines are allowed to quote the page, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202698%92%—
November 2026100%>99%>99%

Links are real links

Counting every site scanned in November 2026, 81% of the sites where it applies (90% of the sites we could read) passed the “Links are real links” check.

Show as a table
Links are real links, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202693%88%—
November 202690%81%81%

Claims come with figures, quotes or sources

Counting every site scanned in November 2026, 51% of the sites where it applies (56% of the sites we could read) passed the “Claims come with figures, quotes or sources” check.

Show as a table
Claims come with figures, quotes or sources, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202674%53%—
November 202656%51%51%

Section headings are phrased as questions

Counting every site scanned in November 2026, 22% of the sites where it applies (50% of the sites we could read) passed the “Section headings are phrased as questions” check.

Show as a table
Section headings are phrased as questions, share of sites by month
MonthApplies toAll sites (new ones included)Same sites
October 202661%30%—
November 202650%22%22%

Webpage markup

Counting every site scanned in November 2026, 0% of the sites we could read published Webpage structured data as a type of its own.

Show as a table
Webpage markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20260%—
November 20260%0%

Brand markup

Counting every site scanned in November 2026, <1% of the sites we could read published Brand structured data as a type of its own.

Show as a table
Brand markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20262%—
November 2026<1%<1%

Organization markup

Counting every site scanned in November 2026, 19% of the sites we could read published Organization structured data as a type of its own.

Show as a table
Organization markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 202626%—
October 202631%—
November 202619%19%

SoftwareApplication markup

Counting every site scanned in November 2026, 2% of the sites we could read published SoftwareApplication structured data as a type of its own.

Show as a table
SoftwareApplication markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20268%—
October 20265%—
November 20262%2%

BreadcrumbList markup

Counting every site scanned in November 2026, 5% of the sites we could read published BreadcrumbList structured data as a type of its own.

Show as a table
BreadcrumbList markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20266%—
October 20265%—
November 20265%5%

WebSite markup

Counting every site scanned in November 2026, 30% of the sites we could read published WebSite structured data as a type of its own.

Show as a table
WebSite markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 202624%—
October 202626%—
November 202630%30%

WebPage markup

Counting every site scanned in November 2026, 7% of the sites we could read published WebPage structured data as a type of its own.

Show as a table
WebPage markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 202614%—
October 202626%—
November 20267%7%

ItemList markup

Counting every site scanned in November 2026, 2% of the sites we could read published ItemList structured data as a type of its own.

Show as a table
ItemList markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20263%—
October 20260%—
November 20262%2%

Article markup

Counting every site scanned in November 2026, 1% of the sites we could read published Article structured data as a type of its own.

Show as a table
Article markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20263%—
October 20267%—
November 20261%1%

NewsMediaOrganization markup

Counting every site scanned in November 2026, 1% of the sites we could read published NewsMediaOrganization structured data as a type of its own.

Show as a table
NewsMediaOrganization markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20262%—
November 20261%1%

NewsArticle markup

Counting every site scanned in November 2026, 0% of the sites we could read published NewsArticle structured data as a type of its own.

Show as a table
NewsArticle markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20265%—
October 20260%—
November 20260%0%

ImageObject markup

Counting every site scanned in November 2026, 1% of the sites we could read published ImageObject structured data as a type of its own.

Show as a table
ImageObject markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20263%—
November 20261%1%

FAQPage markup

Counting every site scanned in November 2026, 3% of the sites we could read published FAQPage structured data as a type of its own.

Show as a table
FAQPage markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20265%—
October 20265%—
November 20263%3%

CollectionPage markup

Counting every site scanned in November 2026, 1% of the sites we could read published CollectionPage structured data as a type of its own.

Show as a table
CollectionPage markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20265%—
October 20262%—
November 20261%1%

Corporation markup

Counting every site scanned in November 2026, 1% of the sites we could read published Corporation structured data as a type of its own.

Show as a table
Corporation markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20263%—
October 20262%—
November 20261%1%

WebApplication markup

Counting every site scanned in November 2026, 1% of the sites we could read published WebApplication structured data as a type of its own.

Show as a table
WebApplication markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20260%—
November 20261%1%

BlogPosting markup

Counting every site scanned in November 2026, <1% of the sites we could read published BlogPosting structured data as a type of its own.

Show as a table
BlogPosting markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20260%—
November 2026<1%<1%

SocialMediaPosting markup

Counting every site scanned in November 2026, 0% of the sites we could read published SocialMediaPosting structured data as a type of its own.

Show as a table
SocialMediaPosting markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20260%—
November 20260%0%

Review markup

Counting every site scanned in November 2026, 0% of the sites we could read published Review structured data as a type of its own.

Show as a table
Review markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20261%—
October 20260%—
November 20260%0%

Service markup

Counting every site scanned in November 2026, <1% of the sites we could read published Service structured data as a type of its own.

Show as a table
Service markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20263%—
November 2026<1%<1%

HowTo markup

Counting every site scanned in November 2026, <1% of the sites we could read published HowTo structured data as a type of its own.

Show as a table
HowTo markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20262%—
November 2026<1%<1%

TechArticle markup

Counting every site scanned in November 2026, 0% of the sites we could read published TechArticle structured data as a type of its own.

Show as a table
TechArticle markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20262%—
November 20260%0%

Person markup

Counting every site scanned in November 2026, 1% of the sites we could read published Person structured data as a type of its own.

Show as a table
Person markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20265%—
November 20261%1%

ProfessionalService markup

Counting every site scanned in November 2026, <1% of the sites we could read published ProfessionalService structured data as a type of its own.

Show as a table
ProfessionalService markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20262%—
November 2026<1%<1%

LocalBusiness markup

Counting every site scanned in November 2026, 3% of the sites we could read published LocalBusiness structured data as a type of its own.

Show as a table
LocalBusiness markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20263%—
November 20263%3%

ProfilePage markup

Counting every site scanned in November 2026, 0% of the sites we could read published ProfilePage structured data as a type of its own.

Show as a table
ProfilePage markup, share of sites by month
MonthAll sites (new ones included)Same sites
August 20260%—
October 20262%—
November 20260%0%

Sites scoring 81–90

Counting every site scanned in November 2026, 37% of the sites visitors scanned scored 81–90.

Show as a table
Sites scoring 81–90, share of sites by month
MonthAll sites (new ones included)Same sites
October 202648%—
November 202637%37%

Turned our scanner away

Counting every site scanned in November 2026, 9% of the sites visitors scanned turned our scanner away.

Show as a table
Turned our scanner away, share of sites by month
MonthAll sites (new ones included)Same sites
October 20265%—
November 20269%9%

Lines that start in November 2026

These lines have one month of figures so far. Each becomes a month-by-month line once a second month of figures arrives. Score ranges are counted from November 2026, when we began recording them, so they have no earlier months. A range with too few sites behind it is left out, so the lines can add up to less than all the sites.

Show as a table
Lines that start in November 2026
SeriesAll sites
Sites scoring 91–10022%
Sites scoring 71–8020%
Sites scoring 61–708%
Sites scoring 51–604%

Checks that have only one month of figures are not listed here again: their latest figure is in Every check.

When we changed how we judge

  1. Broken links, Safe Browsing and a first group of agent readiness checks
  2. Four new checks, and a caching check
  3. A site that turns our scanner away now gets a report, not an error screen
  4. The verdict now follows how well AI can read you, not just the total
  5. Eight checks were tightened after we tested them on real sites
  6. The AI crawler list was refreshed, and page weight is measured properly
  7. "We could not look" is no longer reported as "there is nothing there"
  8. A site that blocks AI crawlers can no longer look healthy
  9. A site that blocks our scanner gets no score at all
  10. Site scans no longer count the same page more than once
  11. A missing llms.txt is reported as a fail again
  12. Rebalanced one check that was counting for more than it proved
  13. A category we could not measure no longer shows as a zero
  14. Every scan now records which version of our scoring produced it
  15. We look in more than one place before telling you something is missing
  16. Your site score now reflects how much of your site is actually right
  17. Your score now counts only what decides whether a machine can read your page
  18. "AI access" now means access, and what you tell AI has its own place
  19. Site scans now use all the pages they are allowed to read
  20. Pages that build themselves in the browser are now read the way a browser reads them
  21. A "Robot or human?" screen is no longer scored as your site
  22. A paywall or a rate limit is no longer called a firewall
  23. A page withheld for legal reasons is no longer called bot protection
  24. A directory of AI crawlers, with who runs each one and what it is for
  25. A site scan now names the pages your robots.txt disallows for us
  26. New checks, and fewer signals only a person can see
  27. Short brand names are compared, and checks we could not run say why
  28. Five checks stopped misreading a few kinds of page
  29. One contact-file check moves to shown, not scored; two others are re-balanced
  30. Analytics look-alikes no longer count, and page titles read the way a browser reads them
  31. When we can't reach a page, we record why
  32. Site scans no longer test a "#" or "back to top" link as a link
  33. A Content-Signal mentioned in a robots.txt comment no longer counts
  34. Placeholder links no longer count as internal links
  35. Pages with search boxes and lazy-loaded images are read in full
  36. Link checks no longer test a link to the page itself
  37. The sitemap your robots.txt points to is now checked
  38. Structured data about a page's main subject is now read
  39. A site whose secure version answers slowly is no longer reported as not using https
  40. A slow secure version no longer marks a site down for its canonical or HSTS
  41. Analytics is found wherever it sits on the page, and more tools are recognised
  42. The www and non-www check no longer skips when the other name does not exist
  43. A link inside a button now names the button
  44. A button inside a link now names the link
  45. Structured data in menus, headers and footers is read where it belongs
  46. Titles, headings and redirects are read the way a browser reads them
  47. Organization markup is recognised for every kind of organization
  48. Fewer false passes on sites that answer missing files with a styled error page
  49. Service catalogs in your structured data are now read
  50. A site that asks us to slow down now gets a patient second look before a check is skipped
  51. The OAuth protected-resource check now reads the file instead of only finding it
  52. Your site scan now compares your pages with each other
  53. Very large pages now finish scanning instead of timing out
  54. Hidden template content no longer counts toward your headings
  55. A site scan no longer spends its pages on the same page twice

Based on the sites visitors have scanned with Bot Appétit. Not a random sample of the web.

July 2026, August 2026, September 2026, October 2026, November 2026

Readiness ladder

In November 2026, 16% of the sites visitors scanned cleared every one of the gates.

The gates are “Automated visitors can reach your pages”, “AI crawlers aren't blocked”, “Reached without a redirect chain”, “Your answer is in the HTML”, “Structured data is present”, “llms.txt is present and valid”: the basics a machine needs before it can use a page.

  1. 10% No gates cleared
  2. 4% One of six gates cleared
  3. 3% Two of six gates cleared
  4. 13% Three of six gates cleared
  5. 32% Four of six gates cleared
  6. 23% Five of six gates cleared
  7. 16% All six gates cleared
Show as a table
How many readiness gates sites clear
Gates clearedShare of sites
No gates cleared10%
One of six gates cleared4%
Two of six gates cleared3%
Three of six gates cleared13%
Four of six gates cleared32%
Five of six gates cleared23%
All six gates cleared16%

Based on the sites visitors have scanned with Bot Appétit. Not a random sample of the web.

Method

  • A site is a host exactly as we scanned it: the www version and the bare domain are different sites.
  • Each site counts once, on its latest scan of the homepage (or its latest scan of any page if it has no homepage scan). A scan that failed outright is never counted.
  • "Turned our scanner away" means the site answered our scanner with a refusal, a bot check or an empty page we could not read. A site that was down or timed out is left out, not counted as turning us away.
  • Figures about a single check count only the sites we could read: a site that turned our scanner away has no check results, so it is left out of them. Within those, a check that found nothing to judge on a page is counted only on the sites it applies to, and the "Applies to" column says how many that is.
  • The headline figures use scans from the last three months. The month-by-month lines use calendar months in UTC, so a month's figure can differ from the headline one.
  • Figures are whole percentages. A share above zero is never rounded down to nothing (it reads "<1%"), and one short of all is never rounded up to all (it reads ">99%"). The JSON file keeps numbers: 0.1 means under 1% and 99.9 means over 99%.
  • A figure appears only when enough different sites sit behind it. Below that it reads "not enough sites yet", and we never merge views to fill a gap.
  • We show statuses and sites counted in ten-point score ranges, never individual scores. A site we could not judge on a check is counted as "couldn't judge", not as a failure.
  • When we change how a check is judged, it is marked on the change-over-time lines and listed in the changelog.
  • Based on the sites visitors have scanned with Bot Appétit. Not a random sample of the web.

Figures generated 2026-11-15 from build 9b4aa1e7cd74f60e. Download the data as JSON or CSV; both hold percentages only.

These figures never name a site. To see which homepages score best, by category, open the Leaderboard.

How to cite this

Milk Moon Studio, "Bot Appétit State of the Web & AI Readiness", botappetit.ai/state-of-the-web/, figures from November 2026.

Figures change every hour, so say which month you took them from.

How does your site compare?

A free scan checks all of this on your page in seconds, with no signup.

Scan your site free

Want it handled? We do that too. No pitch, just a plain-English chat.

Book a call