Research

Astro AI readiness statistics

In November 2026, we could only tell which sites run on Astro where our scanner could read the page, so this page covers those sites and says nothing about how many turned our scanner away.

Updated 11:07 UTC · November 2026

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

All sites on the State of the Web

How Astro sites land

Every page we read gets a score out of 100, the number at the top of its report (how scoring works). Each row is a range of scores. Where too few sites score in the lowest ranges, they are joined into one row.

Not enough sites yet There are not enough sites yet to split them by score.

A platform is only detectable on a page our scanner could read, so these shares are of the Astro sites we could read.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

Better or worse

Not enough history yet.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

AI access

robots.txt disallows some AI crawlers 32%
robots.txt disallows major AI crawlers 5%

Show as a table

How Astro sites treat AI crawlers
What we found	Share of sites
robots.txt disallows some AI crawlers	32%
robots.txt disallows major AI crawlers	5%

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

What Astro sites get right and wrong

Passed more often than on the average site: “Semantic HTML landmarks” (89%, against 53% of all sites); “llms.txt is present and valid” (46%, against 15% of all sites); “Topic is consistent across title, H1 and description” (78%, against 47% of all sites). Least often passed: several checks that no Astro site we could read passed.

Every check, Astro against all sites

Share of sites that pass each check, over the sites the check applies to. The Gap column is the difference in percentage points between the two shares; a dash means not enough sites yet.

Astro sites against all sites: share passing each check
Check	Astro sites	All sites	− 0 +	Gap (points)
Served over a secure HTTPS connection	100%	>99%		Level
HTTP redirects to HTTPS	84%	88%		−4 points
No mixed (insecure) content	100%	99%		+1 point
Core security headers are set	41%	15%		+26 points
Not flagged for malware or phishing	100%	100%		Level
No server version leakage	100%	90%		+10 points
security.txt is published	14%	6%		+8 points
Reached without a redirect chain	97%	98%		−1 point
Caching or validation headers are set	92%	92%		Level
The site settles on one host (www vs non-www)	84%	86%		−2 points
robots.txt is present and not over-blocking	97%	79%		+18 points
Not accidentally set to noindex	97%	99%		−2 points
This page declares its canonical URL	92%	64%		+28 points
Mobile viewport is set	95%	85%		+10 points
Has a title tag	100%	99%		+1 point
Title fits a search snippet	68%	42%		+26 points
Has a meta description	95%	82%		+13 points
Description fits a search snippet	71%	55%		+16 points
The page has an H1	86%	63%		+23 points
Logical heading order	76%	60%		+16 points
Topic is consistent across title, H1 and description	78%	47%		+31 points
Images have descriptive alt text	97%	82%		+15 points
Your page's HTML isn't oversized	100%	97%		+3 points
Open Graph tags for sharing	65%	48%		+17 points
Twitter/X card tags	78%	62%		+16 points
XML sitemap is present and referenced	84%	60%		+24 points
Missing pages return a real 404	100%	85%		+15 points
Images are optimised	41%	22%		+19 points
Language and charset declared	100%	94%		+6 points
hreflang for language variants	—	99%		—
Analytics is installed	68%	61%		+7 points
Search Console / Bing verification	92%	68%		+24 points
A favicon is set	95%	89%		+6 points
No meta-refresh redirect	100%	99%		+1 point
Your content survives without JavaScript	94%	89%		+5 points
Your answer is in the HTML	95%	86%		+9 points
Buttons, links and fields have names	71%	50%		+21 points
Semantic HTML landmarks	89%	53%		+36 points
Content is scannable in self-contained chunks	92%	78%		+14 points
Claims come with figures, quotes or sources	—	51%		—
Section headings are phrased as questions	—	22%		—
Reading the page is affordable for a machine	70%	84%		−14 points
No hidden text written for an AI model	100%	100%		Level
Sitemap dates tell a machine what changed	—	57%		—
The links in llms.txt resolve	—	93%		—
Author / publisher is clear	57%	38%		+19 points
A machine-readable published / updated date	11%	13%		−2 points
llms.txt is present and valid	46%	15%		+31 points
Clean content for agents (Markdown)	14%	9%		+5 points
Structured data is present	62%	45%		+17 points
Structured data is valid	—	97%		—
FAQ / Q&A schema	—	36%		—
Organization schema with sameAs	41%	16%		+25 points
Right schema type for the page	—	69%		—
Schema facts also appear in the page text	—	53%		—
Breadcrumb structured data	—	13%		—
AI crawlers aren't blocked	68%	83%		−15 points
Automated visitors can reach your pages	—	—		—
Engines are allowed to quote the page	94%	99%		−5 points
Your page arrives quickly	91%	94%		−3 points
AI bot rules are explicit and intentional	41%	19%		+22 points
Content usage preferences declared	22%	5%		+17 points
Resource-discovery Link headers	16%	2%		+14 points
No broken links	97%	89%		+8 points
Links don't point at redirects	89%	85%		+4 points
Descriptive anchor text	100%	>99%		Level
Sensible internal/external link mix	95%	88%		+7 points
Links are real links	86%	81%		+5 points
Largest Contentful Paint (LCP)	—	—		—
Cumulative Layout Shift (CLS)	—	—		—
Interaction to Next Paint (INP)	—	—		—
Page weight and render-blocking	—	—		—
MCP Server Card	11%	1%		+10 points
AI catalog	3%	1%		+2 points
A2A Agent Card	9%	1%		+8 points
WebMCP tools in the page	3%	2%		+1 point
AI usage preferences (Content-Usage)	0%	<1%		Level
Licensing terms for AI (RSL)	0%	<1%		Level
Text-and-data-mining reservation (TDMRep)	0%	<1%		Level
Markdown version linked from the page	6%	1%		+5 points
Content signals agree everywhere	—	—		—
Agent Skills index	14%	1%		+13 points
API catalog	8%	1%		+7 points
OAuth authorization-server metadata	8%	4%		+4 points
OAuth protected-resource metadata	5%	4%		+1 point
OpenID Connect discovery	3%	4%		−1 point
Web Bot Auth (operates a signed bot)	0%	<1%		Level
Universal Commerce Protocol manifest	—	—		—
Auth.md agent registration	8%	1%		+7 points
AI discovery via DNS	0%	<1%		Level

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

See how your site compares

A free scan checks all of this on your page in seconds, with no signup.

Scan your site free

Want it handled? We do that too. No pitch, just a plain-English chat.