AI traffic · AI readiness
Can AI read this page? An AI readiness check from your own log
A page can rank well and still give ChatGPT nothing. One robots.txt line keeps a crawler out, the Markdown copy a plugin points AI to answers 404, or the server takes two seconds when AI asks. Skald shows what each AI crawler really got back from each page.
AI readiness, in Skald, is six checks on one page, each marked Good, Needs attention or Not known: may the main AI crawlers read it by your robots.txt, did AI get errors or slow answers, is it in your sitemap (the list of pages your site gives crawlers) and what date that list says it last changed (its lastmod), did its Markdown copy answer, did AI ask for it and get a 404, and when did an assistant last fetch it for someone. All six come from your server's access log and Skald's own copies of robots.txt and the sitemap. No AI is asked anything.
30-day free trial · no card · we set it up within one working day · free to self-host
pick a page
- robots.txt GoodAll 9 main AI crawlers may read it (robots.txt as Skald read it on 25 Oct).
- Crawler errors GoodNo error, redirect or slow answer to AI on this page in the last 30 days (30 recorded).
- In the sitemap GoodIn your sitemap, last changed 14 Oct (its lastmod).
- Markdown copy Not knownAI did not ask for a Markdown copy of this page in the last 30 days, so Skald cannot tell whether one exists.
- AI asking for a 404 GoodNo AI request for this page got a 404 in the last 30 days (30 recorded).
- Last AI read GoodAn assistant last fetched it for someone on 24 Oct (ChatGPT-User 2); 41 times in the last 30 days, 29 in the 30 before.
- robots.txt Needs attentionrobots.txt keeps Google-Extended out of this page. The rule: Disallow: /guides/, under User-agent: Google-Extended.
- Crawler errors GoodNo error, redirect or slow answer to AI on this page in the last 30 days (30 recorded).
- In the sitemap GoodIn your sitemap, last changed 9 Oct (its lastmod).
- Markdown copy Needs attentionAI asked for the Markdown copy 6 times; it answered 404 each time.
- AI asking for a 404 GoodNo AI request for this page got a 404 in the last 30 days (30 recorded).
- Last AI read GoodAn assistant last fetched it for someone on 23 Oct (Claude-User 1); 12 times in the last 30 days, 15 in the 30 before.
- robots.txt GoodAll 9 main AI crawlers may read it (robots.txt as Skald read it on 25 Oct).
- Crawler errors GoodNo error, redirect or slow answer to AI on this page in the last 30 days (30 recorded).
- In the sitemap GoodIn your sitemap (read 25 Oct, 48 pages), without a last-changed date (lastmod).
- Markdown copy Not knownAI did not ask for a Markdown copy of this page in the last 30 days, so Skald cannot tell whether one exists.
- AI asking for a 404 Needs attentionAI asked for this page 9 times and got 404 (GPTBot 5, ClaudeBot 3, PerplexityBot 1).
- Last AI read Needs attentionNo assistant fetched it for someone in the last 30 days; the last time was 2 Oct.
- robots.txt Not knownSkald has no copy of the site's robots.txt yet.
- Crawler errors Not knownNot recorded yet: what AI gets back is kept from the next collected day on.
- In the sitemap Not knownSkald has no copy of the site's sitemap yet.
- Markdown copy Not knownNot recorded yet: what AI gets back is kept from the next collected day on.
- AI asking for a 404 Not knownNot recorded yet: what AI gets back is kept from the next collected day on.
- Last AI read Not knownNot recorded yet: the assistants' reads by page come with the next redraw.
Sample site. The rows shrink as on an eye chart and the lens reads them top to bottom. Day one is a new install before it holds any copy or recorded day: Not known, never a 0.
[01] the robots.txt line
robots.txt for AI crawlers, read the way the crawlers read it
Your robots.txt may let GPTBot into /blog/ and still look closed to a quick reader. Skald judges each page for nine AI crawlers by RFC 9309, the robots.txt standard: of the rules that match the path, the longest wins, and an Allow wins a tie. The standard puts it as “The most specific match found MUST be used” and “the "allow" rule SHOULD be used” when the two are equivalent (RFC 9309, section 2.2.2, read 25 Oct 2026). Each rule below is a strip of tape as long as its pattern.
User-agent: * Disallow: / Allow: /blog/
GPTBot asks for /blog/launch/
- Disallow/1 characterwins
- Allow/blog/6 characterswins
Both rules match. Allow: /blog/ is 6 characters, Disallow: / is 1, so the longer one wins: GPTBot may read /blog/launch/.
A first-match reader stops at Disallow: /, the first rule that matches, and keeps GPTBot out of /blog/launch/.
User-agent: ClaudeBot Disallow: /guides/ Allow: /guides/
ClaudeBot asks for /guides/
- Disallow/guides/8 characterswins
- Allow/guides/8 characterswins
Both rules are 8 characters. In a tie the Allow wins: ClaudeBot may read /guides/.
A first-match reader stops at Disallow: /guides/ and keeps ClaudeBot out.
User-agent: * Allow: /docs/ Disallow: /docs/*/draft
GPTBot asks for /docs/api/draft/
- Allow/docs/6 characterswins
- Disallow/docs/*/draft13 characterswins
The * stands for any characters, so Disallow: /docs/*/draft matches, and at 13 characters it beats Allow: /docs/ (6). GPTBot is kept out of /docs/api/draft/.
A first-match reader stops at Allow: /docs/ and lets GPTBot in, where the RFC 9309 reading keeps it out.
User-agent: * Disallow: /signup/thanks User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Google-Extended Disallow: /guides/
Google-Extended asks for /guides/webhooks/
- Disallow/guides/8 characterswins
Google-Extended has a group of its own, so only its own group's rules count. Disallow: /guides/ matches: Google-Extended is kept out of /guides/webhooks/. Skald writes the lines that would let it in.
Here both readings agree: one rule matches, and it shuts Google-Extended out.
User-agent: Google-Extended Allow: /guides/webhooks/
Sample robots.txt files. A first-match reader takes the first rule that matches, whatever its length. Skald's reader, and the separate reader its second count uses, follow RFC 9309.
[02] the crawler errors line
What AI crawlers got back from the page
A crawler that gets a 403 or a 503 leaves with nothing, and only your server log records it. Skald records every answer an AI crawler or assistant got that was anything but a plain 200 or 304, and every answer that took a second or more when your log records the time. The Crawler errors line sums the last 30 days for one page.
Crawler errors · Good
No error, redirect or slow answer to AI on this page in the last 30 days (30 recorded).
Crawler errors · Needs attention
In the last 30 days (30 recorded) AI got refused (403) 7 times.
What to do: Let AI crawlers through: the server or a firewall refuses them on this page.
Crawler errors · Needs attention
In the last 30 days (30 recorded) AI got slow answers (a second or more) 4 times.
What to do: Speed the page up.
Crawler errors · Good
No error or slow answer to AI on this page in the last 30 days (30 recorded); it was redirected (301) 6 times. The log does not say where a redirect leads, so it is not counted as a problem.
Crawler errors · Needs attention
In the last 30 days (30 recorded) AI got server errors (503) 5 times.
What to do: Fix the server error AI crawlers get on this page.
- 200 or 304: read fine
- 3xx: named, not counted
- 404 or 410: its own line
- 401, 403, 429: refused
- 5xx: server error
- large dart: a second or more
Sample: one dart per AI request in 30 days, by the answer it got. A slow answer is counted whatever its status; in this sample the slow ones were 200s.
pick an address
/guides/webhooks//guides/webhooks.md · 404404nothing to readMarkdown copy · Needs attention AI asked for the Markdown copy 6 times; it answered 404 each time.
/blog/fall-launch//blog/fall-launch.md · 200# HeadingPlain text, no menus.- a listMarkdown copy · Good AI asked for the Markdown copy 8 times; it answered 200 each time.
//index.md · 200# HeadingPlain text, no menus.- a listMarkdown copy · Good AI asked for the Markdown copy 5 times; it answered 200 each time.
/docs/api//docs/api/index.mdx · 200# HeadingPlain text, no menus.- a listone ask answered 404Markdown copy · Needs attention AI asked for the Markdown copy 3 times; once it answered 404.
Sample site. The copy's address is laid over the page it belongs to; a copy that answered 404 is a blank sheet.
[03] the Markdown copy line
Markdown copies that answer 404 to AI
Some plugins and site builders send AI assistants to a page's Markdown copy, a plain-text version with no menus. If nobody made that copy, the assistant asked for your page and got a 404. Skald files each copy under the page it copies and says what it answered.
/a/b.md,/a/b/index.mdand/a/b/.mdare copies of/a/b/;/index.mdis the home page's..mdxand.markdowncount the same, in any case.- A copy that answered anything but a 2xx or a 304 makes the line Needs attention, with what to do: fix the copy, or stop sending AI to it.
- Until AI asks for a copy, the line is Not known. Skald reads only what AI asked for; it sends no requests of its own.
- On skaldanalytics.com the 2.35.0 recount found 874 requests for .md copies, 164 of them by AI assistants (Skald's 2.35.0 release notes (25 Oct 2026)).
[04] the whole site
Pages AI can't read well, worst first
Opening pages one by one to find the broken ones takes too long. AI & bots › Assistants lists every page where AI got an error, a 404 or a slow answer, a Markdown copy that answered with an error, or that robots.txt shuts a main crawler out of. The heaviest problems hang lowest: failed answers first, then robots.txt, then slow ones.
/guides/api-v1/the page: 404 9 timesBring the page back or redirect it/features/refused 7 timesLet AI crawlers through/guides/webhooks/Markdown copy: 404 6 times · robots.txt shuts out Google-ExtendedFix the 404 on the Markdown copy/status/server errors 5 timesFix the server error/guides/getting-started/robots.txt shuts out Google-ExtendedAllow Google-Extended in robots.txt/customers/slow 4 timesSpeed the page up
Sample site. Only addresses shaped like a page are listed: files, feeds, APIs, sitemaps, llms.txt and probes such as /.env are left out. A crawler robots.txt keeps out of the whole site is said once, above the list. Select a row in the app and that page's AI readiness opens.
[05] where it lives
Where to find AI readiness, and what it leaves out
The six lines sit in a page's story, so you see them next to that page's visits and sources. The address keeps the open block, so a link to it opens the same view for a teammate.
Open a page
Traffic › Pages, select a page, then AI readiness. Crawler names, status codes and robots.txt lines each have a copy button; every path has copy and open.
Read the whole site
AI & bots › Assistants, the list of pages AI can't read well, filtered by kind. The filter is in the address too.
Fix, then look again
Each line is worked out again with every data update, so a fixed copy or a new Allow line turns Good on its own.
The limits of the AI readiness check
- It reads what AI got back, in your own log. It asks no AI anything and loads no page itself.
- What AI got back is recorded from Skald 2.35.0 on. Older days say Not known until they are counted again from raw logs you still keep.
- A redirect is named and left out of the count: a log line has no record of where it led.
- robots.txt is a request to crawlers. Assistants that fetch a page for one person may skip it: OpenAI says of ChatGPT-User “robots.txt rules may not apply” (OpenAI's crawler page, read 25 Oct 2026), and Perplexity says Perplexity-User “generally ignores robots.txt rules” (Perplexity's crawler page, read 25 Oct 2026). A Good robots.txt line describes your file; the other lines come from what was actually fetched.
- It keeps 1,000 addresses a day by name and sums the rest; on such a day a line says the page may be among the summed ones.
- A page group (a view) shows it for the whole site only.
keep reading
More on AI readiness check
questions
AI readiness check FAQ
Straight answers, including what Skald does not do.
What does Skald's AI readiness check look at?
Six things for one page: whether robots.txt lets the nine main AI crawlers read it, the server errors, refusals and slow answers AI got from it in the last 30 days, whether it is in your sitemap, whether the Markdown copy AI asked for answered, whether AI asked for it and got a 404, and when an assistant last fetched it for someone. Open a page in Traffic › Pages to see them.
Does Skald ask ChatGPT or another AI about my page?
No. Every line comes from your own server's access log and the copies of robots.txt and the sitemap that Skald keeps. Skald sends nothing to an AI company and makes no AI call to judge a page.
Why does a line say Not known?
Because Skald is missing what that line needs: no copy of robots.txt or the sitemap yet, days before Skald started recording what AI got back, or a page whose Markdown copy AI has yet to ask for. It says Not known instead of guessing, and shows a 0 only when it counted one.
Why are redirects not counted as a problem?
A log line says a page answered 301 or 302, but not where the redirect led. A site may send AI to the page's Markdown copy on purpose, so Skald names the redirects on the Crawler errors line and judges the copy on its own line.
Which AI crawlers does the robots.txt line check?
Nine: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended and Applebot-Extended. The last two fetch nothing: they are robots.txt names that switch the use of what Googlebot and Applebot fetched on or off. Google says “Google-Extended doesn't have a separate HTTP request user agent string” (Google's common crawlers page, read 25 Oct 2026), and Apple says “Applebot-Extended does not crawl webpages” (Apple's Applebot page, read 25 Oct 2026).
Can Skald fix a page for me?
No. Skald reads your log and changes nothing on your server. Where robots.txt is the cause, it writes the Allow lines to copy into your file; the rest of the fixes are yours, and the page's lines show when they worked.
$ tail -f access.log | skald
See what AI does with your site. Free for 30 days.
No card. We set up hosted Skald for you within one working day, or you self-host it free. Every line of your log, sorted and added up.