# Can AI read this page? *An AI readiness check from your own log* > The page https://skaldanalytics.com/ai-traffic-analytics/ai-readiness/ as Markdown. Every page and guide: https://skaldanalytics.com/llms.txt . Skald is source-available (Elastic License 2.0). AI traffic · AI readiness A page can rank well and still give ChatGPT nothing. One robots.txt line keeps a crawler out, the Markdown copy a plugin points AI to answers 404, or the server takes two seconds when AI asks. Skald shows what each AI crawler really got back from each page. *AI readiness*, in Skald, is six checks on one page, each marked Good, Needs attention or Not known: may the main AI crawlers read it by your robots.txt, did AI get errors or slow answers, is it in your sitemap (the list of pages your site gives crawlers) and what date that list says it last changed (its lastmod), did its Markdown copy answer, did AI ask for it and get a 404, and when did an assistant last fetch it for someone. All six come from your server's access log and Skald's own copies of robots.txt and the sitemap. No AI is asked anything. [Start your 30-day free trial](https://skaldanalytics.com/beta/?choice=trial#signup) [Open the live demo](https://spcaeo.github.io/skald/) 30-day free trial · no card · we set it up within one working day · free to self-host - robots.txt for AI crawlers, read the… - What AI crawlers got back from the… - Markdown copies that answer 404 to AI - Pages AI can't read well, worst first - Where to find AI readiness, and what… Traffic › Pages › a page › AI readiness · sample pick a page 1. **robots.txt** *Good*All 9 main AI crawlers may read it (robots.txt as Skald read it on 25 Oct). 2. **Crawler errors** *Good*No error, redirect or slow answer to AI on this page in the last 30 days (30 recorded). 3. **In the sitemap** *Good*In your sitemap, last changed 14 Oct (its lastmod). 4. **Markdown copy** *Not known*AI did not ask for a Markdown copy of this page in the last 30 days, so Skald cannot tell whether one exists. 5. **AI asking for a 404** *Good*No AI request for this page got a 404 in the last 30 days (30 recorded). 6. **Last AI read** *Good*An assistant last fetched it for someone on 24 Oct (ChatGPT-User 2); 41 times in the last 30 days, 29 in the 30 before. Sample site. The rows shrink as on an eye chart and the lens reads them top to bottom. Day one is a new install before it holds any copy or recorded day: Not known, never a 0. [01] the robots.txt line ## robots.txt for AI crawlers, read the way the crawlers read it Your robots.txt may let GPTBot into /blog/ and still look closed to a quick reader. Skald judges each page for nine AI crawlers by RFC 9309, the robots.txt standard: of the rules that match the path, the longest wins, and an Allow wins a tie. The standard puts it as “The most specific match found MUST be used” and “the "allow" rule SHOULD be used” when the two are equivalent ([RFC 9309, section 2.2.2, read 25 Oct 2026](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2)). Each rule below is a strip of tape as long as its pattern. pick a case ``` User-agent: * Disallow: / Allow: /blog/ ``` `GPTBot` asks for `/blog/launch/` 1. Disallow / 1 character wins 2. Allow /blog/ 6 characters wins Both rules match. Allow: /blog/ is 6 characters, Disallow: / is 1, so the longer one wins: GPTBot may read /blog/launch/. A first-match reader stops at Disallow: /, the first rule that matches, and keeps GPTBot out of /blog/launch/. Sample robots.txt files. A first-match reader takes the first rule that matches, whatever its length. Skald's reader, and the separate reader its second count uses, follow RFC 9309. [block AI crawlers](https://skaldanalytics.com/ai-traffic-analytics/block-ai-crawlers/) [AI crawler tracking](https://skaldanalytics.com/ai-traffic-analytics/ai-crawlers/) [02] the crawler errors line ## What AI crawlers got back from the page A crawler that gets a 403 or a 503 leaves with nothing, and only your server log records it. Skald records every answer an AI crawler or assistant got that was anything but a plain 200 or 304, and every answer that took a second or more when your log records the time. The Crawler errors line sums the last 30 days for one page. pick a page Crawler errors · **Good** No error, redirect or slow answer to AI on this page in the last 30 days (30 recorded). - 200 or 304: read fine - 3xx: named, not counted - 404 or 410: its own line - 401, 403, 429: refused - 5xx: server error - large dart: a second or more Sample: one dart per AI request in 30 days, by the answer it got. A slow answer is counted whatever its status; in this sample the slow ones were 200s. pick an address the page `/guides/webhooks/` ** `/guides/webhooks.md` · 404 404nothing to read **Markdown copy · Needs attention** AI asked for the Markdown copy 6 times; it answered 404 each time. Sample site. The copy's address is laid over the page it belongs to; a copy that answered 404 is a blank sheet. [03] the Markdown copy line ## Markdown copies that answer 404 to AI Some plugins and site builders send AI assistants to a page's Markdown copy, a plain-text version with no menus. If nobody made that copy, the assistant asked for your page and got a 404. Skald files each copy under the page it copies and says what it answered. - `/a/b.md`, `/a/b/index.md` and `/a/b/.md` are copies of `/a/b/`; `/index.md` is the home page's. `.mdx` and `.markdown` count the same, in any case. - A copy that answered anything but a 2xx or a 304 makes the line Needs attention, with what to do: fix the copy, or stop sending AI to it. - Until AI asks for a copy, the line is Not known. Skald reads only what AI asked for; it sends no requests of its own. - On skaldanalytics.com the 2.35.0 recount found 874 requests for .md copies, 164 of them by AI assistants ([Skald's 2.35.0 release notes (25 Oct 2026)](https://github.com/spcaeo/skald/blob/main/docs/CHANGELOG-2.35.md#2350-2026-10-25)). [Markdown copies for AI, in the log](https://skaldanalytics.com/insights/markdown-copies-for-ai/) [llms.txt](https://skaldanalytics.com/ai-traffic-analytics/llms-txt/) [04] the whole site ## Pages AI can't read well, worst first Opening pages one by one to find the broken ones takes too long. AI & bots › Assistants lists every page where AI got an error, a 404 or a slow answer, a Markdown copy that answered with an error, or that robots.txt shuts a main crawler out of. The heaviest problems hang lowest: failed answers first, then robots.txt, then slow ones. filter it 1. `/guides/api-v1/`the page: 404 9 times**Bring the page back or redirect it** 2. `/features/`refused 7 times**Let AI crawlers through** 3. `/guides/webhooks/`Markdown copy: 404 6 times · robots.txt shuts out Google-Extended**Fix the 404 on the Markdown copy** 4. `/status/`server errors 5 times**Fix the server error** 5. `/guides/getting-started/`robots.txt shuts out Google-Extended**Allow Google-Extended in robots.txt** 6. `/customers/`slow 4 times**Speed the page up** Sample site. Only addresses shaped like a page are listed: files, feeds, APIs, sitemaps, llms.txt and probes such as /.env are left out. A crawler robots.txt keeps out of the whole site is said once, above the list. Select a row in the app and that page's AI readiness opens. [Find the pages AI can't read, 30 days free](https://skaldanalytics.com/beta/?choice=trial#signup) [Open the demo's list](https://spcaeo.github.io/skald/) [404 redirects](https://skaldanalytics.com/features/404-redirects/) [AI crawl health](https://skaldanalytics.com/ai-traffic-analytics/crawl-health/) [05] where it lives ## Where to find AI readiness, and what it leaves out The six lines sit in a page's story, so you see them next to that page's visits and sources. The address keeps the open block, so a link to it opens the same view for a teammate. 1. **Open a page** Traffic › Pages, select a page, then AI readiness. Crawler names, status codes and robots.txt lines each have a copy button; every path has copy and open. 2. **Read the whole site** AI & bots › Assistants, the list of pages AI can't read well, filtered by kind. The filter is in the address too. 3. **Fix, then look again** Each line is worked out again with every data update, so a fixed copy or a new Allow line turns Good on its own. [AI traffic tracking](https://skaldanalytics.com/ai-traffic-analytics/) [SEO to-do list](https://skaldanalytics.com/features/seo-to-do-list/) ### The limits of the AI readiness check - It reads what AI got back, in your own log. It asks no AI anything and loads no page itself. - What AI got back is recorded from Skald 2.35.0 on. Older days say Not known until they are counted again from raw logs you still keep. - A redirect is named and left out of the count: a log line has no record of where it led. - robots.txt is a request to crawlers. Assistants that fetch a page for one person may skip it: OpenAI says of ChatGPT-User “robots.txt rules may not apply” ([OpenAI's crawler page, read 25 Oct 2026](https://developers.openai.com/api/docs/bots)), and Perplexity says Perplexity-User “generally ignores robots.txt rules” ([Perplexity's crawler page, read 25 Oct 2026](https://docs.perplexity.ai/guides/bots)). A Good robots.txt line describes your file; the other lines come from what was actually fetched. - It keeps 1,000 addresses a day by name and sums the rest; on such a day a line says the page may be among the summed ones. - A page group (a view) shows it for the whole site only. keep reading ## More on AI readiness check - [AI crawl health](https://skaldanalytics.com/ai-traffic-analytics/crawl-health/): Sitemap coverage by Google, Bing and each AI crawler, and the pages they miss. - [Markdown copies for AI](https://skaldanalytics.com/insights/markdown-copies-for-ai/): What a page.md request gets back, the 404s, and why page views fell in 2.35.0. - [Skald pricing](https://skaldanalytics.com/pricing/): Per site, a 30-day free trial with no card, or free to self-host. questions ## AI readiness check FAQ Straight answers, including what Skald does not do. [Read the docs](https://skaldanalytics.com/docs/) [Ask us a question](https://skaldanalytics.com/contact/) **What does Skald's AI readiness check look at?** Six things for one page: whether robots.txt lets the nine main AI crawlers read it, the server errors, refusals and slow answers AI got from it in the last 30 days, whether it is in your sitemap, whether the Markdown copy AI asked for answered, whether AI asked for it and got a 404, and when an assistant last fetched it for someone. Open a page in Traffic › Pages to see them. **Does Skald ask ChatGPT or another AI about my page?** No. Every line comes from your own server's access log and the copies of robots.txt and the sitemap that Skald keeps. Skald sends nothing to an AI company and makes no AI call to judge a page. **Why does a line say Not known?** Because Skald is missing what that line needs: no copy of robots.txt or the sitemap yet, days before Skald started recording what AI got back, or a page whose Markdown copy AI has yet to ask for. It says Not known instead of guessing, and shows a 0 only when it counted one. **Why are redirects not counted as a problem?** A log line says a page answered 301 or 302, but not where the redirect led. A site may send AI to the page's Markdown copy on purpose, so Skald names the redirects on the Crawler errors line and judges the copy on its own line. **Which AI crawlers does the robots.txt line check?** Nine: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended and Applebot-Extended. The last two fetch nothing: they are robots.txt names that switch the use of what Googlebot and Applebot fetched on or off. Google says “Google-Extended doesn't have a separate HTTP request user agent string” ([Google's common crawlers page, read 25 Oct 2026](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)), and Apple says “Applebot-Extended does not crawl webpages” ([Apple's Applebot page, read 25 Oct 2026](https://support.apple.com/en-us/119829)). **Can Skald fix a page for me?** No. Skald reads your log and changes nothing on your server. Where robots.txt is the cause, it writes the Allow lines to copy into your file; the rest of the fixes are yours, and the page's lines show when they worked.