Skip to content
New storyWe built Skald to count visitors. It caught an AI-key thief.
SkaldEvery visit, truly told.
Live demo Start free trial

AI traffic · read from access.log

AI traffic tracking: crawls, live fetches and clicks, kept apart

AI traffic tracking starts where your analytics stops. ChatGPT, Claude and Perplexity read your pages without running a script. Your server log saw every one, and Skald sorts them by why they came.

AI traffic tracking means counting what AI does on your website: training crawlers that read it, assistants that fetch a page while answering someone, and people who click a link in an AI answer. Skald reads all three from your server's access log, because none of the machines run a script tag.

Start your 30-day free trial Open the live demo

30-day free trial · no card · we set it up within one working day · free to self-host

access.log · sample · sorted as it arrives

09:14:01 203.0.113.24 "GET /pricing/" "GPTBot/1.2"

  1. 1 name
  2. 2 address
  3. 3 referrer

GPTBot is on the training list. Its address is on OpenAI's published list.

1Crawlstrain or index, nobody waiting
0Live fetchessomeone is waiting
0Clicks from AIpeople, source shown
0Fakesfiled under bots

lines read 1=every line in exactly one place

Sample lines, sorted by the same three checks Skald runs on your log.

[01] the blind spot

Your analytics says AI barely visits. Your log disagrees.

AI crawlers and assistants fetch your pages without running a single line of JavaScript. A script tag cannot count what never runs it. Your web server writes a line for every request anyway, and that is the line Skald reads.

Drag the bar across one sample day: the same site, seen by a script tag and by its own log.

Read from your server logWhy Google Analytics misses AI crawlers
  • Visits by people412
  • Training crawls2,310
  • Live fetches for a question186
  • Clicks from an AI answer9
  • Likely from AI, kept apart+14
  • Fake AI crawlers73

Sample day. People are counted either way; the machines only show up in the log.

[02] who is waiting

One question to ChatGPT leaves three different traces in your log

A crawl at 3 a.m. is a company building a model. A fetch at 09:14 is a person waiting for an answer about you. A click is that person arriving. Skald files each by why it came, so demand never hides inside crawling.

step through it

A person

"Which analytics shows AI crawlers?"
Clicks your link in the answer

The AI company

GPTBot reads /pricing/ for training
ChatGPT starts an answer
ChatGPT-User fetches /compare/plausible/ to answer

Your access.log

"GET /pricing/" "GPTBot/1.2" training

nothing yet: the question happens inside ChatGPT

"GET /compare/plausible/" "ChatGPT-User/1.0" live fetch

"GET /compare/plausible/" ref chatgpt.com person · from ChatGPT

Training crawl. GPTBot, ClaudeBot, CCBot and others read to train a model; OAI-SearchBot and PerplexityBot read to build an AI search index. Nobody is waiting. You see what each one read and what it cost you. AI crawler tracking

[03] trust the names

Fake "GPTBots" would inflate your AI numbers. Skald catches them.

A user agent is the name a program sends with each request. Anyone can type one, and scanners wear GPTBot's name to look harmless. Skald checks the address behind each claim the way its owner says to. A fake moves to bots as "claims to be GPTBot".

pick a request
Who is probing your site

203.0.113.24 "GET /pricing/" "GPTBot/1.2"

check: Address list

  1. OpenAI publishes the addresses its crawlers use.
  2. 203.0.113.24 is inside one of them.

Verified

Counted as AI · training crawl.

Sample addresses. The owners' lists are fetched once a day; DNS answers are kept for 7 days.

[04] the clicks AI apps hide

ChatGPT sent them. Your analytics says "Direct".

Many AI apps strip the referrer, the address a click came from, so a click from an answer lands as a direct visit. But when the person asked, the assistant fetched that same page from your server minutes earlier. Skald lines the two up and marks the visit "likely", apart from every real count.

likely from ChatGPT

  1. 08:52 ChatGPT-User fetched /pricing/
  2. 08:57 Claude-User fetched /pricing/
  3. 09:02 ChatGPT-User fetched /compare/plausible/
  4. 09:03 ChatGPT-User fetched /pricing/
  5. 09:08 Perplexity-User fetched /docs/setup/
  6. 09:14 A visit lands on /pricing/ with no referrer and no campaign tag

what the AI read before it sent this visitor

  1. 09:03 /pricing/ landed here
  2. 09:02 /compare/plausible/

ChatGPT-User fetched /pricing/ 11 minutes before, inside the window. The visit is marked likely from ChatGPT.

  • Same page only. Perplexity fetched /docs/setup/, not /pricing/, so it never counts for this visit.
  • Two assistants, one visit. When both fetched the page, the visit counts once, as "more than one", never twice.
  • Never added. Likely visits sit beside the real figures. The window is a setting, 15 minutes by default.
Follow it to a lead
Find the AI visits hiding in "Direct" See it in the live demo

[05] llms.txt, Markdown, robots.txt

Stop guessing whether AI reads your llms.txt

llms.txt is a short text file at your site's root that points AI tools to your main pages. Every AI request is filed by what its path asks for: a page, llms.txt, robots.txt, Markdown, a sitemap, plain text or a file. A 404 for /llms.txt still counts as asking for it, so you see the demand before you write the file.

switch

Pages164

llms.txt9

robots.txt3

Markdown7

Sitemap1

Sample day.

suggested /llms.txt · built from your log
# Example Site

> What Example Site does, in one line from its home page.

## Main pages

- [Pricing](https://example.com/pricing/)
- [Compare with Plausible](https://example.com/compare/plausible/)

## Docs

- [Set up in one command](https://example.com/docs/setup/)

## Optional

- [About](https://example.com/about/)

Ranked by 90 days of people's page views plus AI fetches made for someone's question. Pages every crawler is kept from, redirected or gone pages and pages nobody read are left out. Nothing is written to your site: you decide whether to publish it.

llms.txt

[06] cost and coverage

The crawler that costs you most may never read your best page

Per crawler: every request, the bytes you sent, the 90th-percentile answer time and its errors. Beside your sitemap: which listed pages it fetched, which it never has, and what it keeps fetching that you do not list.

pick a crawler
requests
1,184
bytes sent
412 MB
p90 answer time
1,340 ms
404s, 5xx, redirects
38

your sitemap, 40 pages · 21 of 40 fetched · 57 unlisted paths fetched

fetched never fetched /pricing/

Heavy and slow to serve, and it has never fetched 19 of your 40 listed pages, your pricing page among them.

Sample month. A page is slow when it took a second or more, large at 1 MB or more.

AI crawl healthEvery AI crawler, named

[07] your rules

Keep the AI that answers people. Refuse the AI that only trains.

Choose per AI company: allow, block training only, or block. "Block training" keeps the fetchers that answer a person's question and the crawlers that build AI search, so you stay in the answers without feeding a model.

Skald writes these nginx lines for your server. An nginx map is a lookup table in the web server's settings: it reads each request's user agent and answers refuse or let in. The lines take effect once your server admin has connected them. robots.txt is always answered, so a refused crawler can still read your rules.

block AI crawlers
OpenAI
Anthropic
Google
Meta
ByteDance

try "Block training" on OpenAI

nginx map · written by Skald0 crawlers refused
map $http_user_agent $skald_block_agent {
    default 0;
}
map $uri $skald_block {
    default $skald_block_agent;
    /robots.txt 0;
}
robots.txt lines · a suggestion to copy
# nothing refused

Real crawler names from Skald's list. Google's training opt-out, Google-Extended, is a robots.txt token and never a visitor, so it only appears in the robots.txt lines.

See every AI crawler on your site, 30 days free Open the live demo

[08] what it cannot see

Where AI traffic tracking stops, said plainly

Every figure is one Skald can prove from the log. These are the edges, so you never read more into a number than it holds. More in Insights

What it does not do

  • AI clicks are a floor: an app that strips the referrer sends a direct visit. Likely visits are shown apart, never added.
  • Clicks from Google AI Overviews cannot be told apart from other Google clicks. AI Mode can, when the click carries udm=50.
  • Google trains on what Googlebot fetches. Google-Extended is a robots.txt rule, not a crawler you can count.
  • No server or CDN log (Shopify, Webflow, Squarespace: script tag only) means no crawlers or fetches at all.
  • What an assistant said in its answer is not in your log. For that, see AI visibility tracking.

in this section

Go deeper: every part of AI traffic, one page each

AI trafficAI crawler trackingGPTBot, ClaudeBot, PerplexityBot and more, checked against their owners' addresses.See the crawlersAI trafficChatGPT trafficGPTBot, ChatGPT-User and the people chatgpt.com sends, counted apart.See ChatGPT trafficAI trafficPerplexity trafficPerplexityBot, Perplexity-User and the people perplexity.ai sends, apart.See Perplexity trafficAI trafficClaude trafficClaudeBot, Claude-User and the people claude.ai sends, counted apart.See Claude trafficAI trafficAI referral trafficPeople who clicked an AI answer: a floor, with likely visits kept apart.See AI referralsAI trafficGoogle AI Mode trafficClicks marked udm=50 told apart; AI Overviews said plainly to be invisible.See AI Mode trafficAI trafficAI crawl healthSitemap coverage by Google, Bing and each AI crawler, and the pages they miss.See crawl healthAI trafficBlock AI crawlersAllow, block training only, or block, per company, with every refusal shown.See crawler rulesAI trafficllms.txtWho fetched your llms.txt, what else AI asks for, and a suggested file.See llms.txtGuideThe AI traffic guideSee ChatGPT, Claude and Perplexity in your own data, step by step.Read the guideGuideWhat is GEO?Generative engine optimization in plain words, and how to prove it works.Read the guide

keep reading

More on AI traffic tracking

AI visibilityAI visibility trackingWhether assistants name you, read your page, send a visitor and bring a lead.See AI visibilityThe methodHow it worksOne log line, six facts, checked names, a render test, and every figure counted twice.See how it worksPricingSkald pricingPer site, a 30-day free trial with no card, or free to self-host.See the prices

questions

AI traffic tracking FAQ

Straight answers, including what Skald does not do.

Read the docsAsk us a question
How do I track AI traffic on my website?

Read your server's access log. AI crawlers and AI fetchers never run JavaScript, so a script tag never sees them. Skald reads the log your server already writes, names each AI caller, and splits training crawls, live fetches and click-throughs. How server log analytics works.

Is AI traffic counted as people?

No. Crawlers and fetchers are machines and show under AI and bots, in their own units. A person who clicks a link in an AI answer is a person, with the assistant shown as their source.

What does "likely from ChatGPT" mean?

It is an estimate, not a count. A direct visit that lands on a page ChatGPT fetched up to 15 minutes before is marked likely from ChatGPT. Likely visits are shown apart and never added to any figure.

Why do AI clicks look low?

Many AI apps strip the referrer, so the click arrives as a direct visit. Skald counts only clicks that carry the assistant's address, so AI clicks are a floor, not a total. Likely visits fill in part of the gap, kept apart.

Can Skald see Google AI Overviews?

No. A Google click whose address carries udm=50 is Google AI Mode, and Skald shows it as google.com (AI Mode). Clicks from AI Overviews arrive as plain google.com, and Skald never guesses which those were. Google AI Mode traffic.

How do I know a crawler is really GPTBot?

Skald checks each crawler's address against the addresses or DNS its owner publishes. Each one is verified, not verifiable, or spoofed. A crawler using GPTBot's name from the wrong address shows as "claims to be GPTBot", under bots.

$ tail -f access.log | skald

See what AI does with your site. Free for 30 days.

No card. We set up hosted Skald for you within one working day, or you self-host it free. Every line of your log, sorted and added up.

Start your free trial Explore the live demo Run the free site check
Start free trial Live demo