Why your analytics shows almost no AI traffic
Open Google Analytics, filter for ChatGPT, and you may find only a handful of visits. That number is real, but it is the smallest part of what ChatGPT did on your site. The rest never reached your analytics, because of how analytics works.
An analytics script runs inside a visitor's browser. If the browser runs the script, the visit counts. AI crawlers and AI fetchers are not browsers. They ask your server for the page, read the HTML, and leave. No script runs, so nothing is counted.
AI touches your site for three different reasons, and only one of them is visible to a script:
| Reason | Who | Is a person waiting? | Does a script see it? |
|---|---|---|---|
| Training crawl | GPTBot, ClaudeBot, and the index crawlers OAI-SearchBot, Claude-SearchBot, PerplexityBot | No | No |
| Live fetch | ChatGPT-User, Claude-User, Perplexity-User, Gemini-User | Yes, right now | No |
| Click-through | A person clicking a link in an AI answer | It is the person | Yes, if the referrer survives |
Keep these three apart. A crawl is not a reader, and a fetch is not a visit. Added together, they give you a number that means nothing. The rest of this guide shows how to see each one, starting with the record that holds all three. If you want the bigger picture first, read what is GEO.
Step 1. Find your server's access log
Your web server writes a line for every request it answers. That is the access log, and it is the only record that saw the crawlers, the fetches and the clicks. On most servers it is already there:
- nginx: usually
/var/log/nginx/access.log - Apache: usually
/var/log/apache2/access.logor/var/log/httpd/access_log - Caddy: a JSON log, where your Caddyfile says
No server of your own? If your site sits behind Cloudflare, its request log (Logpush, on plans that offer it) or a small Worker can send the requests instead. On Vercel, a log drain does the same, on Vercel Pro. Hosts with no log at all, such as Shopify, Webflow or Squarespace, can show you the people from AI answers through a script tag, but never the crawlers or fetchers.
Step 2. Look for AI names in the log
AI crawlers name themselves in the user agent, the last quoted field of each line. A first look takes one command:
grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" /var/log/nginx/access.log | sort | uniq -c | sort -rnYou will see a count per name. Two warnings before you trust it. First, a name in a user agent is a claim, and anyone can make it: some scanners call themselves GPTBot. Second, the list above is only the best-known names. Meta, Amazon, ByteDance, Common Crawl, Apple, Mistral and others send their own. Skald knows 39 AI crawler names and files each by its reason. See AI crawler tracking for the full set.
Step 3. Check that each crawler is real
The big AI companies publish the addresses their crawlers use. OpenAI publishes lists for GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic and Perplexity publish theirs; Google and Bing can also be checked by reverse DNS. A request that says GPTBot from an address that is not on OpenAI's list is a fake.
Checking this by hand is slow. Skald fetches the lists once a day and checks every request. Each crawler is verified, not verifiable (nothing to check against), or a fake, filed under bots as "claims to be GPTBot". Fakes never count as AI traffic. In the sample below, the fake GPTBot asks for /.env, a file of secrets no crawler needs.
Step 4. Split each assistant by reason
Now take one assistant at a time. Each has its own set of names, and the split matters for each.
- ChatGPT. GPTBot trains, OAI-SearchBot builds the search index, ChatGPT-User fetches for a person. People arrive from
chatgpt.comor the Android app. See ChatGPT traffic. - Claude. ClaudeBot trains, Claude-SearchBot indexes, Claude-User fetches for a person, and the older Claude-Web still shows up. People arrive from
claude.ai. A developer's own Claude Code also says Claude-User, from their own address. See Claude traffic. - Perplexity. PerplexityBot indexes and Perplexity-User fetches for a person. There is no training crawler. People arrive from
perplexity.ai. See Perplexity traffic.
The live fetch is the number to watch. It means someone asked that assistant a question and it read your page to answer, whether or not they ever clicked.
Step 5. Count the people AI sends you
People who click a link in an AI answer are visitors like any other. You know them by the referrer, the address of the page the click came from. In the common log format the referrer is the second quoted field ($4 once the line is split on quote marks), so this counts them:
awk -F'"' '$4 ~ /chatgpt\.com|perplexity\.ai|claude\.ai|gemini\.google\.com|copilot\.microsoft\.com/' /var/log/nginx/access.log | wc -lThat counts requests, not visits: a person who clicks two links in the same answer counts twice. It also misses a lot. Referrers are stripped by privacy settings, by apps that open links in their own browser, and by a site's Referrer-Policy. Those clicks arrive as direct visits. Treat any AI click count as a floor.
Skald counts visits, not requests, and also reads the Android apps' names and links tagged utm_source=chatgpt, perplexity, claude and others. For the clicks that lost their referrer, it makes an estimate: a direct visit that lands on a page an assistant's fetcher read in the same minute or up to 15 minutes before is shown as likely from that assistant. Likely visits are always marked, and never added to the count. See AI referral traffic.
$ grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log | sort | uniq -c | sort -rn$ awk -F'"' '$4 ~ /chatgpt\.com|perplexity\.ai|claude\.ai|gemini\.google\.com|copilot\.microsoft\.com/' access.log | wc -l 2 GPTBot
1 ChatGPT-Userwhat that count missed
- A line says GPTBot from 198.51.100.77, which is not on OpenAI's list in this sample. grep counts it; Skald files it under bots as "claims to be GPTBot".
- Bytespider is missing from the eight names in the command, so grep never sees it. Skald knows 39.
- 6 lines are people's browsers. Step 2 says nothing about them; step 5 does.
Let Skald count it for you, 30 days free Open the live demo
Step 6. See what the AI read before the click
Put steps 4 and 5 side by side and something useful appears. When ChatGPT sends a visitor to /pricing/, the log can show ChatGPT-User fetching /pricing/ a minute earlier, and maybe /reviews/ and /compare/ too. Those are the pages the answer was built from.
Skald keeps that list on every AI visit: the pages that assistant fetched in the 15 minutes before, nearest first, at most 10. Per assistant, it also shows how many visits came after a fetch, and how many landed on the page it had just fetched. It is context, not a count of people, because a fetch can be for someone who never clicked.
Step 7. Check what crawlers never reach
A page no crawler fetches is a page no assistant can quote. Compare your sitemap (the file that lists your pages for crawlers) with the log: for each crawler, which listed pages did it fetch at least once in the last month, and which never? Then look the other way: which pages did crawlers fetch that are missing from your sitemap, and which answered with a 404, a server error, a redirect, or slowly?
Skald does this per crawler, for Google, Bing and every AI crawler and fetcher, and reads your sitemap once a day. See AI crawl health.
Step 8. See what AI asks for, including llms.txt
AI fetchers ask for more than pages. In the log you may see requests for /llms.txt (a short file that points AI assistants to a site's key pages), /llms-full.txt, Markdown copies ending in .md, robots.txt and sitemaps. Count them by what the address names, not by whether your server had the file: a request for an llms.txt you never published still tells you an assistant wanted one.
If you decide to publish one, start from the pages people and assistants actually read. Skald suggests an llms.txt from the last 90 days of your own figures, for you to check and publish. See llms.txt.
Step 9. Decide which crawlers to let in
Once you can see each crawler, you can choose. To stay in AI answers, let the live fetchers in. Some keep the training crawlers out, so their pages stay out of model training. Because the crawlers are separate, both are possible at once.
robots.txt asks; a server rule enforces. Skald's crawler rules give each AI company three choices, allow, block training only, or block, applied through nginx by the name each crawler sends. Some names, such as Google-Extended, are robots.txt only, and get a suggested line instead. See block AI crawlers.
Step 10. Know what Google lets you see
Google is the hard case. Googlebot is a search crawler, and Google-Extended, Google's AI switch, is a robots.txt line that never appears in a log. So there is no honest figure for "Google's AI crawl".
On the click side, Google's AI Mode marks some of its links with udm=50. When a click carries it, Skald labels the visit google.com (AI Mode). A click from an AI Overview carries nothing at all and looks exactly like an ordinary Google click, so nobody can count it. Gemini is different again: it is an assistant, and its clicks come from gemini.google.com. See Google AI Mode traffic.
Step 11. Follow AI visitors to leads
The last question is the one that pays: did anyone AI sent you become a customer? That needs a second record, from the tool where leads arrive: your form, your booking tool, your payments. A signed webhook (a message the tool sends Skald when a lead arrives) from Calendly, Cal.com, HubSpot, Stripe or your own backend lets Skald join a lead to the visits before it, so a lead whose first visit came from ChatGPT shows ChatGPT as its first touch.
Put together, it is a chain: an answer named your page, the assistant read it, a person clicked, that person booked a call. Each link is proven by a different record, and none is added to another. See AI visibility tracking for the full chain.
Doing all of this without the shell
Everything above can be done by hand with a log and a few commands, and it is worth doing once to see your own numbers. Doing it every day, with names checked, visits built from requests, and every figure counted twice, is what Skald is for.
Set-up is one command on an nginx or Apache server, with or without Docker. It finds your access log and starts sending lines to your Skald. Without a server, use the Cloudflare Worker or a Vercel log drain. Skald is free to self-host, and the hosted version has a 30-day free trial with no card; we set it up within one working day. Start from AI traffic tracking, or open the live demo to see every screen in this guide with sample data. For shorter reads on one question at a time, see AI traffic insights.