How to see AI bots in server logs: GPTBot, ClaudeBot and ChatGPT-User, with grep

Why AI bots only show in the server log

Open your analytics and look for GPTBot. You will not find it. That is not a setting you missed. Analytics tools count browsers that run their script, and AI bots do not run scripts. They ask your server for the HTML, read it, and leave.

Your web server writes a line for every request it answers, whoever asked. That file is the access log, and it is the one place where every AI crawler and every AI fetcher leaves a trace. If you have shell access to the server, you can see them in a few minutes with tools that are already installed.

This note walks through it with grep, awk and sort, on nginx or Apache. At the end it says what a by-hand look misses, and where a tool helps.

Step 1: find the log and read one line

On most servers the log is already there:

• nginx: /var/log/nginx/access.log, or one file per site if your config says so
• Apache: /var/log/apache2/access.log on Debian and Ubuntu, /var/log/httpd/access_log on Red Hat and its cousins
• Caddy: a JSON log, wherever your Caddyfile points it

Older days are rotated into access.log.1, access.log.2.gz and so on. Use zgrep in place of grep to search the compressed ones too.

Both nginx and Apache write the combined format by default. One line looks like this (sample data):

203.0.113.24 - - [22/Oct/2026:09:14:01 +0000] "GET /pricing/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"

Read it left to right: the address that asked, the time, the request, the status your server answered, the bytes it sent, the referrer (the page the visitor came from, here - for none), and the user agent. The user agent is where a bot says its name. Here it says GPTBot.

Step 2: one grep for the AI names

Start with the best-known names and count how many lines each one has:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|meta-externalagent|CCBot|Bytespider|Amazonbot" \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

On a small business site, a day might print something like this (sample data):

    412 GPTBot
    233 ClaudeBot
    118 meta-externalagent
     96 PerplexityBot
     41 OAI-SearchBot
     27 ChatGPT-User
     12 Claude-User
      5 Perplexity-User

Those are requests, not visits and not people. One crawler reading 400 pages is 400 lines. Keep that in mind whenever you put these numbers next to visitor figures.

The list above is a start, not the whole set. Skald's own list holds about 40 AI names, from Apple, Amazon, ByteDance, Common Crawl, Cohere, Mistral, DuckDuckGo and others. One name you will never find is Google-Extended. It is a robots.txt token, not a user agent: Googlebot does the fetching, and Google-Extended only tells Google whether it may use what it fetched for its AI models.

Step 3: split them by why they came

An AI bot comes for one of two reasons, and the difference matters.

Training and index crawlers read your pages ahead of time. GPTBot, ClaudeBot, CCBot and Bytespider collect pages to train models. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the index an assistant searches later. In both cases nobody is waiting.

Live fetchers read a page because a person asked an assistant a question just now. ChatGPT-User, Claude-User, Perplexity-User and MistralAI-User are the common ones. A live fetch means someone's answer was being built from your page, whether or not they ever clicked.

Count the two groups apart:

# training and index crawlers
grep -cE "GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent|OAI-SearchBot|Claude-SearchBot|PerplexityBot" /var/log/nginx/access.log

# live fetches for a person's question
grep -cE "ChatGPT-User|Claude-User|Perplexity-User|MistralAI-User" /var/log/nginx/access.log

If your second number is small next to the first, that is normal. Crawlers read everything; fetchers read only what someone asked about. The live fetches are the ones to watch. For the OpenAI names in detail, read GPTBot vs ChatGPT-User.

Step 4: see which pages each bot read

In the combined format, the requested path is the seventh field when you split a line on spaces, and the status is the ninth. So this lists the pages GPTBot read most:

grep GPTBot /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Swap GPTBot for ChatGPT-User and you get the pages ChatGPT fetched while answering people. That list is often short and telling: a pricing page, a how-to, a comparison. Those are the pages an assistant thought answered someone's question.

Then look at what your server told each bot:

grep ClaudeBot /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -rn

A crawler that gets many 404 or 301 answers is spending its visit on pages that are gone or moved. A run of 5xx answers means your server failed it. None of that shows anywhere else. You can also look for what bots asked for beyond pages: grep -E "llms(-full)?\.txt|\.md " finds requests for llms.txt and Markdown copies, even when you never published one.

Step 5: find the people AI sent you

A person who clicks a link in an AI answer is not a bot. Their browser has an ordinary user agent, and what gives them away is the referrer. In the combined format the referrer is the fourth field when you split on double quotes:

awk -F'"' '$4 ~ /chatgpt\.com|perplexity\.ai|claude\.ai|gemini\.google\.com|copilot\.microsoft\.com/ {print $4}' \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

Two warnings. This counts requests, so one person reading five pages counts five, and every image and stylesheet their browser loads may count too. And it misses people: many apps strip the referrer, so a click from an AI answer often arrives as a direct visit. Any AI click count you make this way is a floor, not a total.

Step 6: check that GPTBot is really GPTBot

Here is the catch with everything above. A user agent is a claim, and anyone can write GPTBot into one. Scanners do it all the time, hoping a site lets "AI crawlers" through. A sample of what that looks like:

192.0.2.200 - - [22/Oct/2026:03:12:44 +0000] "GET /.env HTTP/1.1" 404 162 "-" "GPTBot/1.2"
192.0.2.200 - - [22/Oct/2026:03:12:45 +0000] "GET /wp-login.php HTTP/1.1" 404 162 "-" "GPTBot/1.2"

A real crawler does not ask for /.env. To be sure, check the address against what the owner publishes. OpenAI publishes the addresses of GPTBot, OAI-SearchBot and ChatGPT-User (for example openai.com/gptbot.json), and Anthropic and Perplexity publish theirs. This checks one address against OpenAI's GPTBot list:

curl -s https://openai.com/gptbot.json -o gptbot.json
python3 - 192.0.2.200 <<'EOF'
import ipaddress, re, sys
ip = ipaddress.ip_address(sys.argv[1])
nets = re.findall(r'"([0-9a-fA-F.:]+/\d+)"', open("gptbot.json").read())
print("on the list" if any(ip in ipaddress.ip_network(n) for n in nets) else "not on the list")
EOF

Google and Bing are checked by DNS instead. Run host on the address: a real Googlebot's name ends in googlebot.com or google.com, and running host on that name gives the same address back. A name that does not round-trip is not Google.

One case looks odd but is fine. When a developer's Claude Code fetches a page for them, it says Claude-User, from the developer's own address. No list can hold every developer's laptop, so that fetch cannot be verified. That makes it unverifiable, not fake.

What a by-hand look misses

Everything above is worth doing once. It shows you your own numbers in ten minutes. Doing it every day is where it gets hard:

• Requests are not visits. Turning lines into visits means grouping by browser and time, dropping assets, and deciding who is a person. A browser that loads a page but none of its files is probably not one.
• Names change. New AI agents appear every few months. A grep list is out of date the day you write it.
• Address lists change. They need fetching and checking every day, and a list that cannot be fetched should not turn a real crawler into a fake.
• Some people are hidden. A click from an AI answer that lost its referrer looks like any direct visit.
• Logs rotate. Most servers keep a week or two. After that, the history is gone.

What Skald adds on top of grep

Skald reads the same log, every two minutes. It sorts every line into one of five groups that add up to the whole log: people, AI assistants, search engines, bots and scanners, and your own traffic. AI traffic is split by reason: training crawl, live fetch, click-through. You can read how that works in how server log analytics works.

Each crawler is checked once a day against its owner's published list or DNS, and filed as verified, not verifiable, or spoofed. The fake GPTBot above shows as "claims to be GPTBot", under bots, and never as AI. See AI crawler tracking for the full set of names.

For the clicks that lost their referrer, Skald makes an estimate and keeps it apart: a direct visit that lands on a page an assistant fetched in the minutes before is marked likely from that assistant, never added to the count. And on each visit from an AI answer, it lists the pages that assistant fetched in the 15 minutes before. See AI referral traffic.

Every figure is counted twice, by a separate audit with its own parser. A day the two counts disagree on is withheld, not guessed. Set-up is one command on an nginx or Apache server; without a server, a Cloudflare Worker or a Vercel log drain (on Vercel Pro) sends the requests instead. To see every screen in this note with sample data, open the live demo. To see your own log, start the 30-day free trial; we set it up within one working day. For every step from the first grep to the first lead, read the AI traffic guide.