Server log analytics · how it works
How server log analytics works: from one log line to five groups
How server log analytics works, in order: your server writes one line per request, Skald reads six facts from it, checks who sent it, and files it in one of five groups. Then an independent audit counts everything again.
How server log analytics works, in one sentence: your web server writes a line for every request it answers, and Skald reads each line, checks the caller's name against the addresses its owner publishes, tests whether a browser really drew the page, files the line in exactly one of five groups, and counts it all a second time by a separate road. It is the method behind server log analytics.
30-day free trial · no card · we set it up within one working day · free to self-host
- read 192.0.2.200 "GET /" 200 "-" "GPTBot/1.2"6 facts
- name the user agent says GPTBot, an AI training crawlerclaim
- check 192.0.2.200 is not in OpenAI's published listspoofed
- file bots and scanners · "claims to be GPTBot"bots
- audit a second road recounts the day from the raw logcounts match
Sample line. A crawler's name is a claim until its owner's published addresses say so.
[01] one line, six facts
What one access log line tells Skald
Your server writes this line whether a script ran or not. This one is a person who clicked a link in a ChatGPT answer. Click any part to see what it is, and what Skald does with it.
access.log · one sample line
click any partThe referrer: who sent them
What it is
The page the visitor came from, when the browser says. Here https://chatgpt.com/.
What Skald does with it
An assistant's address marks a click from an AI answer: a person, with ChatGPT as the source. A new outside source starts a new visit. Many apps strip the referrer, so these clicks are a floor, not a total.
this line → people · source ChatGPT · click-through
[02] trust the names
Crawler names checked against their owners' published addresses
Anyone can write "GPTBot" or "Googlebot" in a request. The crawlers that matter publish how to check them, and Skald checks every one.
- Address lists: Google, Bing, Apple, OpenAI, Perplexity, Anthropic and DuckDuckGo publish the addresses they crawl from.
- Reverse DNS: the address's name ends in the owner's domain and resolves back to the same address (Google, Bing, Apple, Yandex).
- The owner's network: Meta's own announced address blocks.
The lists are fetched once a day, before the daily count, and the last good copy is kept if a fetch fails. Nothing touches the network while counting, so both counting roads read the same answers. A list that cannot be reached never moves a fetch: the claim stays, as "not verifiable".
- 09:14:02 203.0.113.24 "GPTBot/1.2" · in OpenAI's listverified
- 09:14:05 203.0.113.66 "Googlebot/2.1" · DNS matchesverified
- 09:14:09 203.0.113.90 "NewAgent/0.3" · no list publishednot verifiable
- 09:14:12 192.0.2.200 "GPTBot/1.2" · not in OpenAI's listclaims to be GPTBot
Sample data. A spoofed crawler is filed under bots and scanners, never under AI assistants.
[03] who made the request
The render test and the too-fast rule: person or bot?
Rules run in a fixed order, and the first that matches files the line. A browser's name is the weakest claim, so it is tested last.
The render test
A browser that opens a page fetches its own scripts and styles. A scraper takes the HTML and leaves. So a browser-like visit counts as people only with proof that a browser drew the page: the page's own files (set per site in Settings > Sites > Page assets), an event from the page's own script, or the same browser having drawn a page earlier.
Too fast to be a person
A visit that opens 6 or more pages, all within 2 seconds of its first, is filed as a bot. A fast reader clicks through 3 pages in 2 seconds and stays a person.
One probe marks the visitor
A client that asks for a secret or config file, such as /.env, is a bot for the whole day, so the pages it read between probes never count as a visit.
- 09:14:01 198.51.100.2 "GET /healthz" "curl/8.5.0"ours · your address list
- 09:14:02 198.51.100.140 "GET /docs/" "ChatGPT-User/1.0"AI · live fetch
- 09:14:03 203.0.113.24 "GET /pricing/" "GPTBot/1.2"AI · training
- 09:14:04 203.0.113.80 "GET /blog/" "Slackbot-LinkExpanding"link preview: Slack
- 09:14:05 203.0.113.66 "GET /blog/" "Googlebot/2.1"search · verified
- 09:14:06 192.0.2.91 "GET /wp-login.php" "python-requests/2.31"probe
- 09:14:07 192.0.2.55 "GET /pricing/" Chrome/130, no page filesnever rendered
- 09:14:08 192.0.2.56 7 pages in 1 second, Chrome/130too fast to be a person
- 09:14:09 198.51.100.7 "GET /" Chrome/130, then its scriptspeople
Sample data. Each line goes to one group only. A visit counts whole or not at all, so the groups still add up to the log.
[04] pages and visits
What counts as a page view, and what counts as a visit
Who asked is one question. What they asked for is another, and Skald answers it on its own, with plain rules.
A page view is a page
A GET or HEAD answered 2xx or 304, that is not a file. Scripts, styles, images, fonts, robots.txt and sitemaps are files. Redirects are not page views: the browser lands on a page that is logged on its own.
API calls are not pages
Everything under
/api/, and beacon paths such as/collect, is evidence of a visit, never a page of its own or a step between pages.One click, one page view
On a Next.js site whose log records the prefetch header, a click inside the site is a page view and a prefetch never is. A page and a click to the same path within 2 seconds count once.
A visit is an address and a browser
With no gap over 30 minutes. A page arriving from a new outside source (an assistant, a search, a campaign) starts a new visit, so the source that brought someone back is shown.
Plain words for gaps
A visit with no referrer and no campaign tag is "Direct". A value the log did not have is "Not recorded". Neither is ever called unknown, or guessed.
People, as an upper bound
An office behind one address is one visit; a phone moving from wifi to mobile data is two. So visits are at most this many people, and requests are never people.
[06] honest limits
What a server log cannot know
These limits sit next to the figures in Skald, not in a footnote. They are why people is an upper bound and AI clicks are a floor.
What it does not do
- It cannot prove a person. A user agent is a claim, and a scraper that drives a real browser and loads the page's files looks like a reader.
- It does not count a person who blocks scripts and styles, or one whose browser restores 6 or more tabs at once and reads nothing after.
- It cannot see a click from a Google AI Overview: it arrives as plain google.com. Google AI Mode is told apart only when the click carries
udm=50. - It cannot count AI clicks whose referrer was stripped. Those clicks are a floor; "likely from ChatGPT" visits are estimates, shown apart and never added.
- It does not know a new AI assistant until its name is added to the lists. Until then it falls to bots, or to people.
- It sees nothing without a log. On Shopify, Webflow, Squarespace or the script tag alone, AI crawlers, search crawlers and bots are not seen at all.
keep reading
More on how server log analytics works
questions
How server log analytics works: FAQ
Straight answers, including what Skald does not do.
What does one access log line contain?
A standard access log line holds six facts: the address the request came from, the time, the method and path asked for, the status the server answered, the referrer (the page the visitor came from, when the browser says), and the user agent (the name the caller gives itself). Skald reads all six, and the host name when the log records it.
How does Skald know a crawler is really GPTBot or Googlebot?
It checks the address against what the crawler's owner publishes: address lists (Google, Bing, Apple, OpenAI, Perplexity, Anthropic, DuckDuckGo), reverse DNS (Google, Bing, Apple, Yandex), or the owner's network (Meta). Each hit is verified, not verifiable, or spoofed. A spoofed one is filed under bots as "claims to be GPTBot". The lists are fetched once a day, never while counting.
How does Skald tell a person from a bot with a browser's name?
With a render test. A browser that opens a page goes on to fetch the page's own scripts and styles; a scraper takes the HTML and leaves. A browser-like visit counts as people only with evidence that a browser drew the page: its page files, an event from the page's own script, or the same browser having drawn a page earlier. A visit that opens 6 or more pages within 2 seconds is filed as a bot.
What does "counted twice" mean?
A separate audit recounts every figure from the raw log with its own parser and its own code. The two roads share only the lists of names. If the counts for a day disagree, Skald withholds that day and lists it, rather than show a figure that may be wrong.
Why is a figure blank instead of zero?
Blank means not recorded: Skald did not have the data, for example a breakdown from before it began collecting it, or a country the log did not carry. Zero means Skald counted and found nothing. A figure that was not recorded is never shown as 0, in the dashboard, the API or the MCP server.
Is a visit the same as a person?
No. A visit is one address and one browser with no gap over 30 minutes, so an office behind one address is one visit, and one person moving from wifi to mobile data is two. That is why Skald says visits are at most this many people, and never counts requests as people.
$ tail -f access.log | skald
See what AI does with your site. Free for 30 days.
No card. We set up hosted Skald for you within one working day, or you self-host it free. Every line of your log, sorted and added up.