Skip to content
New storyWe built Skald to count visitors. It caught an AI-key thief.
SkaldEvery visit, truly told.
Live demo Start free trial

set-up · cloudflare · any plan

Cloudflare log analytics, cache hits included

Cloudflare log analytics needs a log of every request, and a page served from Cloudflare's cache never reaches your server. The Skald Worker sends Skald a record of each one, on any Cloudflare plan.

Cloudflare log analytics is reading the requests Cloudflare answered for your site, not only the ones it passed on to your server. Skald gets them from the Skald Worker, a small Cloudflare Worker that posts one record per request, or from Cloudflare's own Logpush log.

Start your 30-day free trial Open the live demo

30-day free trial · no card · we set it up within one working day · free to self-host

skald worker · sample recordscache
  1. 09:14:02 203.0.113.24 "GET /pricing/" 200 HIT "GPTBot/1.2"AI · training
  2. 09:14:03 198.51.100.7 "GET /" 200 HIT ref "google.com"people
  3. 09:14:05 198.51.100.140 "GET /docs/" 200 MISS "ChatGPT-User/1.0"AI · live fetch
  4. 09:14:06 203.0.113.66 "GET /blog/" 200 HIT "Googlebot/2.1"search
  5. 09:14:08 192.0.2.91 "GET /.env" 404 MISS "zgrab/0.x"bots · probe

Sample records. Three of these five never reached the origin server: Cloudflare answered them from its cache.

[01] the cache gap

Why your server log misses Cloudflare's cached pages

When Cloudflare has a page in its cache, it answers the visitor itself. The request never reaches your server, so your server's access log never sees it. Server log analytics needs every request, so a Cloudflare site needs Cloudflare's side of the story.

  1. A request comes in

    A person, a search crawler or an AI crawler asks Cloudflare for a page. Cloudflare answers from its cache, or passes it on to your server.

  2. The Worker passes it on unchanged

    The Skald Worker runs in front of your site. It changes nothing, blocks nothing and adds no wait: the response goes out as soon as Cloudflare has it.

  3. Then one record goes to Skald

    After the response has gone out, the Worker posts one small record of the request to your Skald. A failed post is dropped, never retried, and never touches the page.

Cloudflare's own request log, Logpush, can do the same job, but its HTTP requests dataset is not on every plan. The Worker works on any plan, the free one included.

how server log analytics worksYour server's own log: nginxSkald vs Cloudflare Web Analytics

[02] set-up

Set up the Skald Worker for Cloudflare

Skald makes the sending token and fills in the steps. You deploy the Worker, give it two settings and add your site's route.

  1. In Skald

    Settings > Connections > Send this site's visits to Skald, then Cloudflare. The dialog shows the send address and a sending token that can only add records for this one site.

  2. Deploy the Worker

    Press Deploy to Cloudflare on the Worker's page on GitHub. It copies the Worker into a new repository in your GitHub or GitLab account and deploys it. It asks for SKALD_INGEST_URL and SKALD_TOKEN, with no defaults.

  3. Add your site's route

    The button deploys the Worker with no route, so it sees nothing yet. Add example.com/* for each host of your site, in the Worker's Settings > Domains & Routes, or in wrangler.toml.

The Worker on GitHubadd a site to Skald

Or by hand, from your own machine, in a copy of the Worker's folder:

~/skald-edge
# the sending token, as a secret: never in wrangler.toml
$ npx wrangler secret put SKALD_TOKEN
~/skald-edge
$ npx wrangler deploy
wrangler.toml · edit the names first
routes = [
  { pattern = "example.com/*", zone_name = "example.com" },
  { pattern = "www.example.com/*", zone_name = "example.com" },
]

[vars]
SKALD_INGEST_URL = "https://stats.example.com/traffic-ingest/example.com"

Copy wrangler.toml.example over wrangler.toml, set your hosts and your Skald's send address, then run the two commands. Leave Skald's own dashboard host out of the routes if it shares the zone.

[03] never counted twice

The Worker alone, or beside your server's log

A request Cloudflare passed on to your server is in your server's log as well. Skald knows which records to keep, so nothing is counted twice. Flip it to see.

flip it
worker records · samplecache
  1. 09:14:02 "GET /pricing/" 200 HIT "GPTBot/1.2"kept
  2. 09:14:03 "GET /" 200 HIT ref "google.com"kept
  3. 09:14:05 "GET /docs/" 200 MISS "ChatGPT-User/1.0"kept
  4. 09:14:08 "GET /.env" 404 MISS "zgrab/0.x"kept

The Worker is the site's only log

No server log reaches Skald, so every Worker record is kept, hit or miss. The site's log format is edge_json. When you choose Cloudflare for a new site that has received nothing yet, Skald sets this for you and writes it to the audit log.

Keep your tool. Add Skald beside it, 30 days free Open the live demo

[04] what is sent

What a Skald Worker record holds, and what it costs

A record holds what an access log line holds, plus the cache status and Cloudflare's request id. No cookies, no other headers, no body. Skald drops any other field before it writes.

FieldHolds
tswhen the request came in
ipthe visitor's address (CF-Connecting-IP)
method, host, pathwhat was asked for, with the query
status, byteswhat was sent back
referer, uawho sent them, and what they claim to be
cachecf-cache-status: HIT, MISS and the rest
idCF-Ray: a record posted twice is read once

Three settings, all optional

  • SKALD_CACHE_HITS_ONLY: send only cache hits, when Skald reads your server's log too.
  • SKALD_SKIP_EXTENSIONS: file types not to send. Keep your pages' scripts and styles in: a browser fetching them is how Skald tells a person from a scraper.
  • SKALD_SAMPLE: the share of requests sent, 1 by default. Skald does not scale a sample up, so below 1 every figure is that share of the real one.

The cost

Each request through the Worker is a Worker request on your Cloudflare plan's allowance or bill. The post to Skald is a subrequest, which Cloudflare's pricing page says it does not bill. Check that page for your plan.

On Skald's side, each record is one post, under its own limit per site: 1,200 a minute, bursts of 300. A post over the limit is refused and that request is not counted, so the person who runs your Skald should set it above your busiest minute.

[05] the other road

Cloudflare Logpush: reading Cloudflare's own log

If your plan has Logpush's HTTP requests dataset, Skald can read it as a CDN log. It sees every request, cache hits included.

  1. Push the log to a bucket

    A Logpush job for the zone, dataset HTTP requests, to R2 or Amazon S3, one JSON object per line. List the fields Skald reads: Logpush has no "all fields" option.

  2. Append it to one log

    On the server Skald runs on, sync the bucket every 15 minutes and append each new file to one log that logrotate rotates. Point Skald at it with the format cloudflare_json.

The fields, the sync script and the logrotate lines are in CDN-LOGS.md on GitHub. Read Cloudflare's log only, never your server's log as well: a cache miss is in both.

Why we suggest the Worker instead

  • Every Skald figure is counted twice, the second time by an independent audit with its own reader. Today that reader does not read Logpush lines.
  • So on a day read only from Logpush, the two counts cannot agree, and Skald withholds that day rather than publish a figure nobody checked.
  • The Worker's records are read by the audit, so their days are checked and shown.
  • Logpush delivers in batches, so a day's last lines can arrive late. Run the daily report late enough for them.
How the second count works

[06] in your dashboard

What Skald shows for a site behind Cloudflare

Every request Cloudflare answered, sorted into five groups that add up to the whole log: people, AI assistants, search engines, bots and scanners, and your own traffic.

AI crawlers and fetchers do not run scripts, but they pass the Worker like everyone else. Skald names each one and checks it against the addresses its owner publishes. A fake shows as "claims to be GPTBot", under bots.

What it does not do

  • No country in the record. Skald looks the address up in a GeoIP country file when one is set on its server; without one, the country says Not recorded.
  • The record carries no request headers, so a page opened by a click inside a Next.js site is not counted as a page view. Only the first page of each visit shows.
  • A refused post is lost: the Worker never sends it again.
  • It does not block or change any request. It only reports what Cloudflare answered.
  • Skald itself cannot run as a Cloudflare Worker: it needs an always-on process and a disk. It runs on hosted Skald, or on a server you choose.
AI crawler trackingRun Skald on your own serverCloudFront log analytics
Skald crawler detail view with sample data: requests by day, pages, status mix and share from verified addresses
Sample data. One crawler opened: what it read, and whether its addresses check out.

keep reading

More on Cloudflare log analytics

Set-up · VercelVercel log analyticsA log drain posts every request to Skald. Nothing to change in your site's code.Set up VercelDocs · set-upAdd a site to SkaldAddress, proof, the way in, the install and a Test. It saves how far you got.Add a sitePricingSkald pricingPer site, a 30-day free trial with no card, or free to self-host.See the prices

questions

Cloudflare log analytics FAQ

Straight answers, including what Skald does not do.

Read the docsAsk us a question
Does the Skald Worker work on the free Cloudflare plan?

Yes. The Worker works on any Cloudflare plan, the free one included. Each request through it counts as a Worker request on your plan's allowance or bill, so check Cloudflare's own Workers pricing page for your plan's numbers.

Can the Skald Worker slow down or break my site?

No. The Worker passes every request on unchanged and returns the response as soon as Cloudflare has it. It posts the record to Skald only after the response has gone out. A failed post is dropped without a retry, and if the Worker's code fails, Cloudflare sends the request on as if the Worker were not there.

Will a request be counted twice if Skald also reads my server's log?

No. A request Cloudflare passed on to your server is in your server's log too. When Skald reads both for one site, it keeps only the Worker's records with cache status HIT and takes the rest from your server's log.

Why do my Cloudflare visits have no country?

The Worker's record holds what an access log line holds, and no country. Skald then looks up each address in a GeoIP country file, when the person who runs your Skald server has set one. Without a file, the country says Not recorded. It is never guessed.

Should I use the Worker or Cloudflare Logpush?

Use the Worker. It works on any plan, and its records are counted twice, once by an independent audit. Logpush's HTTP requests dataset is not on every plan, and today the audit does not read Logpush lines, so a day read only from Logpush is withheld.

Does the Worker see AI crawlers like GPTBot?

Yes. AI crawlers do not run scripts, but every request through Cloudflare passes the Worker, so their requests are sent like any other. Skald then names each crawler and checks it against the addresses its owner publishes. How AI crawlers are identified.

$ tail -f access.log | skald

See what AI does with your site. Free for 30 days.

No card. We set up hosted Skald for you within one working day, or you self-host it free. Every line of your log, sorted and added up.

Start your free trial Explore the live demo Run the free site check
Start free trial Live demo