set-up · fastly · json logging
Fastly log analytics, from a JSON endpoint
Fastly log analytics needs Fastly's own log, because a page served from Fastly's cache never reaches your server. Fastly has no fixed format, so you paste one line that Skald reads. Read the limits first: today those days are withheld.
Fastly log analytics is counting the requests Fastly answered for your site, cache hits included, from a logging endpoint that writes one JSON object per request. Skald reads it with the log format fastly_json.
30-day free trial · no card · we set it up within one working day · free to self-host
- 09:14:02 {"client_ip":"203.0.113.24","url":"/pricing/","response_status":200,"request_user_agent":"GPTBot/1.2"}AI · training
- 09:14:04 {"client_ip":"198.51.100.23","url":"/docs/","request_referer":"https://chatgpt.com/"}people · from ChatGPT
- 09:14:06 {"client_ip":"203.0.113.66","url":"/blog/","request_user_agent":"Googlebot/2.1"}search
- 09:14:09 {"client_ip":"192.0.2.91","url":"/wp-login.php","response_status":404}bots · probe
Sample lines, with most fields left out. Fastly writes one JSON object per request.
[01] read the cdn's log
Why Fastly's log, not your origin server's
A page in Fastly's cache never reaches your server, so your server's access log never sees that request. Fastly's own log sees every one, cache hits included.
Server log analytics then works as for any log: every line sorted into people, AI assistants, search engines, bots and scanners, and your own traffic. Read Fastly's log only, never your server's as well: a cache miss is in both, and would be counted twice.
Read this first: what it does not do yet
- Days are withheld today. Every Skald figure is counted twice, the second time by an independent audit with its own reader. That reader does not read Fastly lines yet, so the two counts cannot agree, and the daily run withholds those days rather than publish figures nobody checked.
- The one command on the server behind Fastly works, and its days are checked, but it misses every page Fastly answers from its cache.
- Fastly writes a file each period, an hour by default, so the day's last lines arrive late. Run the daily report late enough for them.
- A country only when you log it, as a two-letter code. Skald never guesses a code from a country's name.
[02] the log format
The Fastly JSON log format Skald reads
Fastly has no fixed log format: you write it in the logging endpoint. Leave the endpoint's log line format at Blank, so each line is the JSON object alone, and paste this into its Log format field, as one line.
{"timestamp":"%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V","client_ip":"%{req.http.Fastly-Client-IP}V","geo_country":"%{client.geo.country_code}V","host":"%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V","url":"%{json.escape(req.url)}V","request_method":"%{json.escape(req.method)}V","request_referer":"%{json.escape(req.http.referer)}V","request_user_agent":"%{json.escape(req.http.User-Agent)}V","response_state":"%{json.escape(fastly_info.state)}V","response_status":%{resp.status}V,"response_body_size":%{resp.body_bytes_written}V,"time_elapsed":%{time.elapsed.usec}V}Fastly's own example
This is the JSON format from Fastly's S3 endpoint guide, with two changes.
A country code
geo_countrylogsclient.geo.country_code, two letters, instead of the country's name.The time it took
time_elapsedis added, in microseconds. Logging from your own VCL? Emit the same object, one per line.
[03] the endpoint
Amazon S3 or SFTP: where Fastly writes the log
Skald reads one log and its rotations. Fastly can put its files in a bucket for Skald's server to copy, or write them straight to that server.
A bucket, then a copy every 15 minutes
Create an Amazon S3 logging endpoint with the format above. On Skald's server, a short script run from cron every 15 minutes syncs the bucket and appends each new file, in name order, to one log. logrotate rotates that log daily.
Straight to Skald's server
Create an SFTP logging endpoint that writes to the server Skald runs on. There is no bucket, so the script drops its aws s3 sync line: point the endpoint's path at the folder the script appends from. logrotate rotates the log daily.
logs:
dir: /var/log/cdn
base: access.log
format: auto # or fastly_json
| Field | Also accepted |
|---|---|
timestamp | time; ISO 8601, Unix seconds, ms, µs or ns |
url | request_url, or path plus query |
request_method | method |
response_status | status, resp_status |
request_referer | referer |
request_user_agent | user_agent |
response_body_size | response_bytes, bytes |
geo_country | country_code, two letters |
time_elapsed (µs) | elapsed_ms, time_elapsed_ms |
[04] already logging?
A Fastly JSON format you already have may work
Skald accepts the common other names for each field. Two are required: a timestamp and a URL. A line without them is counted as unreadable, never guessed.
Skald tells a Fastly line from a Caddy one, which has a request object, and from a Cloudflare one, which has ClientRequestURI, by its keys. With the format auto, a log that mixes them with nginx lines still reads, for the day you move a site behind a CDN.
[05] what you get
What Skald can show from a Fastly log
Once a day's figures pass both counts, the same dashboard as any site: AI crawlers checked against their owners' addresses, live fetches, people and where they came from, scanners and your own traffic, apart.
Until the audit reads Fastly lines, those days stay withheld. If you need checked figures now, read the server behind Fastly with the one command, and know its figures leave out the cached pages.
keep reading
More on Fastly log analytics
questions
Fastly log analytics FAQ
Straight answers, including what Skald does not do.
Can Skald read Fastly logs?
Yes. Skald reads a Fastly logging endpoint that writes one JSON object per request, with the log format fastly_json. It also accepts the common other names for the same fields, so a JSON format you already have may work as it is.
Why are my Fastly days withheld?
Every Skald figure is counted twice, the second time by an independent audit with its own reader. Today that reader does not read Fastly lines, so the two counts cannot agree, and the daily run withholds those days rather than publish figures nobody checked.
Which Fastly log format does Skald need?
Fastly's own example JSON format with two changes: geo_country logs the two-letter country code instead of the country name, and time_elapsed is added, in microseconds. Paste the line from this page into the endpoint's Log format field, with the line format left at Blank.
Should I use S3 or SFTP for the Fastly endpoint?
Either works. With S3, a short script on Skald's server copies the bucket and appends new files to one log. With SFTP, the endpoint writes straight to that server's folder, so no bucket and no copy step are needed.
Can I read my origin server's log instead?
You can, with the one command on the server behind Fastly, but pages Fastly answers from its cache never reach that log, so its figures are too low. Never read both for one site: a cache miss is in both and would be counted twice.
$ tail -f access.log | skald
See what AI does with your site. Free for 30 days.
No card. We set up hosted Skald for you within one working day, or you self-host it free. Every line of your log, sorted and added up.