AI crawler collector

AI crawlers don't run JavaScript, so no analytics script sees them. Send your server's request log to the collector and the AI visibility page shows which crawlers read which pages, whether each one is verified or spoofed, and how many visitors each assistant sends back.

Every option needs an API key with write access (create one under Account and API keys). If the key reaches more than one site, add ?site=<site id>.

POST https://pathrail.io/api/c?source=server
Authorization: Bearer sk_pr_...
Content-Type: application/json

[{ "ua": "Mozilla/5.0 ... GPTBot/1.2", "path": "/pricing", "ip": "20.171.0.1", "ts": "2026-10-21T09:00:00Z", "status": 200 }]

The body can be a JSON array, one object, or newline-delimited JSON, and may be gzipped. Up to 50,000 lines a request. Lines from ordinary browsers are ignored, so you can send everything.

Vercel

Project settings, Log Drains, add a custom endpoint with the URL below and the header Authorization: Bearer sk_pr_.... Pathrail answers Vercel's verification request itself.

https://pathrail.io/api/c?source=vercel

Cloudflare

A Worker in front of your site, sending only requests that look automated:

export default {
  async fetch(req, env, ctx) {
    const res = await fetch(req);
    const ua = req.headers.get("user-agent") || "";
    if (/bot|crawl|spider|gpt|claude|perplexity|bytespider|ccbot/i.test(ua)) {
      ctx.waitUntil(fetch("https://pathrail.io/api/c?source=cloudflare", {
        method: "POST",
        headers: { authorization: `Bearer ${env.PATHRAIL_KEY}`, "content-type": "application/json" },
        body: JSON.stringify([{ ua, path: new URL(req.url).pathname, ip: req.headers.get("cf-connecting-ip"), ts: new Date().toISOString(), status: res.status }]),
      }));
    }
    return res;
  },
};

Cloudflare Logpush http_requests jobs work too: point an HTTP destination at the same URL.

nginx

Log in JSON and ship the file with any log shipper (Vector, Fluent Bit) to the collector as newline-delimited JSON:

log_format pathrail escape=json '{"ua":"$http_user_agent","path":"$request_uri","ip":"$remote_addr","ts":"$time_iso8601","status":$status}';
access_log /var/log/nginx/pathrail.log pathrail;

# vector.toml
[sources.nginx]
type = "file"
include = ["/var/log/nginx/pathrail.log"]

[transforms.parse]
type = "remap"
inputs = ["nginx"]
source = ". = parse_json!(.message)"

[sinks.pathrail]
type = "http"
inputs = ["parse"]
uri = "https://pathrail.io/api/c?source=nginx"
encoding.codec = "json"
framing.method = "newline_delimited"
request.headers.Authorization = "Bearer sk_pr_..."