A/B tests

Most A/B tools pick a winner on clicks or signups. Pathrail picks it on money: the revenue per visitor each variant brought in, from the payments Stripe, Lemon Squeezy or Polar report. Conversion to paying and an optional goal show alongside.

1. Start an experiment

Open A/B tests in the site's dashboard. Give the experiment a key like pricing-headline, two to four variants with weights (the first is the control), and, if you like, a path so only visits under it count.

2. Show each visitor their variant

Ask the script once the page has loaded. Pass only the key: the script picks up the variants and weights you saved in the dashboard.

pathrail.variant("pricing-headline", function (v) {
  if (v === "b") document.querySelector("h1").textContent = "Ship analytics that pay for themselves";
});

// the same thing through the queue function
pathrail("variant", "pricing-headline", function (v) { /* ... */ });

The function runs as soon as the script knows the experiment: at once for a returning visitor, whose browser keeps the list of your tests, and otherwise when the first page view is answered, usually well under a second. It gets null for a key Pathrail doesn't know, or when no answer comes within 3 seconds (an ad blocker, say), so keep your current version as the default. A stopped experiment gives its control and records nothing, like the server API.

A visitor always gets the same variant, on every visit and every page, because the choice comes from their visitor id and the key. The first call on a page records that the visitor saw the experiment. Name variants starting with a letter, so the order you write them is kept.

To get the answer straight away, without the function, pass the same variants and weights as in the dashboard:

const v = pathrail.variant("pricing-headline", { control: 50, b: 50 });
pathrail("variant", "pricing-headline", ["control", "b"]); // a list splits evenly

Or with no JavaScript: mark one element per variant. The script shows the visitor's variant and hides the rest. Add hidden to all but the control so nothing flickers.

<h1 data-pathrail-exp="pricing-headline" data-pathrail-variant="control">Simple analytics</h1>
<h1 data-pathrail-exp="pricing-headline" data-pathrail-variant="b" hidden>Analytics that pay for themselves</h1>

The elements use the saved weights too; until the script knows them the control shows, so a first visit may switch once. To split without waiting, or for a test you haven't saved, add data-pathrail-weight="70" to an element of each variant. In cookieless mode there is no lasting visitor id, so every visitor sees the control and nothing is recorded.

3. Or split on your server

With a write key from your Account page, ask for the variant. Pass the browser's window.pathrail.visitorId as visitor_id and you get the same answer the script would give. Without it, pass user_id or email. The exposure is recorded unless you send "record": false; a stopped experiment always answers with the control.

curl https://pathrail.io/api/v1/experiments/pricing-headline/assign \
  -H "Authorization: Bearer sk_pr_..." -H "content-type: application/json" \
  -d '{"visitor_id": "k3j4h5g6f7d8s9a0"}'
# {"experiment": "pricing-headline", "variant": "b", "running": true}

GET /api/v1/experiments lists the experiments with their variants and weights, and GET /api/v1/stats/experiments returns every experiment with its results.

How results are read

• Visitors: everyone who saw the experiment while it ran, counted in the first variant they saw.
• Revenue: each visitor's first payment after they first saw it, up to the day it stopped. Refunds and later renewals are left out, so one big renewal can't swing the result.
• Revenue per visitor: that revenue over all visitors in the variant, paying or not. This decides the winner.
• Chance better: how likely each variant beats the control, from a Welch test on revenue per visitor and a two-proportion test on conversion.

The verdict waits for at least 100 visitors in each variant and 10 paying customers in all. A variant 95% or more likely to beat the control on revenue per visitor is called the winner. Below that the page says it is too close to call, so let it run. Decide on a sample size up front and don't stop the first day a variant looks good: early leads often fade.