Skip to main content
Academy · Module 8 of 9

Measuring and testing honestly

Define your metrics once and stick to them, compare against your own baseline, and understand why most email A/B tests are too small to tell you anything.

12 min read · Course overview

Objectives

In this module

  • Define the core outreach and deliverability metrics.
  • Explain why published reply-rate benchmarks disagree.
  • Calculate the sample size an A/B test needs.
  • Avoid the “peeking” mistake.
  • Set up a weekly deliverability check.
8.1

Define the metrics once

Write these definitions down and use them every time, or you'll compare numbers that don't mean the same thing:

  • Delivered: sent minus bounced.
  • Bounce rate: bounces ÷ sent. Keep it under 2% [1,2].
  • Reply rate: replies ÷ delivered. Decide whether out-of-office replies count; usually they shouldn't.
  • Positive reply rate: interested replies ÷ delivered. This needs the reply triage from Module 6.
  • Meeting rate: meetings booked ÷ contacts.
  • Spam complaint rate: reports ÷ delivered. Under 0.1% target, never 0.3% [3].
  • Unsubscribe rate: unsubscribes ÷ delivered.
  • Open rate: don't use it (Module 1).
8.2

Why benchmarks disagree

Published reply-rate benchmarks range from 0.74% (Smartlead's median, 2026) [2] through 1.5% (Woodpecker's median, Q4 2025) [4] to 3.43% (Instantly's average, 2025) [1] and higher.

They aren't measuring the same thing. Some report means, which a few small warm campaigns pull up, and some report medians. Some divide by emails sent including follow-ups, others by contacts. Some count auto-replies. Each platform's customers are a different mix of agencies, small businesses and enterprises. The only benchmark that works is your own baseline under your own definitions.

8.3

Most A/B tests are too small

Evan Miller's rule of thumb for the sample size each variant needs is n ≈ 16 × p(1−p) ÷ δ², where p is your baseline rate and δ the absolute difference you want to detect [5]. Plug in typical cold-email numbers and the problem is obvious.

Most campaigns never reach those sizes. “Variant B got 6 replies, A got 4” is noise. Test big differences (a different offer, a different ask, a different segment or angle), not word swaps. Measure replies or positive replies, never opens.

Worked example
Baseline reply rate p = 2% (0.02). You want to detect a lift to 3% (δ = 0.01).

n ≈ 16 × 0.02 × 0.98 ÷ 0.01²
  ≈ 16 × 0.0196 ÷ 0.0001
  ≈ 3,136 contacts per variant

To detect a half-point lift (δ = 0.005): ≈ 12,500 per variant.
8.4

Reading your own numbers

Track each campaign against the same baseline every month: positive reply rate, meetings per 100 contacts, bounce rate and complaint rate side by side. A campaign that lifts replies while complaints climb is not a win; it is borrowing against the domain's reputation.

Look at the replies themselves, not just the count. Ten replies that say “not relevant to me” tell you the segment is wrong, which no subject line test will fix. Tag every reply with the triage categories from Module 6 and review the tags with the numbers. The pattern in the words often explains the pattern in the rates.

8.5

Don't peek

Checking a test repeatedly and stopping the moment it looks significant inflates false positives dramatically: in Miller's simulation, from a nominal 5% to 26.1% [5]. Decide the sample size and stopping rule before you start, or use a sequential or Bayesian method designed for continuous monitoring.

Below roughly 1,000 contacts per variant, treat any result as directional at best [5]. Pool results across segments only when the audiences really are comparable.

8.6

A weekly deliverability check

Fifteen minutes a week catches most problems before they become a blocklisting. Check:

  • Google Postmaster Tools spam rate for each sending domain [3].
  • Bounce rate by mailbox and by domain [2].
  • Blocklist lookups at check.spamhaus.org [6].
  • DMARC aggregate reports: any legitimate sender failing? Any source you don't recognise?
  • Any new SMTP error codes, such as 5.7.26 or 4.7.28 at Gmail [3] and 5.7.515 at Outlook.com [7].
Key takeaways

Remember this

  • Fix your metric definitions before comparing anything.
  • Benchmarks disagree by several times over; compare against your own baseline.
  • Detecting a 1-point lift on a 2% reply rate needs about 3,100 contacts per variant.
  • Set sample size and stop rules in advance; don't peek.
  • Fifteen minutes of deliverability checks a week prevents most fires.
Exercise

Size your next test

Before you run another A/B test.

  1. Step 1

    Write down your baseline positive reply rate from the last 90 days, using the definitions above.

  2. Step 2

    Decide the smallest lift that would change what you do. Calculate the sample size per variant.

  3. Step 3

    If you can't reach that sample in a reasonable time, test a bigger change, such as a different offer or segment, instead.

  4. Step 4

    Put the weekly deliverability check on your calendar.

In AngelSend

Where this lives in the product

Analytics

The Analytics page charts outreach and engagement over time; every stat on a campaign card opens a panel explaining exactly what that number counts.

Subject A/B tests

Campaigns can test subject lines and pick a winner automatically; on small sends, treat that winner as a lead to confirm, not a conclusion.

Bounce guard

A sending mailbox whose hard-bounce rate climbs past its threshold is paused automatically, before it damages the domain's reputation.

Check yourself

Quiz

Pick an answer to see the explanation.

  1. Q1Why do published reply-rate benchmarks differ so much?

  2. Q2Roughly how many contacts per variant to detect a 2% → 3% reply-rate lift?

  3. Q3What's wrong with stopping a test when it first looks significant?

Sources

  1. [1]Cold Email Benchmark Report 2026 — Instantly
  2. [2]The State of Cold Email 2026 — Smartlead
  3. [3]Email sender guidelines — Google
  4. [4]Cold Email Benchmarks — Woodpecker
  5. [5]How Not To Run an A/B Test — Evan Miller
  6. [6]Blocklist FAQs — Spamhaus
  7. [7]Strengthening Email Ecosystem: Outlook's New Requirements for High-Volume Senders — Microsoft

This course summarizes published rules, regulator guidance and third-party research as of October 2026. It is not legal advice; check with counsel before emailing into a jurisdiction. Vendor benchmark figures are each vendor's own data and definitions.

7. Email after the sale9. Using AI without losing trust

Practice on
your own outreach.

Start with a $3.99 Starter Pack — no subscription required.

Get started freeBook a demo

Starter pack from $3.99 · CAN-SPAM compliant · Reputation protected