Abraham says tiny split tests tell you nothing
Five variables the Leadbird founder says stopped mattering at 2M sends a month, and the one that is hardest to copy at small volume.
Nick Abraham posted a list of five variables he says he stopped caring about after sending 150M cold emails for more than 1,000 clients, and says he currently sends 2M a month for 300. The list is clever copy, tiny copy A/B tests, big case study numbers, branded domains and the sender name. He returned to the split test point in a second post, needling the people who disagreed with him, which is where this stops being a list and starts being an argument he has made before, earlier.
The split test example
His illustration is deliberately small. Variant A gets 2 positive replies, variant B gets 1, and the team calls A the winner on a technical 100% lift. Abraham's position is that this says nothing about what happens on the next 100,000 emails, and that he has never seen tiny syntax changes reproduce consistently at his volume. When a test does move, he says, the two versions usually differ by more than wording, a different angle, offer, proof or audience.
The rest of the list
On copy, his argument is that a strong offer carries an average email. The examples he gives are plain, including "We'll send you 3 qualified candidates for the role you're hiring for." On case studies, he contrasts a $6M figure against a $50M figure and says both names mean nothing outside lead gen, while "We helped Apple with its rebrand" carries no number and lands harder.
The branded domain point is operational. He says the old setup ran roughly 14 days warming domains plus roughly 14 days ramping volume against a 30-day contract, so the first month went to getting to full send volume. Now his team keeps pools of non-branded domains warming before a client signs, pulls the relevant domains on signature and creates inboxes under the client's sender names, which he says takes onboarding to live in under 72 hours. On sender names, he says new inbox names go into campaigns almost immediately without a meaningful change in results once the domain is warm.
For people sending, the catch is in the same number that makes the claim strong. Abraham can dismiss a 2-versus-1 result because he has another 100,000 sends behind it. An operator running a few thousand emails a month does not have a larger sample waiting, so the practical takeaway is not that testing is pointless, it is that the only differences worth testing at that volume are the big ones he names, offer, audience and framing.
there are genuinely people who believe small split test data is valuable data
I've NEVER seen tiny syntax changes consistently reproduce at our scale.