A/B testing for ads: deciding what to test first

An ad account has a dozen things you could test at any moment (headline, image, audience, bid strategy, landing page, button color) and no built-in way to tell which one is worth the two weeks it takes to get a real answer.

Most testing advice stops at "test everything," which is not a plan. It's an invitation to spend a quarter moving pixels around while the thing actually holding your conversion rate down sits untested. The failure here is rarely a shortage of test ideas. It's that nobody ranked the ideas already sitting in the backlog.

The usual failure mode is a small account running four or five tests at once on a budget too thin to reach a real answer on any of them. Results whipsaw week to week, nothing hits statistical significance, and the conclusion drawn is "testing doesn't work for us": when the actual problem was never running one test at a time on the element that mattered most.

The fix is a scoring method, not more discipline.

The priority score: impact divided by resource

Beginners Guide To Ab Testing opens with a simple matrix: score every candidate test on two axes, impact and resource cost, then divide one by the other. Impact runs 1 to 5: a 5 means the element has a direct line to conversion or revenue, a 1 means it nudges behavior at the margins. Resource cost runs the same scale, but backwards: 5 means the test eats real design and development time, 1 means someone can ship it this afternoon.

The book's own worked example is the useful part. Headlines score 5 for impact and 1 for resource, for a priority score of 5. Images score 5 for impact but 3 for resource, landing at 1.67. Button colors score 1 and 1, for a priority score of 1. Audience targeting scores 5 for impact but 5 for resource too, so it also lands at 1, tied with button colors, despite being a completely different kind of test.

That last pairing is the point. Two tests can share a priority score for opposite reasons: one because it barely matters, the other because it matters enormously but costs too much to run yet. The score tells you what to do this week, not what matters most in the abstract.

The guide then sequences the ranked list into three phases instead of a flat list. Weeks one through four go to foundation elements (primary headline, main image, the core value proposition) because these are cheap to change and carry the most weight. Weeks five through eight move to calls-to-action, audience targeting, and bidding strategy, once the message itself is settled. Weeks nine through twelve are reserved for layout changes and cross-channel work, the tests that touch the most moving parts and are hardest to read cleanly if the message underneath them is still shifting.

Run the phases out of order and you end up optimizing a layout around a headline you're about to replace.

Where people go wrong

The most common mistake is testing whatever's easiest to implement rather than whatever scores highest. Button colors get tested constantly for exactly this reason: a developer can ship a color swap in ten minutes, and it feels like progress. The framework's own matrix scores that test a 1. A headline swap, also fast, scores a 5. Ease of implementation and priority score are not the same axis, and treating them as interchangeable is how testing programs spend months on the wrong end of the matrix. Going deeper on this system is what we cover across the whole account, not one test, in the paid ads and landing pages guide.

The second mistake is skipping the baseline audit. Without a documented starting point (current conversion rate, cost per acquisition, historical performance trend, return on ad spend) a result that looks like a win might just be a normal week. The guide's fix is a performance baseline document, built before the first test launches, not after, so a 12% lift means something instead of being lost in the account's usual week-to-week noise.

The third is not writing anything down. The framework includes a test documentation template: element tested, hypothesis, implementation details, results summary, and the learning points that came out of it. Teams that skip this step tend to rerun a headline test that already lost eight months earlier, because nobody remembered, or lose the one finding worth keeping when the person who ran the test moves to a different account.