Predicting advertising success: Why you need to combine research methods

Having spent much of my career in advertising research, I’ve always been keen to understand which copy-test metrics predict in-market sales responses most accurately.

Different research companies favor different measurement approaches, including:

• Traditional prompted closed questions (e.g. ad enjoyment rating scale).

• Intuitive questions or tasks (e.g. using pictures of facial expressions to ask about emotional responses or measuring how quickly people respond to a question, getting closer to system 1).

• Inference (e.g. comparing purchase intent before vs after exposure to the advertising or behavioral measures such as whether people watch the full ad).

Each agency shows evidence for the validity of their approach and the inadequacies of other approaches. They can’t all be right. 

My collaboration with research platform company Zappi has given me a unique opportunity to find out which measurement approach is best. They have given me access to their database of over 2,300 US video ads, researched using their Zappi Amplify Ad System. In this article, I’ll describe what I learned in my analysis of that database.

Zappi’s method and database includes metrics based on all three of the approaches I described above. 

It also includes a derived statistic, "Creative Effectiveness,” that I have used as a surrogate for sales effectiveness in my analysis. This metric, which was developed and tested by Zappi in collaboration with its biggest clients, represents the combination of Zappi’s creative testing measures that best predicts in-market sales effectiveness as estimated via econometric modelling. No single metric can predict sales uplifts accurately, but a combination of Zappi’s metrics provides a remarkably strong correlation. 

The relationship between the Creative Effectiveness score in Zappi Amplify and ROI from market mix modeling ranges from 0.8 in the US to 0.6 in Mexico and Russia. According to Zappi’s head of advertising research Kim Malcolm, their Creative Effectiveness measure has predicted their client’s sales results more accurately than competitor methodologies in every parallel test they have conducted.

My analysis of the Zappi database sheds light on why this is the case. It shows that the best prediction of sales effectiveness requires a combination of all three measurement approaches.

1. There are three ingredients of effective advertising

The measures that correlate with overall effectiveness can be thought of as belonging to three areas: Attention, Branding and Persuasion. 

It re-confirms the well-established idea that effective advertising must command attention, showcase the brand name and make the brand seem relevant/worth buying. Attention and Branding ensure the advertising will build long-term memories, while Persuasion is important for producing an immediate sales response.

2. Emotion is key to commanding attention

Claimed attention, enjoyment and overall emotion all correlate highly with each other — suggesting that triggering an emotional response is key to gaining people’s attention. Again, this is widely accepted within the industry. This analysis also indicates that making your advertising highly distinctive is another way to command attention.

3. Brand Distinctiveness is a major success factor

The fact that Brand Distinctiveness correlates strongly with overall effectiveness supports the findings of the Ehrenberg Bass Institute captured in How Brands Grow by Professor Byron Sharp. Not surprisingly, the more distinctive the ad, the more distinctive the brand seems. 

This reinforces the idea that featuring distinctive brand assets in your advertising enhances its sales effectiveness.

4. Traditional verbal questions correlate highly with each other

I was lucky enough to work with Gordon Brown (founder of research company Millward Brown, now Kantar —  not the UK prime minister) in the early 90s looking for verbal questions designed to assess advertising effectiveness that didn’t correlate highly with each other. It was almost impossible! The same is true here. If someone loves an ad, they find it hard to criticize it in any way. This “halo” effect means that an ad’s shortcomings are often under-represented in the results (and vice versa).

5. The three different measurement techniques correlate less with each other

Unaided Brand Recall involves showing people the ad, disguised within a reel of other ads. It represents the proportion of people who mention the brand when asked which brands they remember seeing advertised. This measure doesn’t correlate with any other metric — yet it does correlate with overall creative effectiveness and contribute to the prediction of in-market sales results. Without it, sales predictions are less accurate.

The same applies to Purchase Uplift. This measure is the change in how many people said they are interested in buying the brand before vs after a disguised exposure to the ad. Purchase Uplift contributes to overall effectiveness — but has low correlations with other measures. Its highest correlation is with the traditional verbal rating scale for persuasion, as you might expect.

6. Zappi’s “Emotional Intensity” metric is unique

This Emotional Intensity metric involves respondents using emojis to indicate their moment-by-moment emotional reactions to what they are seeing/hearing as they watch the ad. Using emojis makes the task very intuitive because most people use emojis quickly and intuitively in their messages and social media posts every day. Also, the way responses are scored isn’t something consumers are aware of so they can’t answer the way they think they should or base it on how they’ve answered any previous question. Emotional intensity reflects how many emojis people click on as they watch the ad. It doesn’t matter if they are positive or negative — the audience just needs to feel something. 

If you want your ad to be memorable, you need to take people on an emotional journey. You don’t want to leave people with a negative emotion, but you do want them to experience highs and lows as they watch the ad. The same is true of movies and theatrical productions.

Founder of advertising research company Ameritest has published strong evidence for how having emotional peaks and troughs contribute to ad memorability and effectiveness. Zappi’s Emotional Intensity metric is a practical way of quantifying this effect. The fact that it correlates with viewers watching an ad through to the end suggests that it measures the ad’s ability to tell a captivating story. This is gold dust for teams aiming to create advertising that sticks in memory and builds long-term brand success.

In conclusion, this data set indicates that if you want the most accurate prediction of an ad’s in-market sales effectiveness, you’ll need to include metrics derived from traditional verbal questions, more intuitive questions or tasks and inference from disguised techniques. 

This explains why all three approaches are used by different research agencies. They are all valid in their own right, but combining them is the best approach. 

Zappi’s new flagship pre-testing method, Amplify, uses all three which, according to Zappi’s R&D work, makes it 60% more predictive of an ad’s ROI than other research approaches. Reach out to learn more about how Amplify can help you create your best ads.

Dan White is a marketing expert and illustrator with 33 years’ experience in brands, advertising, and market research. He has worked with many of the world’s biggest brands and advised on over 500 advertising campaigns. His summaries and illustrations have influenced hundreds of thousands of marketing and insights professionals through his social media activities and three best-selling books. In this piece he draws on his experience of the challenges faced by market research of the past in delivering what insights teams and marketers need. And he looks at how much more impact the industry could have today if it embraces recent advances in technology.