Everything you need to know about market research automation

Market research automation is big news. It was identified as one of the top three game changers in this year’s GRIT report, after big data, and storytelling.

GRIT Game Changers report shows that market research automation has increased as a game changer year over year. Another study has shown that 80 percent of researchers believe market research automation will grow; 60 percent say it enables them to deliver projects faster and 50 percent say they’ve used it to lower costs.

So, what do you need to know about market research automation, the potential benefits, and how to implement it in your organization?

Let’s start with the basics.

If you Google “Market research automation” you’ll find articles about a range of technologies, tools, and techniques ranging from data analysis solutions to DIY platforms to AI.

More broadly speaking, a useful definition is that market research automation is “the use of systems with minimal human intervention, reducing or eliminating unnecessary human labour activities and thus allowing people to focus on high-intelligence processes.”

Another definition gives us some more insight into the value of market research automation, saying it is “the absence of humans in the fundamental execution of certain steps. Automation enables both experts and non-experts alike to leverage technology for greater efficiency.”

It’s useful to think about market research automation in terms of whether what’s being automated is an entire end-to-end research project workflow, or a stand-alone element of a research project or process.

Market research automation began with discrete stand-alone elements; the “low hanging fruit” such as data collection, and analysis with packages such as SPSS. It is, by definition, more difficult to automate an entire end-to-end project, but in recent years, as market research automation has become more mainstream, and new tools and techniques have been developed, this approach is becoming more common.

The move towards automated data collection rather than traditional human face-to-face or telephone research has been happening for some time, especially in quantitative research.

There is a vast range of automated data collection software available, from DIY tools such as SurveyMonkey, to expert-designed product suites such as Zappi for advertising and innovation, or Qualtrics for experience management.

Further data from this year’s GRIT report shows that the majority of researchers use online and mobile surveys in their portfolio of their top three techniques: online surveys are the top method for 59 percent of quant researchers, and mobile-only surveys are ranked second at 38 percent.

Sampling

Sampling has also become increasingly automated. Sample and panel providers, such as Dynata or Cint offer services that make it easier and faster to find sample and fulfill survey quotas, by aggregating disparate sources and automating the process of cleaning sample, and monitoring and adjusting as projects proceed.

This leads to faster execution, lower operating costs, and reduced error rates. There are concerns about the negative impact of taking the human element out of sampling. For example, unless it is intelligently managed, automation can lead to a poor respondent experience: having to repeatedly complete demographic information, frequently being out of quota for surveys of interest, or being unable to access surveys they have been invited to.

However, today’s intelligent automated and programmatic sample providers are working on solutions that actually contribute to improving the participant experience, as well as reducing fraud, thereby improving data quality.

Data Cleaning

There are any number of tools available that can take the pain out of one of the most important, but also the most repetitive and soporific of tasks. From checking and correcting formatting, de-duplicating and spotting discrepancies based on previous answers, much data cleaning can now be done automatically. From inexpensive tools such as MS Excel, which offers a number of features that can speed up the process, to full scale BI solutions such as Sisense, data cleaning doesn’t have to be a manual chore.

Data Analysis

Most quantitative research analysis has been automated to some degree for some time, using packages such as SPSS, for example. But now, newer market research automation tools, such as NVivo are taking on the labor intensive work that is typical of much qualitative research, bringing structure to text or image-based data, using techniques such as tagging or categorizing responses, creating heat maps, or machine learning and natural language processing.

These tools can enable qualitative research at a scale that would be prohibitive for humans to accomplish, help to bring together qual and quant data to improve consumer understanding, and also enable comparing and analyzing across projects to deliver macro level business insights.

Data Visualization

Another of the most time-consuming elements of research has legions of researchers poring over their data in Excel, to create hundreds of PowerPoint slides. Not only is this inefficient, the results aren’t always seen as high value by busy stakeholders, who don’t want to look at hundreds of slides.

Market research automation tools, such as MarketSight, can now automate the process of bringing the data to life, enabling stakeholders to extract insight and meaning, at a glance.

Dashboards that focus on business issues, show KPIs together with direction of travel, or comparisons with other data and industry benchmarks, can take much of this work away from researchers, leaving them free to focus on understanding what the research means for the business.

Respondent Engagement

It’s critically important to keep respondents happy and willing to continue taking part in market research. At scale, this can be complex and laborious.

Most market research automation platforms, such as Zappi, now have the same sort of inbuilt communications tools as social media or content platforms that enable you to set up respondent invitations, reminders, thank-you notes and other communications based on programmable criteria, and triggered by project events.

This type of market research automation refers to platforms that enable researchers to choose their project parameters, such as sample size, audience type, and location; input their questionnaire requirements, often choosing from pre-defined question banks, as well as bespoke questions; upload their stimulus; and then sit back, and await results.

The type of projects that are most often automated in this way tend to be for research into tactical areas such as concept, copy and claims testing. For example, brands want to understand how well their creative is performing before their activity starts (to assess messaging), while it is live (to see how the target audience is reacting) and when completed (to determine ROI).

These types of tests are great candidates for automation as they can – and should – be fairly standardized, and typically require frequent repetitions, as brands test multiple instances of product or creative.

Automation can improve the accuracy of this kind of research, speed up the collection of data and save money. Market research automation platforms of this type are designed by experts, to be used by researchers, and their internal customers alike, and have guardrails programmed in to reduce the risk of errors, and to increase the quality and consistency of the outputs. For example, rating scales can be predetermined, so that the user doesn’t have to create their own.

Cost Effectiveness

It is self-evident that anything that takes humans out of a process will ultimately, once any initial investments are out of the way, substantially reduce costs. The advent of online research completely changed the cost model for quantitative survey research by taking the costs of human telephone interviewers out of the process. Similarly, anything that reduces the need for researchers to do time-consuming, monotonous tasks, such as data cleaning or making PowerPoint decks, will save money.

Speed

Similarly, automated processes which aren’t relying on how fast individual humans can check tables or conduct interviews have brought market research project turnaround times down from weeks to days or even hours. In today’s digital world, this is essential. Brands can’t afford to wait weeks for the consumer insights they need to stay competitive.

Scale

The combination of cost effectiveness and speed make it possible to do more research. In particular, the end-to-end tactical projects described above have become so fast and cheap to do that marketing stakeholders are likely to want to test more of their creative ideas. In addition, advances in automating the structuring of text or image data has made it possible to do qualitative research at the scale of quant.

Accuracy

Repetitive tasks such as data cleaning, entry or checking are notoriously error prone, so implementing market research automation can greatly reduce human error.

Other forms of automation, such as mobile-based research can improve accuracy by being deployed in context. For example, surveys can be generated by consumer touch points, such as being in a store, or having a service experience, which makes it much more likely that respondents will be able to answer accurately about their behaviour, than relying on their memories of the event some time later.

Higher Value Work

Market research automation enables Insights professionals to spend less time on low value, repetitive tasks, and more time curating consumer insights and driving change in the business. For example, data visualization tools and dashboards that present data in chart form can replace the hours that researchers often spend making PowerPoint decks.

Automation can’t do everything, and not everything can be automated. 

One industry commentator noted that, “there is an immense challenge of applying machines to any tasks that call for flexibility, judgment, or common sense… tasks that cannot be substituted by computerization are generally complemented by it”.

This is good news for researchers, as it means that we still need humans to bring context to research findings, make sense of the results and apply them to business decision making. A great example of the importance of applying human intellect and common sense comes from the publishing sector.

Trinity Mirror’s group’s editor-in-chief Lloyd Embley has observed that if they used the same automated analysis for print as for the online title, they would put the crossword on the front page, as many people turn to it first and it’s the piece of content that people spend the most time engaged with.

However, market research automation can make researchers nervous that the robots are coming to take their jobs. As discussed above, the reality is that some elements of the Consumer Insights role will be driven out by automation, and this is good news, as it will elevate the role within the business. However, it is important to consider how to address these concerns and manage change effectively when implementing automated solutions.

There are some key steps you can take to get market research automation right.

1. Analyze what your business wants from automation, and which traditional processes you want to replace. Are you looking to self-serve using tactical end-to-end market research automation tools, or are you looking for suppliers that part-automate some of your processes, such as data visualization or qualitative data analysis.

2. Choose the features you need, based on your objectives and goals. Is low cost most important? Or ease of use? Are you looking for out-of-the-box solutions or do they need to be customizable? Do you want to include your own data into the analysis? And pull together data from across projects?

3. Choose your supplier carefully. What type of support do they offer? If you go for a self-serve model, will your stakeholders be supported 24/7? What is their cost model – ad-hoc or subscription based – and how does that work with your projected use of the tools?

4. Then build a relationship with your supplier. When you rely on automated tools, there is a certain amount of trust involved. They must be designed by experts who must have thought through any possible issues. Look at their position in the industry: are they a thought leader and will their solution grow with you and your needs?

In summary, market research automation is the present and the future of market research.

Its use will only grow, and that is great news for researchers, suppliers, marketers, and brands who want to do more for less, save money and time, and do better research.