From Pre-Designed Surveys to Analytics-Powered Research: The New Normal

Published on February 3, 2017 by Mayank Dibya

Market research as a technique to evaluate customer/consumer perception is a century-old concept; it was the genesis of leveraging concepts of data mining to devise optimal marketing strategies.

Over the years, researchers have used surveys and questionnaires as primary tools to gauge customer responses towards their products and services. The key to capturing customer sentiment lies in the maturity of the construct. Therefore, it is imperative to assess if one should rely only on the experience and skill of the researcher while forming the constructs, or should statistical concepts and algorithms be used to automatically generate intelligent and customized questionnaires.

A combination of the approaches presented below can define the right questions and identify trends better than a human researcher, eliminating bias with which they approach a problem and form the construct.

Clustered item pool: A pool of items is created using advanced statistical concepts of clustering/segmentation. The algorithm trains on the input from a pool of responses for various questions. This creates a cluster of the individual items tailored around the responses. Each cluster of item pool represents a specific response pattern.

Optimized items in a cluster: Techniques such as factor analysis/principle component analysis and correlation are used to optimize the number of items or questions, which need to be asked to measure a particular construct.

Administered response pattern: Each cluster is mapped to a set of responses. The specific items to analyze the response pattern is presented to a respondent. These responses then trigger a specific cluster of items mapping the response pattern.

Data analysis: Advancements in data science have given birth to machine learning techniques used extensively to unearth insights, which traditional statistical models fail to identify owing to certain limitations. Using both supervised and unsupervised algorithms, the responses can be mined to generate valuable insights.

So does that mean the work of researcher is limited to analyzing trends and insights post data collection? .

These techniques lack one invaluable quality that forms the backbone of market research: a researcher’s human bias. Bias can actually be a good thing, as not all insights can be gained from logic. We need researchers with the ability to take into account social, economic, historical, and cultural situations, while constructing a survey, which an algorithm or a mathematical technique cannot correctly assess.

Given that much of the interpretation of quantitative data depends on qualitative experiences, statistical and mathematical approaches will not always produce the best results.?

The key, therefore, is a hybrid approach. Like traditional data mining, advanced analytical concepts and machine learning techniques are tools for researchers to help capture correct sentiment.

---

---