I was recently asked by one of our clients how to achieve statistical significance in their decision-making based on the research we are helping them with. Our client is looking for a high level of “confidence”, because they are making strategic decisions with significant impact. In answering their question, I provided them with ways of understanding statistical significance, and eventually talked them out of attempting to achieve it.
If the collection of the data is free or very inexpensive, then achieving it can be a valuable exercise, but it is rarely free or inexpensive. I discourage the focus on statistical significance for three main reasons.
First, achieving it can be very expensive. The more questions you ask and the more you segment your population, the more data you must have in order to achieve it. The cost-benefit ratio is rarely in your favor. You can achieve a high level of “comfort” with the information (as opposed to the technical term “confidence”), with much smaller numbers.
Secondly, just because you achieve it doesn’t mean you should base your decisions on it. Many, many studies have had some minor fault in them that cause the results to be somewhat suspect. It may be in the audience surveyed, the way the questions were asked, the motivations of the participants, the self-selecting properties of people who participate in research, the way the audience was determined, and many other possibilities. When focusing on statistical significance, our clients often assume the data is fully and completely correct, and they assume they can base a decision on that data, when this is often a false assumption.
Thirdly, and perhaps most importantly, is that the data collected and analyzed for statistical significance is just one set of data to analyze. When making important decisions, companies should also collect information from partners, employees, competitors, best-practices examples, third-party sources, industry experts and others.
Good decision-making can’t be made in a vacuum, but it also doesn’t require investing in large, expensive studies. Gather data from all available resources, and in the end, trust your gut.
Glenn Gow is an expert in marketing performance, Coach, Board Advisor, Author, Speaker, Podcast Host and Founder & Advisor of Crimson Marketing. Follow me on Twitter, LinkedIn, Google+. To get a free copy of Crimson’s One-Page Marketing Metrics Funnel, visit here.