Are You Statistically Significant?
Survey researchers use significance testing as an aid in expressing the reliability of survey results. We use phrases such as “statistical significance”, “margin of error”, and “confidence levels” to help describe and make comparisons when analyzing data. Statistical significance is often misunderstood and misused in organizations today. Since more and more companies are relying on data to make critical business decisions, it’s an essential concept for managers to understand.
Understanding Statistical Significance
Suppose you want to test a new marketing campaign or a new product to determine preference over the existing campaign or product. You will want to run a test or survey with as many potential customers as possible. Since you can’t survey all potential customers, you utilize a representative “sample group” for the survey.
Common sense tells us that the larger the sample, the better the representation of all potential customers. To assist in determining the correct sample size, we look for statistical significance based on a defined “Confidence Level” and “Confidence Interval” of our survey.
Confidence Level
As a set standard, most researchers seek a 95% Confidence Level for surveying. From a statistical point of view, this means that if you were to conduct the survey 100 different times, you would receive the same results 95 times out of 100.
Confidence Interval
The other component that factors into statistical significance is the “Confidence Interval”. This percentage is always stated as a “plus/minus” percentage or range. For instance, if your survey results show that 54% of your participants favor Message A over Message B, and you have a Confidence Interval of +/-4%, then the range of preference is statistically between 50%-58%.
Unfortunately, when the Confidence Level is 95%, many people interpret this as “there’s a 95% chance that Message A’s true percentage lies between 50% and 58%” which is incorrect. The correct way to interpret these results is that 50-58% of respondents prefer message A, which will hold true 95% of the time if the survey were to be conducted again. As a standard, a quantitative survey should seek a Confidence Interval between +/-4%-5%.
Two Most Important Factors in Determining Statistical Significance
The two most important factors in determining statistical significance are sample variance and sample size.
Sample Variance
Sample variance refers to how alike or unlike survey participants are relative to your research profile. It is important to understand that the greater the variation in the population, the larger the sampling error.
For instance, if you conduct two studies – one with all U.S. farmers and the other only with corn/soybean farmers in Midwestern States, you need to take into consideration the difference in sample variation. Again, common sense will tell you that the corn/soybean farmers will have more in common with other corn/soybean farmers in the Midwest than with strawberry growers in California.
To reduce sampling error, always design your research based on specific audience profiles that have common profile characteristics. If you need input from different or varied groups, then you need to increase your sample size for statistical significance of each group.
Sample Size
To calculate sample size for any study, you need to use a complex formula which considers both Confidence Level and Interval. Fortunately, thanks to the Internet, there are several reliable websites that provide a sample size calculator for your convenience.
Sample sizes determine both the Confidence Level and Interval. The larger the underlying research population, the larger the required sample size.
But sample sizes are not linear. For instance, if you are surveying certain segments of wheat farmers in North America with a 95% Confidence Level and a +/-4% Confidence Interval, the proper sample sizes are as follows:
| Total Audience Population | Sample Size Needed |
| 250 | 177 |
| 2,500 | 484 |
| 10,000 | 566 |
| 25,000 | 586 |
| 100,000 | 597 |
Keep in mind that sample size and sample variation go hand in hand. If your research needs input from both corn/soybean farmers in the Midwest and strawberry farmers in California, you need to ensure that you have the proper sample size for each subgroup. In other words, in order to maintain the appropriate sample size and gain statistically significant results from each group, you must increase the number of survey completes based on the population of each subgroup.
If you need assistance in the planning or design of your next market research project, contact Mark Vogel: [email protected].