Statistical Significance and Sample Size

The first question asked in most marketing research is “How many people should we talk to?” This implies that a certain level of confidence is required in the answer. The most common method of ensuring a certain level of confidence is to select a sufficient sample size.

There are three interrelated aspects to calculating the correct sample size. The first is the statistical confidence level, usually expressed in a percentage or as the p value. This is commonly used to denote the accuracy of the research. The second is the allowable error, how accurate the respondents must be in providing their answers. The lower the allowable error required, the larger the sample size must be at any given confidence level. The third is the variance in the population of people being surveyed. Because this is typically not known, the acceptable practice is to set this as equal to a normal distribution of responses. These three numbers are used to calculate a sample size.

A more fundamental question related to determining sample size is the comfort level of the managers with decision-making. Some managers are comfortable making decisions with lower levels of statistical significance based on their experience; others require a higher level. Colin Powell was asked this question about the Gulf War and said that if he was 70% confident of the situation, he could make his decision. Many managers work at a higher comfort level but may not require a 95% confidence level with 5% allowable error. A researcher should ask the decision-maker how precise an answer they need to make their decision. This will ultimately determine how many people are needed for a study.