I was recently asked my opinion on a piece of survey research and how valid I thought it was. One of the easiest tests for validity of any piece of research is in its disclosure statement and methodology.
The more a study discloses, the more likely it is the study sponsor is willing to stand behind the results publicly, including how valid the results are. The less a study discloses, the more likely it is the study sponsor has something to hide.
Compare the disclosure statements of the following studies:
The above study is by the European Union, and is thorough in its disclosure.
The above study is a private study by a law firm, and is minimal and vague in its disclosure.
Which study is more credible? Based on the amount of disclosure, I feel confident in the European Union’s study. I don’t feel confident in the law firm study.
If we want our studies to be credible, we must err on the side of disclosing as much as possible about our methodology, about how our survey was conducted.
Best Practices for Survey Disclosure
So, what should we include in any kind of survey or study done for marketing purposes? The American Association for Public Opinion Research (AAPOR) publishes a set of disclosure standards which meet their code of professional ethics. It’s one of the gold standards in the field of market research; adhering to AAPOR guidelines in our methodology disclosure statements will help prove credibility.
The following information belongs in every disclosure statement, from sections 1 and 2 of the AAPOR standards:
- Who sponsored the research study, who conducted it, and who funded it, including, to the extent known, all original funding sources.
- The exact wording and presentation of questions and responses whose results are reported.
- A definition of the population under study, its geographic location, and a description of the sampling frame used to identify this population.
- If the sampling frame was provided by a third party, the supplier shall be named. If no frame or list was utilized, this shall be indicated.
- A description of the sample design, giving a clear indication of the method by which the respondents were selected (or self-selected) and recruited, along with any quotas or additional sample selection criteria applied within the survey instrument or post-fielding.
- The description of the sampling frame and sample design should include sufficient detail to determine whether the respondents were selected using probability or non-probability methods.
- Details about the sample design, including eligibility for participation, screening procedures, the nature of any oversamples, and compensation/incentives offered (if any).
- Sample sizes and a discussion of the precision of the findings, including estimates of sampling error for probability samples and a description of the variables used in any weighting or estimating procedures.
- The discussion of the precision of the findings should state whether or not the reported margins of sampling error or statistical analyses have been adjusted for the design effect due to clustering and weighting, if any.
- Which results are based on parts of the sample, rather than on the total sample, and the size of such parts.
- Method and dates of data collection.
- Sources of weighting parameters and method by which weights are applied.
- The methods used to recruit the panel, if the sample was drawn from a pre-recruited panel or pool of respondents.
If we see a survey in the news that includes all of the above points in its disclosure statement and methodology, it’s likely the survey was well-executed and the data is likely credible.
As marketers seeking to represent our companies well, we must insist upon holding ourselves to these standards – as well as any partners or vendors we work with.
Better Research Leads to Better Results
Before we base any decisions on survey and study research, we must know how credible the research is. Bad research yields bad results, and decisions made on bad results lead to catastrophic failure.
If you’re making decisions about the health of your business based on other people’s research, use disclosure and methodology statements as a way to judge how valid the research is.
Want to read more like this from Christopher Penn? Get updates here:
Get your copy of Marketing Blue Belt!
Also published on Medium.