Making sense of election surveys

DATABASE

TO help guide the public in comprehending the significance of surveys and questionnaires, noted sociologist Howard Schuman explains in plain language the nature, importance and problematic aspects of surveys in Contexts magazine, the newest publication of the american Sociological association.

A professor and research scientist emeritus at the University of Michigan, Schuman noted in his article, “Sense and Nonsense,” that surveys have appealed since ancient times to two human propensities: (1) Gathering information by asking questions and (2) attempting to learn more about one’s environment by examining a small part of it (i.e., sampling).

Through an illustrative example of the sampling conducted by the infamous Literary Digest poll prediction of the 1936 US presidential election, Schuman demonstrates both the problems and progress made in survey methodology. On the basis of a very large sample—2 million completed and returned surveys out of 10 million distributed—the Literary Digest had incorrectly predicted that Franklin D. Roosevelt would lose decisively

in the election to George Dewey.

“At the same time, pollster George Gallup, using many fewer cases but a better method, accurately predicted that Roosevelt would win. Gallup used quotas in choosing respondents in order to represent different economic strata, whereas the Digest had worked mainly from telephone and automobile ownership lists, which in 1936 were more likely to include wealthy people who were likely to be opposed to Roosevelt,” explained Schuman. (There were other sources of bias, as well.) As a result, the Digest disappeared from the scene, and Gallup became a household name.

Schuman highlights two advances in survey methods since the 1930s and 1940s that allowed the modern survey to develop and flourish.

“The first is the emergence of probability sampling, which is Interestingly, however, for mathematical reasons, reasonably accurate estimates can be obtained—from populations of very different sizes—with sample sizes of around 1,000, and only when extraordinary precision is needed do samples need to be much larger.

fundamental to social science research today and helps overcome the problem of biased sampling of respondents. The second relates to the development of greater precision in asking questions in surveys and in replicating and testing the validity of the questions themselves,” said Schuman.

Obtaining a good sample depends on more than adhering to principles of good probability sampling (i.e., selecting respondents from the population in proportion to the degree to which they are representative of that population). Well-done surveys also depend on the quality of their implementation.

Thus, when members of a population who are selected to be in the sample are not successfully interviewed or do not respond to specific questions, the omissions can seriously compromise the validity and reliability of the survey and are as important as the sample size. Interestingly, however, for mathematical reasons, reasonably accurate estimates can be obtained— from populations of very different sizes—with sample sizes of around 1,000, and only when extraordinary precision is needed do samples need to be much larger.

The “margin of error” plus/minus percentage figures that accompany most media reports of polls reflect the size of the final sample, but they do not reveal the sampling methods or the extent to which the targeted individuals or households were actually included in the final sample. For instance, the percentage of people who refused to take part in a survey is particularly important.

In some surveys, the percentage is small, within the range of 5 to 10 percent. But even in the best nongovernment surveys, the refusal rate can reach 25 percent or more, and it can be far larger in the case of poorly executed surveys. The opinions of nonresponders in a population may be very different from those of responders, thus their noninclusion biases results.

Many survey reports are based on such poor sampling procedures that they do not deserve to be taken seriously. This is especially true of reports based on “focus groups,” which offer human interest but are subject to vast amounts of error. Internet surveys also cannot represent the general population

adequately at present, though this is an area where some serious attempts are being made to compensate for the inherent difficulties and sampling bias.

Schuman also describes several techniques researchers use to improve survey questioning—such as asking several different types of questions about any important issue; varying the form, wording and context of questions; and introducing comparisons into analyses.

These techniques are utilized in order to overcome variations in response that are produced not by the responders’ actual opinions but by the peculiar form, wording or frame of reference of questions in a survey. Using several examples from actual surveys, he shows how different ways of asking the same question can produce substantial differences in outcomes.

Despite the methodological

advances, some issues still remain doubtful. For example, pollsters still face the task of interpreting the meaning of questions and the associations among questions, but this is true in all types of research, including those using field observations.

For Schuman, surveys remain one of the best tools for learning about large populations. “In the end, however, with surveys, as with all research methods, there is no substitute for both care and intelligence in the way evidence is gathered and interpreted. What we learn about society is always mediated by the instruments we use, including our own eyes and ears,” concluded Schuman.

Founded in 1905, ASA is a nonprofit membership association dedicated to serving sociologists in their work, advancing sociology as a science and profession, and promoting the contributions and use of sociology to society.

The author is a member of ASA and he can be reached at cecilio.arillo@gmail.com