Public Opinion Polls
 Lecture Notes
 Last Updated: Wednesday, October 19, 2005


Objectives
After this section, you should be able to...
Understand why sample are needed and why they work
Understand the different methods of drawing a sample and thier consequences.
Undertand the confidence interval of a public opinion poll.
Know the diffent form of questions that are asked on surveys.
Recognize the different types of bias that can be introduced by question wording.
Be familar with the bias that can be produced by different interviewers.
"Polling" and "Surveys" are synonymous. George Gallup began using the term "poll" in his earliest polls to connect his surveys to the election and add a degree of importance.
Who uses polling?
 Campaigns
Media
Government
Marketing
Courts
Academics
 Understanding Polls
 To evaluate public opinion polls we have to ask:
Who is questioned?
How are they asked?
What questions are they asked?
 Who is asked?
 A "Sample"
A subset of the population we use to make inferences about the population in general
 Why do we use samples?
 Sampling
 Sampling Frame
A listing of the target population from which a sample is drawn
Usually not possible to assemble
 Types of Samples
 Nonprobability Samples/Straw Polls
Probability or Random Samples
 Nonprobability Samples
 "Straw Polls"
People may self-select or be chosen by some non-rigorous method.
You can not estimate the likelihood someone will be include
 Nonprobability polls
 "Typical" people
Purposive
Volunteer subjects/self selection
Haphazard sampling
Quota
Snowball sample
 The Perils of Straw Polls
 Shere Hite vs. Dear Abbey
Hite distributed 100,000 survey and found that 70% of women married 5 or more years had affairs.
Abby received 200,000 responses to a column and found 15% of wives were unfaithful
 The Literary Digest Poll
 Scientific Polling
 Based on random samples
 Understanding Samples
 What happens if you flip a coin 100 times?
 Distribution of Coin Flips
Sampling Error
 Error that arises from trying to represent a population with a sample.
Not a mistake on the part of the survey researcher, it is caused by the nature of random samples.
Can not be calculated for a Nonprobability sample
 The Advantage of Random Samples
 Randomness insures that the only source of error/bias in the sample is sampling error -- which we can calculate
 Margin of Error or Confidence Interval
 Every poll has a confidence interval.
Accurately reported it will say something like "± 3% with a 95% confidence level."
 Confidence Interval
 A poll shows a candidate having the support of 47% of respondents with ± 3%
This means the candidate’s support is--most likely--between 50% and 44%.

Most surveys use a 95% confidence interval or a 5% alpha level.
This tells us that in 95% of the polls the percentage found in the sample will fall within the confidence interval around the actual percentage that exists in the population
 Tradeoffs in the confidence interval
 Level of certainty
Do you want to be 90% or 95% certain?
 Precision of results
Is ±3% accurate enough or more precise than you need
 Sample size
The bigger the sample, the better
 Sample Size and Confidence Interval
Tradeoff between margin of error and sample size

Stratified Samples
 What if you want to know something about the characteristics of sub-units?
For example, what if you want to compare GW Bush’s rating in Texas to the nation?
 Sampling Problems
 "Non-coverage" errors
Wrong population sampled
Non-response errors
 "Non-coverage" errors
 Phone surveys
About 30% of phone numbers are unlisted
About 5% of homes do not have a phone (25% of households with income under $15,000 a year)
Answering machines are used to screen calls
5% of phone interviews are not finished because the respondent hangs up
Students, soldiers, the homeless and people in hospitals
 Wrong population sampled
 Targeting the wrong people
How do you predict voting?
 Non-response errors
 Polling’s "Dirty Little Secret"
What happens when people don’t want to answer pollster’s questions?
Are people who do not respond different from those who do?
 Causes of Non-responses
 "Sugging"
"Frugging"
"Cugging"
Answering machines
Changes in work
Too many surveys
 Bias in non-responses
 In one experiment, after 1 attempt to reach respondents, 38% of the sample was working outside the home
After 4 tries the percentage had risen to 58%
After 7 tries the percentage had risen to 63.5%
 Declining Response Rates
 Today most polling firms find that they can not reach about 20% of their sample--even after making 6-7 attempts to reach them.
Some firms use specially trained interviewers to contact reluctant respondents
 "Weighted" Results
 Sometimes in order to correct for nonresponse errors that lead some groups to be under-represented in a poll

How are they asked? By mail, in person or by phone?
 Reaching respondents
 By phone
 By mail or e-mail
May be slow since respondents may respond at their leisure
Survey is self administered by respondent and may not be filled out by correct person

In person

Presents risks and interviewer bias
 Interviewer Bias- Gender
 Agree/Disagree Question: "The decision to have an abortion is a private matter that should be left to the woman to decide without government question."
Respondent
InterviewerFemaleMale
Female84%77%
Male64%70%
Interviewer Bias- Race
 Respondents will shape their responses based on the race (or perceived race) of the person asking the questions.

Perceived race of interviewers

BlackWhite
Black elected officials can best represent the interests of the black community64%49%
American society is fair to everyone14%31%
The American legal system is unfair to blacks84%72%
The police are too much like just another gang to stop gang violence.52%42%
Cluster or Multistage Sampling

What is asked? -- Questionnaire Construction
Open Ended Questions
 Lets respondents use their own words to answer questions
"What do you like/dislike about the Republican Party?"
"What do you do you consider the most important problem facing the nation today?"
 But, hard to summarize or "code"

Closed ended Questions

Respondents choose from a list of answers selected by the survey researcher
"Do you approve or disapprove of the way Bill Clinton is handling his job as president?
7-point scales
 The seven point scale comes from the psychological conclusion that individuals have difficulty reliably making distinctions between more than seven options.
In phone surveys it is generally regarded as having no more than five response categories.
 7-point scales
 "Recently there has been a lot of talk about women’s rights. Some people feel that women should have an equal role with men in running business, industry, and government. Others feel that women’s places is in the home"
 Agree Disagree
 Respondents are asked to agree or disagree with a series of statements
 Agree Disagree
 "Should marijuana be legalized?"
 Feeling thermometer
 100° Very warm and very favorable feeling
85° Quite warm or quire favorable feeling.
70° Fairly warm or …
50° No Feeling at all
40° Slightly cold or slightly unfavorable feeling
30° Fairly cold or fairly unfavorable feeling
15° Quite cold or …
0° Very cold or very unfavorable.
 Branching
 Sometimes you want to get at detailed/complicated responses without overwhelming the respondent.
For example:
"Do you consider yourself a strong Democrat, a weak Democrat, an independent leaning Democratic, an independent, an independent leaning Republican, a weak Republican or a strong Republican?
 Branching
 Rather than having a long question can have a series of short questions
Do you consider yourself a Democrat, Republican, and Independent or what?
Do you consider yourself a strong or weak _____?
 Batteries of Questions
 Sometimes a series of questions can be used to get at tough to measure issues like "liberalism" or "conservatism"
 What messes up question?
Loaded Questions
"Push" Polls
Uncertainty and Ambiguity
 Loaded Questions
 Questions designed to return a particular response.
 Loaded Questions
 "Do you believe that smut peddlers should be protected by the courts and Congress, so they can openly sell pornographic materials to your children?"
 Loaded Questions
 "A recent study by a psychology professor at a leading university concluded that the amount of violence children see on television has an effect on their likelihood of being aggressive and committing crimes. From what you have seen or heard about this subject, do you agree…"
 Loaded Questions
 "Which segment of the television viewing audience do you think will suffer more if federal funding for PBS was cut back sharply or eliminated--children or adults?"
 Loaded Questions
 Do you aree or disagree with the statement
"When (Bernhard) Goetz said in his confession that he used dum-dum bullets, that he was sorry he didn’t gouge out the eyes of the four he shot, and that if he could have reloaded his gun fast enough, he would have taken out after them, he looked more like a ‘Death Wish" gunman out stalking to kill criminals, not an innocent victim trying to defend himself."
1985 Harris poll
 Push Polls
 Polls that provide respondents with bits of information to see what impact they have on citizens’ choices
 Abuse of Push Polls
 When information is provided that is intended to influence-rather than measure-respondents’ opinions
Thousands of homes are called.
 Push Polls
 "As you may know, in 1974, Jerry Springer, who had gotten married 6 months earlier, was arrested on a morals charge with 3 women in a hotel room. He also used a bad check to pay for the women’s services, and subsequently resigned as mayor of his city. Does this make you much more likely, somewhat more likely, somewhat less likely, or much less likely to support Jerry Springer for governor this year?"
 Push Polls
 "If you knew that Pat Buchanan had said women were less equipped psychologically for the working world,’ would you be more likely to vote for him, less likely, or does it make no difference in your vote?"
 Uncertainty & Ambiguity
 When respondents are unsure about the facts or the question being asked they may offer up answers that are not reliable.
"Nonattitudes" or "Doorstep responses "
Uncertainty
 "If you do a 20-minute poll on environmental issues, you’ve engaged most people in a longer conversation than they’ve ever had about the environment in their lives."
 Uncertainty
 "Do you favor or oppose the Public Affairs Act of 1975?"
 "Do you favor or oppose the Public Affairs Act of 1975 or don’t you have an opinion"
Ambiguity
"Does it seem possible or does it seem impossible to you that the Nazi extermination of Jews never happened?"
 Ambiguity
 "Should military force be used if necessary if U.S. forces in Panama are threatened?"
CNN Newsnight Poll
"Women in Combat"
 Ambiguity
 "How many public officials do you think are a little bit dishonest-most, some, a few, or none?"
 Types of Question Problems
 Context Bias
Social Desirability Bias
Consistency/Moderation Bias
Agree-ability bias
No Middle Point
Unbalanced Questions
Response Choice Order
 Context Bias
 Information contained in the question influences the response
 "President Clinton said that the 1975 Public Affairs Act should be repealed. Do you agree or disagree?"
"The Republican Congress said that the 1975 Public Affairs Act should be repealed. Do you agree or disagree?"
 Context Bias
"The U.S. has 36,000 troops and airmen in South Korea. If North Korea invaded South Korea, we have a firm commitment to defend South Korea with our own military forces. If South Korea were invaded by Communist North Korea, would you favor or oppose the U.S. using troops, air power and naval power to defend South Korea?" [43%]
"As things stand now, do you feel the war in Korea has been worth fighting, or not?" [27%]
 Social Desirability Bias
 People don’t like to admit to unpopular beliefs or actions.
Just ask them about
 Consistency Bias
 Moderation Bias
 Agree-ability bias
 People prefer to agree rather than disagree.
Respondents to consistency agree on agree/disagree questions while not seriously reading the questions.
 No Middle Point
 If people who feel neutral forcing them in one direction or another draws out something random or unrelated to the question
 Unbalanced Questions
 Having more responses on one side than another.
 Response Choice Order
 When questions are read aloud to respondents they tend to choose among the last few options
When questions are read by the respondent they tend to choose among the early options
 Ethics in Polling
Professional Standards
 The American Association for Public Opinion Research (AAPOR) and the National Council of Public Polls (NCPP) has published a set of standards.
 How should polls be reported?
 In 1997 pollster Frank Luntz was formally chastised by the American Association of Public Opinion Researchers for failing to reveal details of polls that allegedly showed public support for the Contract with America
 Other Types of Surveys
Panel Studies
 Polls in which the same voters is questioned repeatedly over time.
Designed to study change in opinions
 Problems with Panel Studies
 People become sensitized by questions and may end up behaving differently than other citizens.
 Exit Polls
Exit Polls
 Surveys taken as voters leave the polling place
In recent years conducted by Voter News Service (VNS) in a cooperative effort with numerous news outlets
Usually short
Usually large samples
In 1996 150,000 voters from 1,500 precincts were interviewed to insure samples of all 50 states
In 1998 120,000 voters from 1,200 precincts were interviewed
 Focus Groups
Focus Groups
 7 to 10 people (even though they may range from 4 to 12) who do not know each other who brought together to generate a discussion that will reveal their perceptions in a friendly, casual atmosphere
 Advantages of Focus Groups
 Avoids the "Passive Respondent"
Ability to explore the reasons behind opinions
Simulates normal interaction
Cost
Focus Groups in Politics
 Shape surveys
Probe ideas
Test advertising/themes
 Disadvantage of Focus Groups
 Not a representative sample
 On-line polling
Problems with e-mail polls
 Voluntary
No population list
Not anonymous
Computer users younger, whiter, more male, better educated and wealthier than population
 Web Polls/Panels
 Some researchers are using on-line surveys that contact the same respondents on a repeated basis to create electronic panels.
 Internet Polls
 People magazine’s website poll that named Howard Stern's sidekick "Hank the Angry, Drunken Dwarf" as one of the 50 most beautiful people in the world
Electronic ballot stuffing