There’s no breakthrough in our experiment yet, but the differences are intriguing.
It won’t be long until we learn the results of the 2022 midterm elections — and how the polls fared.
No one can be sure about outcomes on either front. The polls systematically underestimated Republicans in recent cycles, and pollsters have blamed something called nonresponse bias — the possibility that Donald J. Trump’s supporters are less likely to respond to surveys than demographically similar voters.
Nonresponse bias is a serious challenge for pollsters, who can’t hope to know anything about the people who won’t take their surveys. Without that information, pollsters don’t have a way to even account for their existence, much less devise a way to reach them.
So this fall, The New York Times tried something unusual: We paid people to take a poll.
In partnership with Ipsos, The Times paid respondents up to $25 dollars to take a political survey in Wisconsin, the battleground state where polls infamously erred in 2016 and 2020.
Beginning in early September, Ipsos mailed thousands of hard-to-miss priority mail and first-class envelopes to thousands of households across Wisconsin. The mailing contained a $5 bill and a letter promising an additional $20 if the respondent either took the survey online or returned the enclosed questionnaire in a provided return envelope.
At the same time, we fielded a traditional Times/Siena phone poll and an online probability-based panel survey on Ipsos/KnowledgePanel. This experimental design allows us to compare the results of the two groups of respondents, which, in turn, might offer a portrait of the kinds of people who might be less likely to be represented in a typical survey — for example, a moderate who isn’t a political junkie, someone who doesn’t really enjoy talking to strangers, and, yes, a Republican-leaning voter.
The data is still preliminary, and it will probably take at least six months, if not longer, before we can reach any final conclusions. But there is one immediate difference between the two groups, and that is in the polls’ response rates: Nearly 30 percent of households have responded to the survey so far — a figure dwarfing the 1.6 percent completion rate in the parallel Times/Siena poll.
How Voters Reached by Mail — and Paid $25 — Are Different From Those Reached by Phone
While respondents who received $25 were significantly more likely to participate in a political survey, they were not meaningfully different in their political views or composition than a similar group of voters who did not receive money.
These voters have similar political
preferences
IPSOS
NYT/SIENA
PHONE
Senate
44%
43%
Ron Johnson Republican
44%
45%
Mandela Barnes Democrat
Governor
42%
41%
Tim Michels Republican
46%
46%
Tony Evers Democrat
How these voters are most different
IPSOS
NYT/SIENA
PHONE
Gaps of more than three percentage points
Almost never answers calls on personal phone from non-recognized numbers
55%
40%
42%
30%
Identifies as politically moderate
Uses signs outside the home to deter strangers such as “No Trespassing” or has considered it
31%
20%
Has one to two doses of the Covid-19 vaccine
31%
20%
Thinks immigrants who are here illegally should be deported back to their home country
47%
38%
75%
68%
Born in Wisconsin
23%
17%
Has a valid hunting license
62%
56%
Is married
Doesn't trust major conservative news sources
70%
64%
44%
40%
Voted for Trump in 2020
Prefers to have a job working on a computer
19%
15%
Thinks immigrants who are here illegally should be allowed to work and should be given citizenship
30%
34%
Almost certain to vote in the November election
59%
64%
Has three or more doses of the Covid-19 vaccine
48%
54%
Prefers to have a job interacting with people
42%
50%
Follows what's going on in government and politics most of the time
42%
58%
How these voters are most similar
IPSOS
NYT/SIENA
PHONE
Gaps of three percentage points or less
Doesn't trust mainstream national media
57%
56%
50%
47%
Thinks most people can be trusted
Lived in current home for 10 years or more
49%
47%
44%
45%
Voted for Biden in 2020
Volunteered through or for an organization in the last 12 months
43%
40%
Prefers to have a job working with his or her hands
34%
31%
Not confident all votes will be counted accurately
30%
28%
26%
24%
Has an ATV
20%
23%
No doses of the Covid-19 vaccine
Thinks immigrants who are here illegally should be allowed to work but should not be given citizenship
21%
19%
Commented on or shared posts about political issues on social media in the last 30 days
17%
18%
14%
14%
Has a valid concealed carry license
Moved into current home one year ago or less
9%
6%
How Voters Reached by Mail — and Paid $25 — Are Different From Those Reached by Phone
While respondents who received $25 were significantly more likely to participate in a political survey, they were not meaningfully different in their political views or composition than a similar group of voters who did not receive money.
These voters have similar political preferences
IPSOS
NYT/SIENA
PHONE
Senate
44%
43%
Ron Johnson Republican
44%
45%
Mandela Barnes Democrat
Governor
42%
41%
Tim Michels Republican
46%
46%
Tony Evers Democrat
How these voters are most different
IPSOS
NYT/SIENA
PHONE
Gaps of more than three percentage points
Almost never answers calls on personal phone from non-recognized numbers
55%
40%
42%
30%
Identifies as politically moderate
Uses signs outside the home to deter strangers such as “No Trespassing” or has considered it
31%
20%
31%
20%
Has one to two doses of the Covid-19 vaccine
Thinks immigrants who are here illegally should be deported back to their home country
47%
38%
75%
68%
Born in Wisconsin
23%
17%
Has a valid hunting license
62%
56%
Is married
70%
64%
Doesn't trust major conservative news sources
44%
40%
Voted for Trump in 2020
19%
15%
Prefers to have a job working on a computer
Thinks immigrants who are here illegally should be allowed to work and should be given citizenship
30%
34%
59%
64%
Almost certain to vote in the November election
48%
54%
Has three or more doses of the Covid-19 vaccine
42%
50%
Prefers to have a job interacting with people
Follows what's going on in government and politics most of the time
42%
58%
How these voters are most similar
IPSOS
NYT/SIENA
PHONE
Gaps of three percentage points or less
57%
56%
Doesn't trust mainstream national media
50%
47%
Thinks most people can be trusted
49%
47%
Lived in current home for 10 years or more
44%
45%
Voted for Biden in 2020
Volunteered through or for an organization in the last 12 months
43%
40%
34%
31%
Prefers to have a job working with his or her hands
30%
28%
Not confident all votes will be counted accurately
26%
24%
Has an ATV
20%
23%
No doses of the Covid-19 vaccine
Thinks immigrants who are here illegally should be allowed to work but should not be given citizenship
21%
19%
Commented on or shared posts about political issues on social media in the last 30 days
17%
18%
14%
14%
Has a valid concealed carry license
9%
6%
Moved into current home one year ago or less
That said, an initial glance at the topline findings may be sobering for anyone who hoped that $25 and higher response rates would break through to reach the coveted “hidden” Trump vote. While there were important differences between the high- and low-incentive surveys — including some that hold promise for improving Times/Siena surveys and others going forward — there was not necessarily obvious evidence of a breakthrough to a vastly different pool of respondents.
While the mail survey showed the Republican incumbent, Ron Johnson, and the Democrat Mandela Barnes tied among registered voters — a tally that’s similar to the results of other polls — the raw, unadjusted respondents to the mail and telephone surveys both leaned Democratic by a considerable margin, including a six-point margin in the race for U.S. Senate in the high-incentive mail survey.
(It’s important to note that these poll results are “old” — both the mail-in surveys and the Times/Siena polls were conducted over a period of weeks, beginning in September. It would be a mistake to assume these results are representative of voters today.)
The results by demographic group were uncannily similar as well. White working-class voters — both before and after weighting — had nearly identical partisan preferences in the two surveys. Registered voters in Wisconsin’s Third District — the Obama-to-Trump district in the state — backed Mr. Barnes in both surveys.
The relatively small differences between the high-incentive mail survey and other polls could be framed as a good or bad thing for surveys — and it raises the polling stakes for today’s election.
On the one hand, the small to modest differences suggest that the Times/Siena poll and other low-incentive surveys remain competitive with a high-incentive survey with vastly higher response rates. On the other, it might be interpreted to mean that the “hidden Trump” vote remains out of reach — that $25 can’t reach a far more representative sample. (Of course, we won’t know which of these is true until the election is over.)
The Times/Ipsos mail study offered relatively little evidence to support some of the most popular theories about nonresponse bias, like social trust.
Surprisingly, compared with those on the phone survey, a slightly larger share of respondents in the mail survey — 50 percent — said people could be trusted. The mail respondents were also just as likely to say they volunteer in their communities.
The pandemic has also been cited as a possible explanation for the polling error in 2020, on the assumption that liberals staying home might have been likelier to take political surveys. But telephone survey respondents were more likely to say the pandemic didn’t change how often they left home. Mail respondents were only three percentage points likelier to say they never worked from home.
And maybe most surprising of all, the mail survey respondents were just as likely to say they trusted the mainstream media (interviewers do not introduce the Times/Siena poll as a New York Times survey) and even less likely than telephone survey respondents to say they trusted conservative media. Election denialism was just as common on the phone as it was in the mail poll.
In all of these cases, it is possible that neither the telephone nor the mail study broke through to a group of people who typically don’t respond. Maybe the people who wouldn’t respond to either survey are far less likely to trust the mainstream media, and simply continue to remain out of reach.
That said, there were some differences between the two surveys. The mail survey did show greater support for Republicans than the telephone survey did. The difference was usually fairly minor — only a point, or sometimes two. It was not usually particularly meaningful, but it was there.
And the differences were larger on other questions, which offer a portrait of the kind of voters who don’t respond to surveys and who may be underrepresented in political discourse.
Political moderation and disengagement
The high-incentive mail survey respondents were far less politically engaged, more moderate and less likely to vote than those who responded on the telephone.
This should not come as a huge surprise: Political surveys are surely more appealing to political junkies, who tend to vote and hold ideologically consistent views.
But the magnitude of the differences was striking: The share of mail respondents who said they were “moderate” was more than 12 points higher in the mail survey than in the Times/Siena poll, with both liberals and conservatives representing a smaller share of the telephone sample.
How we’re covering Election Day. The Times has more than 40 correspondents around the country reporting on the voting and any issues that may arise at the polls. Nearly every department at The Times contributes to our coverage, including the Graphics, Photo, Data and Video teams. David Halbfinger, our Politics editor, describes it as “a coast-to-coast, whole-of-the-newsroom, all-hands moment.”
Similarly, the share of people who said they were absolutely certain to vote in November was higher in the telephone poll than in the mail survey. There are countless such examples in the data, from how often they take surveys to how closely they follow the news.
Social engagement
Another major difference between the two surveys was how much respondents appeared to like engaging and interacting with strangers.
The people who took the telephone survey, for instance, were likelier to say they wanted to have a job that involved working with people, as opposed to working with their hands or on a computer.
And the voters who took the mail survey were likelier to own or say they considered getting a “no trespassing” or “no solicitation” sign. The respondents to the typical telephone survey were 10 percentage points likelier to say they hadn’t considered putting up such a sign; Republicans had a commanding lead among those who had.
Perhaps relatedly, the survey found a wide gap on preferences about immigration: 47 percent of respondents to the mail survey said undocumented immigrants should be deported back to their home country, compared with 38 percent in the Times/Siena poll.
It’s important to note that the greater incentive and higher response rate are not the only differences between the two surveys. The mail survey is self-administered; the telephone survey is conducted by a live interviewer. The presence of a live interviewer could have deterred a sliver of respondents from offering a potentially uncomfortable answer like support for deportation.
New registrants?
There was another set of differences that can’t easily be attributed to the divergence from an interview to a self-administered survey: those suggesting that the telephone survey reached a relatively mobile and unsettled group of people.
The mail survey had more people who were married, more people who were born in Wisconsin, and more people who had in-state area codes than Times/Siena respondents.
It’s a pattern consistent with pre-existing concerns about the kinds of voters who have cellphones available on voter registration files — a data set that’s the foundation of the Times/Siena poll. Almost by definition, the voters who provided their cellphones when they registered to vote did so over the last decade or so (someone who registered in 1992 almost certainly would not have had a cellphone to write down on the form). While the Times/Siena poll takes a variety of steps to account for this issue, the Ipsos survey results suggest it may not have been enough.
Whether differences over the number of out-of-state cellphone numbers would have helped shift the Times/Siena survey toward the final result is exactly the kind of question that will be addressed in the final analysis.
That analysis will begin soon after the final results in Wisconsin, which we’ll be watching closely tonight.
Ruth Igielnik and Alicia Parlapiano contributed reporting.
"still" - Google News
November 08, 2022 at 09:25PM
https://ift.tt/OVvC8lj
Are the Polls Still Missing ‘Hidden’ Republicans? Here’s What We’re Doing to Find Out. - The New York Times
"still" - Google News
https://ift.tt/AX8rS5x
https://ift.tt/FiU72xQ
Bagikan Berita Ini
0 Response to "Are the Polls Still Missing ‘Hidden’ Republicans? Here’s What We’re Doing to Find Out. - The New York Times"
Post a Comment