I have a long list of methodological issues with PPP polls. Today, the list goes onto a second page.
It turns out that PPP permits respondents who give “all 1s” to every question, which distorts their results.
Yesterday’s North Carolina survey brought signs of the problem. For some reason, possible Republican Senate candidates like Mark Harris, Heather Grant, and Greg Brennon received higher favorability ratings among “very liberal” voters than conservatives.
I asked PPP if they had an explanation for this phenomenon. They had an answer: “Looks like there were [a] dozen respondents pressing 1 on every question so they would be favorable toward everyone and very liberal.”
Is that possible? Yes. PPP is an automated polling firm, which means the respondent listens to an automated recording and responds by dialing a number ("if you’re 18-29 years old, press 1”). So it’s not impossible to imagine that a few respondents might decide to troll PPP by dialing “1” in response to every answer. For today’s North Carolina survey, those voters would be classified as white, 18-29 years old, female, very liberal, Democratic, supporting Kay Hagan for reelection, and having a “favorable” impression of every candidate—including the Republicans.
So PPP’s explanation lines up with the problem. The most obvious anomaly, after all, was a bunch of liberal Democrats having a favorable impression of some Republicans. The same anomaly shows up for 18-29 year olds, too—which is exactly what we’d expect.
At first glance, this might seem like a minor error. But it’s not. It doesn’t just influence favorability ratings, and it’s a bigger deal than “a dozen” respondents makes it seem.
Why? The “all 1” respondents are treated as 18-29 years olds—and 18-29 year olds get more weight. The need for weighting young voters is simple: Young voters have lower response rates, and, just for good measure, PPP doesn’t call voters with cell phones. In a randomly selected survey from October 2012, 18-29 year old voters were just 6.7 percent of PPP’s final sample—even after PPP randomly deleted old, white voters. To compensate, PPP weights its 18-29 year olds by two-fold—so PPP probably doubled the weight of these 12 “all 1” respondents. In a poll with a base of 600 respondents, that’s not small.
All in all, the trolls could have represented about 4 percent of PPP’s weighted sample—and they unanimously voted for Kay Hagan, who was also option “1.” Given that Hagan held a lead of no more than three points against three of her Republican challengers, it’s quite possible that the trolls were decisive.
If true, that’s pretty disturbing. Who knows the extent to which other PPP polls were adulterated by “all 1” trolls, who would systematically advantage the incumbent candidate (always the option for #1, according to PPP). When asked on Twitter whether the “all 1s” problem was “common,” PPP did not offer a response—disconcerting in its own right.
This is an amateurish error. PPP should be excluding these responses, and they surely know it. But on the other hand, it’s potentially an excusable error. It’s the type of problem that a polling firm could plausibly miss for a while, even if they couldn’t ignore the problem once they found out about it.
But PPP probably did know about the "all 1s" problem.
Aaron Strauss, former Director of Targeting and Data for the DCCC, alerted Daily Kos about the “all 1s” problem after DailyKos/SEIU released its first dataset in 2011:
On Mon, Jan 17, 2011 at 11:32 AM, <xxxx> wrote:
Kos (and PPP),
I think it is absolutely fantastic that you are releasing the raw data set of your polling -- thank you very much for that. One of the benefits of releasing the data is that the whole community can look at it and identify problems. For instance, in this most recent dataset (1/14-1/16), the respondent on row #771 (or #770 if you don't count the header) answered "1" to every single question. I suspect these are not actual opinions from the respondent, and I think answer sets like these should be filtered out in the future.
Also, while I have you: (1) are cell phone numbers filtered out of your lists so you don't accidentally violate the law by auto-dialing them? and (2) are the results weighted?
Thanks,
Aaron
On Twitter, Strauss wrote that he assumed PPP fixed this issue, since “Markos said he passed my bug report to PPP.” Strauss “figured it was an easy fix.”
But Strauss’ assumption isn’t necessarily wrong. Although it’s unclear whether they deliberately fixed the issue, they did so in effect by changing the question about age. Now, a “1” meant “under 18 years old,” not “18-29”
January 14, 2011: “If you are 18 to 29 years old, press 1. If 30 to 45, press 2. If 46 to 65, press 3. If you are older than 65, press 4.”
October 25, 2012: “If you are under 18 years old, press 1. If you are 18 to 29 years old, press 2. If 30 to 45, press 3. If 46 to 65, press 4. If older than 65, press 5.”
Presumably, PPP made this change for a reason. And regardless of why PPP made the change, it was a smart decision.
But apparently and inexplicably, PPP decided not to use the same age question for its non-DailyKos/SEIU surveys. PPP disclosed this on Twitter after I asked why (what I had assumed to be) their age question didn’t exclude the “all 1s.” And unfortunately, using different question for DailyKos looks suspiciously like using different questions when PPP's raw data is subject to public scrutiny.
This is also at least the second occasion where PPP has disclosed to me that their DailyKos/SEIU surveys were different from their normal surveys, which they previously said were the same. In both instances, the differences resulted from the questionnaire. PPP did not ask about the ’08 election in the DailyKos/SEIU polls, which PPP was surreptitiously using to weight its non-DailyKos/SEIU samples. And in both instances, the differences were sufficient to alter the composition of the sample.
Unlike the weighting by respondent’s reported vote in 2008, the “all 1s” problem should be easy to remedy: It's extremely hard to imagine that PPP would defend keeping the respondents. And yet PPP didn't fully act on Strauss' recommendation. Yesterday, PPP didn’t say they would be deleting the trolls in the future. They were unwilling to say whether the "all 1s" problem occurs frequently. And PPP didn't want to continue the conversation. After I asked them how much they weighted the 18-29 year old "all 1s," PPP responded by saying: "I don't really understand what you're asking. I have to go teach a class for the next two hours. Take care."
Disengagement is a mistake, because there are many unanswered questions in addition to the questions implicit in the critiques I've already offered. For instance, only 9 percent of 18-29 year olds think the ACA rollout was "very successful." If there were 12 "all 1s," I'd expect no fewer than about 14 percent of 18-29 year olds to get on board. So what's going on?
Altogether, PPP's decision to allow "all 1s" is negligence or worse, with the potential to systemically adulterate PPP's polls. Having little to say about the problem doesn't inspire confidence, either.
Image via shutterstock.com.