I’ve written at length about PPP’s troubling methodological choices. The firm failed to disclose important methodological decisions, offered inconsistent or baffling explanations, and continues to employ an unscientific and inconsistent approach. Altogether, it’s difficult to distinguish PPP’s polling from weighting toward an intended result. Now the question is: “how should we use PPP going forward?”
It’s a tough question.
On the one hand, PPP’s approach makes it difficult to distinguish the preferences of its sample from PPP’s assumptions about the outcome of the election. For example, if PPP shows a shift from a tie to Obama+5, it’s not always clear whether that’s because PPP’s sample was better for Obama, or perhaps because there was a shift in PPP’s thinking about the outcome. The same questions exist when PPP’s polls seem consistent. Consider a few recent Georgia polls, where PPP showed the same result, but with two unjustifiably different electorates. If PPP had treated its samples similarly, the results would have been different, very different.
The problem becomes even clearer when you look at PPP’s results over the long-term. There are systemic patterns in seemingly random changes in the composition of PPP’s electorate. The overall effect is stabilizing, and moves PPP’s results closer to the average of all polls.
There’s a simple reason why PPP uses this unusual (to be generous) approach: They wouldn’t be accurate otherwise. PPP’s samples are astonishingly unrepresentative. They don’t get voters with a cell phone, so they miss young and non-white voters. They don’t conduct interviews in Spanish, so they miss the most Democratic-leaning Hispanic voters.
You can see it in the cross-tabs. PPP’s surveys systemically underestimate the president’s support among Hispanics—they only showed Obama ahead by an average of 26 points nationally, compared to Obama’s 44 point victory in the exit polls and 52 point margin in the final Latino Decisions survey. PPP’s most recent national survey only had 18-29 year olds at 9 percent of “voters”—they’re getting so few young voters that they can’t even “weight” 18-29 year olds up to their actual share of the electorate without breaking their own rules for weighting.
To be oblique: There’s a reason PPP is trying to figure out how to include cell phone users in its surveys. To be blunt: The small number of non-white and young voters renders their polling inaccurate, and they know it. In the interim, they’ll play around with the weights until the final result looks right. I’m cautiously hopeful that incorporating cell phone voters will improve PPP’s samples to the point that they don’t need to engage in these distorting weighting practices.
On the other hand, these methodological quirks only move the needle so far. Weighting within wide “target ranges” gives PPP the power to fiddle with the result, but perhaps only by about 6 points at most—and mainly in racially polarized state like Georgia. And in the narrowest yet most important sense, PPP is still conducting a “survey”—they’re actually calling voters, who are actually giving them real responses. So if PPP puts out a survey showing a close race, it’s pretty likely that the race is relatively close. And conversely, if PPP puts out a survey showing a blowout, the race is probably a blowout.
So what’s the solution? Treat PPP like a relatively inaccurate pollster. Like Rasmussen, PPP's numbers aren’t irrelevant: if they show a 20 point lead somewhere, the race probably isn’t close; if they show a close race, the contest might be competitive. But when it comes to the details, the result is mainly a reflection of PPP’s assumptions. If PPP’s assumptions were consistent and well-understood, that would be acceptable if still contestable. The problem is that the assumptions are in constant flux—so much so that they can’t really be understood as “assumptions” at all. Instead, weighting looks more like a tool that PPP uses to get the result it wants. At the very least, PPP’s stated explanation of its own approach is highly consistent with that possibility.