The Problem with Polls: Jill Lepore Strikes Again

Professor Lepore

In an era of identify theft (when people are reluctant to reveal any personal information), with cell phones replacing landlines, with caller ID, the task of people trying to do political polling is daunting. If you want to know how the affluent white female over 50 crowd sees Donald Trump, you have to ask a lot of personal questions, that is, if you can get her to pick up her home landline which has caller ID and you can't use robo calling for her cell phone, because that's illegal, so just getting people to talk to you and once you have, getting them to divulge their demographics is met with suspicion.

As Jill Lepore notes in her wonderful New Yorker piece on polling, in the 1930's the response rate to polls was over 90%--people who were asked to participate in opinion polls said yes over 90% of the time. Today that rate of response is in the single digits. One might ask, can you really get a representative sample of what people are thinking and how they are likely to behave--i.e, will they go to the polls and vote--when 95% of people won't talk to you. 

What does it say about those people who do respond? Are they not "atypical" to begin with, as witnessed by their very willingness to be interviewed and are you not basing your idea of "public opinion" on a cohort which is, ipso facto, atypical?

There are basically two types of polls:  1. Opinion polls which aim to tell what the public thinks about issues  2. Election polls which hope to predict the outcome of elections. 

Watching on Election night, I have often been stunned by how well the numbers models seem to work:  If Obama gets more than 30% in this particular part of this particular county in Pennsylvania, you can call the state for him when less than 20% of voting centers have finished their count in Pennsylvania.  Magic. Big numbers. How do they do that?  Of course, that sort of analysis is pretty distinct and different from polls, but still, it suggests social and political scientists might actually know what they are talking about.

On the other hand,  I always wince when I hear some politico say, "Well, Romney is not doing well with white women," or "Obama has a problem with white men."   The fact is, there are likely other fellow travelers with that demographic designation, "white men": like Southern, high school educated, gun owning, church going white men might not be fertile ground for Obama. But you didn't need a poll to tell you that.

On the other hand, canvassing in Hampton, New Hampshire, I met as many blue collar white gun-owning men who contradicted that conventional wisdom, who voiced support for Obama,  as I met those who fulfilled it. 

And talking to gun owners and gun enthusiasts, I find a great range of belief about the place of guns in a free society--Hillary Clinton may have tapped into that by saying gun owners should leave the NRA which does not represent them. She understands there are nuances in opinions beyond the "you will have to pry my gun from my cold dead fingers" crowd.

Watching West Wing and every morning show, polls are loosely quoted by people who haven't the faintest idea how those polls were conducted--by phone, by land line, by internet, by door to door? They don't even know exactly what question was asked.

People who take a poll by landline phone own landline phones which many young people do not.  The landliners tend to be older, and more conservative. 

People who respond to a poll voluntarily on the internet are not, by definition, randomly selected; they are self selected and tend to be more extreme.

Professor Lepore notes that in 1947 Herbert Blumer said public opinion simply does not exist, absent its measurement. "Pollsters proceed from the assumption that "public opinion" is an aggregation of individual opinions, each given equal weight." And this is something of an absurdity, because you are trying to make a single white woman with an income over $100K stand for all single affluent white women and she will depart from that cohort in all sorts of ways.

This is an old argument, really, not just in sociology, but in anthropology, where Ruth Bennedict and Margaret Mead once argued you could know all about an individual's values and beliefs if you simply knew the values of the culture in which he was raised, but it turned out each individual differed significantly from the norm for that group, when you actually got down to individual cases.

I see this all the time in my office, as people voice support for Mr. Obama's desire to keep our armies off Middle Eastern soil, support for unfettered gun rights, animosity toward shooters who kill patients at Planned Parenthood while expressing a horror of abortion.  An individual is often all over the map. How does a poll capture that?
People quote Rush Limbaugh without even knowing the source of their opinion.  How fluid that opinion may be, how it might affect the way they vote, if they vote, is an unknown.




In my world, I look at studies every week in the medical journals and the hard part of analysis of the study is the Methods section:   figuring out the details of what they did and how the approach they used may produce a false results. We are not given that sort of detail in most polls and certainly, Donald Trump doesn't want to know any of that, only that he won.

The problem for our political system is not there are polls; the problem is everyone from the candidates to the candidates' staffs, to the pundits, (even David Brooks and Mark Shields) read the conclusions rather than the Methods sections, so they don't really understand how limited the information they are holding up really is.