Fudging The Numbers

The summer after my freshman year of college, I worked part-time for a well-regarded polling outfit in Connecticut. For a few hours in the evening or on weekends, I would sit at a drab cubicle in a converted raised ranch house and stare at a computer screen. The computer would dial random numbers from the areas in which were were interested. For example, if we were polling New York State, the computer would select one of New York's zip codes, then one of the three-digit prefixes for that zip code, and finally choose four random numbers. So I might end up with 607-257-1234. With most calls, either the number would be disconnected or no one would pick up.

Since I never knew what kind of number my computer was calling, I had a few surprises when people did pick up. I talked to strip clubs, police and fire departments, the New York City Morgue, bizarre people, and plenty of pissed off regular people. I would make 250 calls in a typical night, but only end up with 3-6 (sometimes zero) completed surveys. To complete one poll, it might take 5 nights of calling, with 80 paid employees making calls on each of those nights.

My point is that a lot of expense, effort, and time was required to produce each poll. It would have been much easier for our bosses to have simply made up some numbers and released them to the press.

This, in fact, is what one polling company may have done.

It began with this:

AAPOR, the American Association for Public Opinion Research, yesterday reprimanded Atlanta-based PR firm Strategic Vision, LLC for failing to disclose even basic information about their political polls.

Nate Silver did some more research and compiled a devastating statistical analysis of Strategic Vision's polls, in which he cautiously accuses the company of making up numbers:

But this data is not random. It's not close to random. It's not close to close. Which brings up the other possibility: Strategic Vision is cooking the books. And whoever is doing so is doing a pretty sloppy job. They'd seem to have a strong, unconscious preference for numbers ending in '7', for instance, as opposed to those ending in '6'. They tend to go with round numbers that end in '5' or '0' slightly too often. And they much prefer numbers with high trailing digits like 49 and 38 to those with low ones like 51 and 42.

Strategic Vision has of course denied all of this, but they've also refused to release cross-tabs and other detailed data that might vindicate their polls. And tonight, we have evidence that "Atlanta-based" Strategic Vision may not even be located near Atlanta.

So what does all of this point mean? Strategic Vision may have tricked everyone. If they thought McCain was gaining on Obama in Pennsylvania before last fall's election, they may have taken an older poll and tightened the race by a couple of points. It's an effective strategy, and we should be impressed that they were able to get away with it for this long. But it's also extremely dishonest.

There's been a lot of debate recently about the future of polling. As more people shift from using land-lines to cell phones, public opinion polls are unable to reach an increasingly large percentage of the population. I just hope that the answer which emerges is not to simply make up numbers. While candidates tend to play down the significance of polls, they serve plenty of noble and scholarly uses. It'll be interesting to see where the Strategic Vision case goes. Check FiveThirtyEight for updates.