Who knew GOP would romp? 'Nerdy analysts' had a sense

A "powerful wave."

"Democrats got drubbed."

"Democalypse 2014."

OK, that last one comes from The Daily Show With Jon Stewart. But humor aside, postelection reports reflect a widespread notion: Republicans took Democrats by surprise Tuesday as they claimed control of the Senate, built their largest House majority since Herbert Hoover's presidency, and racked up wins in statehouse after statehouse.

Was it really an earthshaking shock? It shouldn't have been, at least to anyone paying attention to a handful of nerdy prognosticators who, in recent years, have transformed the way Americans read and interpret preelection polls.

Consider Joni Ernst's Senate victory in Iowa, where she will replace Democrat Tom Harkin. Years ago, conflicting election-eve polls - one from the Des Moines Register showing Ernst up by 7 percentage points, another from Quinnipiac University showing her in a dead heat with Democrat Bruce Braley - would have left traditional pundits befuddled.

But on Tuesday, Ernst did just what these nerdy experts had predicted: She won Iowa by 8.5 percentage points - consistent with the forecasts of seven leading models, though a larger margin than any foresaw.

Even as Tuesday's results also showed the limits of the analysts' craft, the outcomes again demonstrated the power of poll aggregation and "meta-analysis," the statistical tools used by experts such as Nate Silver, the onetime baseball-stats geek who founded FiveThirtyEight.com, and Sam Wang, a neuroscientist who takes breaks from research to run the Princeton Election Consortium.

Just as in 2012, the meta-analysts took much of the guesswork out of Election Day outcomes. The irony is that this year, they showed just what they didn't in 2012, when Mitt Romney supporters wanted to see the preelection polls as skewed: systemic polling error favoring Democrats.

Wang says he and his fellow analysts fell victim to "the midterm polling curse" - the historical unreliability of polls in midterm elections, known for their low and unevenly distributed turnout. "In midterm elections, polling biases are much more common and much larger than in presidential years," Wang told me Wednesday.

This wasn't, I should note, something Wang offered with the aid of 20-20 hindsight. Weeks before the election, Wang advised readers to consider his predictions cautiously because of a pattern of systemic error - in either direction - that can show up.

This year's meta-analysis star - in a game scored only in hindsight - was statistician and political scientist Drew Linzer, who warned readers of the liberal Daily Kos website that the Republicans had a 90 percent shot at Senate control. He had this year's best Brier score - an aggregate measure of his predictions.

Even Linzer couldn't predict the magnitude of a GOP victory, which apparently looked smaller in preelection polls because of that systemic bias favoring Democrats - a kind of error pollsters constantly strive to avoid.

How does Wang explain this year's systemic failure, which he says led to Republicans' outperforming his projections by an average of about 5 percentage points, and outperforming Silver's numbers by an average of 4 points?

His best guess, he says, is what the Cook Report's David Wasserman called an "epic turnout collapse" for less-motivated Democrats, which Wasserman saw especially in states lacking top-ticket contests.

Wang says one of the most surprising results - the near-tie in Virginia, where Democrat Mark Warner likely faces a recount - was in a state with high levels of ideological and demographic diversity. Both those factors probably confounded turnout predictions.

The basic limitation of meta-analysis is that it's reliant on the quality of the underlying polls and susceptible to that time-honored computer-science principle called "garbage in, garbage out."

The polls weren't garbage, of course, but politicians and analysts have only just begun to decipher how and why they missed. Meanwhile, meta-analysis is here to stay.