I really like 538 in general. It’s a fun website, the writers don’t seem to take themselves all that seriously, and they do a decent job making quantitative stuff accessible-ish to non-quant readers. I might occasionally nitpick the accuracy of their descriptions, but I understand that they’re not writing for stat-geeks, they’re trying to make quantitative stuff accessible to more people, and that’s an admirable pursuit.
As far as polling goes - I think Nate Silver is an incredibly smart guy, but I also think he (slightly) oversells the magic of his models. On the flip side, though, I think he does a pretty reasonable job noting the inherent uncertainty of said models; the problem (and I see this every day in my own specialty as well) is that most people are uncomfortable with uncertainty and probabilities.
Models are designed to project probabilities so we can make informed guesses about what might happen. If Nate Silver’s model projects an 85 percent chance that Hilary will win going into Election Day, sure, he would be projecting a Clinton victory. But if Trump ends up winning, that doesn’t mean Nate Silver’s model was actually wrong. The model still gave Trump about a 1-in-7 chance of winning. And I think Silver always does insert appropriate disclaimers; again, the problem is often readers who race past such disclaimers, discarding any nuanced thought in the rush to come away with one single answer.
For example, this article…
…is full of little gems. He goes through several individual polls, compares them against their own previous versions, speculates about potential reasons for subtle changes in each poll and variation between the polls. It’s great stuff. And yet, if he’s projecting Clinton at 85 percent to win a week out but Trump ultimately wins, the conclusion for some people will be “LOL, Nate Silver and his geeks were wrong, math is stupid.” Even though Silver has (correctly) hedged:
“So then: Democrats have nothing to worry about, right? Nope, we wouldn’t say that, either. The race could easily tighten further. And our forecast gives Trump better odds than most other models because it accounts for the possibility of a systemic polling error, a greater risk than people may assume. A 16 percent chance of a Trump presidency isn’t nothing — as we’ve pointed out before, it’s about the same as the chances of losing a “game” of Russian roulette.”
This one, too…
“So although the wide spread in the polls this year may reflect challenges in the polling industry, you shouldn’t make a habit of berating the polls that seem to be outliers or use a somewhat unorthodox methodology. Aggregating mechanisms like polling averages and betting markets are powerful precisely because they reflect a diverse array of approaches and opinions, and they lose their power when they’re subject to herding or groupthink. You also shouldn’t cherry-pick the two or three polls that fit your narrative when there are dozens of them out there, of course. But ignoring or “debunking” the polls you don’t like is often almost as bad a sin as cherry-picking the ones you do like.”