From the beginning of this election season I have been highly skeptical of the odds Nate Silver has placed on various candidates winning various primaries.
Silver’s odds, and his secret sauce reasons for them, are now in question in multiple places.
I sent an Op-Ed on this idea to the New York Times in January. As expected, they would not touch it. Since then, evidence against Silver has mounted in what’s clearly a “late to party” phenomenon.
Last weekend I had a phone discussion with Salil Mehta, a top-selling mathematics and statistics book author, about mathematical discrepancies and other biases in Silver’s estimates.
Mehta, investigated further and agreed with me in a report.
Mathematicians Welcome
Let me state upfront, for the second or third time, that I think Silver is brilliant. His calls on prior elections have been amazing.
However, his latest projections have been negatively amazing. I also question his math. Since he does not answer emails, at least mine, let’s start with a look at the math. Mathematicians welcome to chime in.
Here is a series of Silver’s “Polls Only Projections” at the time I captured them. Some results may have changed if you look at his Five-Thirty-Eight website today.
Wisconsin Polls
Note that the single most relevant poll had a weighting of 1.04. The only other positively weighted poll weighed in at 0.02.
I do not question Silver’s weightings. I presume he knows far more about which polls are more relevant and why than I do.
I do question his math, and his undisclosed secret sauce.
Via the magic of Silver’s secret sauce, Silver comes up with this projection. Once again, that projection has changed since my capture.
Wisconsin Weighted Averages
I looked at the preceding snapshot and said WTF?
Here are my calculations:
| Wisconsin Polls | Weight | Percentage of Total Weight | Cruz | Trump | Kasich |
|---|---|---|---|---|---|
| Emerson Mar 20-22 | 1.04 | 0.981132075 | 36 | 35 | 19 |
| Marquette Feb 18-21 | 0.02 | 0.018867925 | 19 | 30 | 8 |
| Mish Totals | 1.06 | 1 | 35.67924528 | 34.90566038 | 18.79245283 |
| Silver Totals | 1.06 | 1 | 35.5 | 32.9 | 17.9 |
If we assume Silver is correct in his weights, the true weighted average of the poll weighted 1.04 and the second poll weighted 0.02, for Donald Trump is 34.9% not Silver’s 32.9%.
That is one of the things I spoke with Salil Mehta about. In addition, we discussed the odds of two 90+ percent failures in a short time span and other anomalies.
At first I assumed Silver missed reporting on a poll (now since out), but I found discrepancies in other states as well. Rather than post all of the Silver screen shots I captured I will just cut to the chase and show my tables.
New York
| New York Polls | Weight | Percentage of Total Weight | Cruz | Trump | Kasich |
|---|---|---|---|---|---|
| Emerson Mar 14-16 | 0.58 | 0.828571429 | 12 | 64 | 1 |
| Sienna Feb 28 – Mar 3 | 0.12 | 0.171428571 | 11 | 45 | 18 |
| Mish Totals | 0.70 | 1 | 11.82857143 | 60.74285714 | 3.914285714 |
| Silver Totals | 0.70 | 1 | 11.6 | 58.8 | 2.7 |
Based on true weighted odds, accepting Silver’s weights, I calculate Trump’s odds of winning New York at 60.7%. Silver says they are (were at the time) 58.8%
California
| California Polls | Weight | Percentage of Total Weight | Cruz | Trump | Kasich |
|---|---|---|---|---|---|
| NSON Mar 9-10 | 0.67 | 0.35078534 | 22 | 38 | 20 |
| PPIC Mar 6-15 | 0.62 | 0.32460733 | 27 | 38 | 14 |
| Smith Mar 2-9 | 0.59 | 0.308900524 | 20 | 25 | 15 |
| Field Dec 16-Jan 3 | 0.03 | 0.015706806 | 25 | 23 | 1 |
| Mish Totals | 1.91 | 1 | 23.05235602 | 33.7486911 | 16.20942408 |
| Silver Totals | 1.91 | 1 | 22.4 | 32.5 | 15.9 |
Silver’s Polls Only Projections, Not Polls Only
In every instance that I have investigated, Trump is on the losing end of these calculations.
Even Paul Krugman is commenting on “Tarnished Silver“.
In a discussion with Mehta regarding geometric weightings and other possibilities, neither of us could come up with an explanation other than what I label “secret sauce”.
In other words, Silver’s “Polls Only” projections are not polls only.
If you want another opinion, please consider Salil Mehta’s Mercurial Pollster Results.
New York Times Chimes In
Today the New York Times chimed in with How the G.O.P. Elite Lost Its Voters to Donald Trump.
Lovely.
I sent the New York Times an article on “attitudes” and Silver’s projections in January. Today, this concept is supposedly “news”.
The sorry thing about all this is one must give the Times 10 business days to respond.
Media Submission Recommendations
Here is my suggestion: Don’t bother with submissions unless you are some high profile person, in which case, no matter how inane your theory, the Times is likely to use your article.
To prove I am not biased against the New York Times, the same applies to the Financial Times, Bloomberg, and the Wall Street Journal, all of whom I have made timely submissions on various topics.
Your best hope of being published in mainstream media is
- Toe the line
- Have a big name (in which it does not matter how stupid a thing you write)
- Write an op-ed agreeing with something the media and big names agree with
The more timely your idea, the more you should consider sending it to me, ZeroHedge, or anyone else but mainstream media.
Mike “Mish” Shedlock
Dear Mish
Until you see checks from the CIA in the mail you are not in the club.
Derf
As I mentioned in the last Mish article on Nate Silver, he has a good point that Nate ought to do a better explanation of how he gets to his numbers for us to trust them.
Mish seems to have it in his head that he has to discredit Nate’s numbers. This third (that I’ve come across) one is unnecessary and is a red herring.
Look, it seems rather petty to argue about a small percentage difference when it is an inexact process to begin with, especially when we don’t know the exact calculations made by Nate Silver to get his results.
Why focus on the minutiae of someone else’s assessment of the polling that shows Trump likely falling behind in Wisconsin (we now know that is true).
Forget Nate Silver, and make the case for Trump.
There is so much other baggage with Trump, and he changes his positions so frequently, it is a wonder how anyone can trust that Trump is any less a “warmonger” than the other candidates.
Discuss that, in light of several of his recently adopted positions, such as his new (and unconvincing, bumbling) pro-life stance.
Oh Mike, the NY Times is an INSTITUTION. It exists to serve its own ends, not yours, not anyone’s. For that matter, the MSM is an institution of the highest order — the fourth estate or somesuch.
Truth is the enemy of institutions, and you must be a member of their club for your existence to be acknowledged.
But know that you have a loyal readership of intelligent truth-seekers with the ability to think and read critically, most of whom probably read Pravda On The Hudson as a contrarian indicator or purely for laughs.
When I first came to the Untied States from London, I was astonished at the utter bullshit I read in the NYT.
It’s like government economic statistics:
The bankers know the loan numbers are bullshit but assume the rest are correct,
The realtors know the home sale numbers are bullshit but assume the rest are correct,
The manufacturers know the production numbers are bullshit but assume the rest are correct,
And so on ad infinitum.
Anybody who is expert in any subject will deride the NYC’s coverage of the topic.
The NY Times bought the Boston Globe for 1.1 billion and sold it for 70 million.
They give Keynesian Paul Krugman who has declared personal bankrupcy a national bully pulpit to dispense his lunatic economic and political views.
I don’t expect to hear the reasoned opinions of Mish Shedlock to be published in the Old Gray Lady anytime soon.
Having said that, please don’t stop trying.
I wish Krugman had really declared bankruptcy but that was in a satire.
Reminds me of The Onion article “14 People die of marijuana overdose the first day of Colorado legalization!” Some idiot sheriff somewhere saw that and said “I told you so!”
Perfect …………
Mish, you and I can also add to, ZeroHedge as well, you guys are so great to bring out the reality in the conspiracy world……. Driven by one party member (the US Army) regardless who is in power ……
Has ZeroHedge wrote in one of his articles quote: Martin Armstrong Asks “When Will Trump Be Assassinated?” http://www.zerohedge.com/news/2016-03-29/martin-armstrong-asks-when-will-trump-be-assassinated
Thank you for the truth ………
NYTimes paid subscriptions are down to 600,000 in a city of 8 million. NYTimes paid subscriptions are declining 6% per year. In twenty years NYTimes paid subscribers will be 175,000. NYTimes is a parochial paper except for unread third world government subscriptions around the world. Integrity is nil. Accuracy is less.
On Wisconsin: The Marquette poll is about 6 weeks old. Much has happened since, including the departure of Carson, Bush and Rubio. Note, the total in the Marquette poll is 57%; the total in the Emerson poll is 90%. In other words, the Marquette poll is 1) outdated and 2) not reflective of the movement of respondents/voters to the remaining candidates. Siliver is reflecting this fact.
If Silver has a secret sauce, it is measurement of momentum, or apparent change. Note that Trump has gained 5% between the polls, Cruz 17%, Kasich 11%. Sounds about right to me.
Richard
Who said:
Give me a fulcrum and a place to stand and I shall move the world. ?
The Internet is the ultimate lever. Mere mortals like Mish (and Charles Hugh Smith, and Paul Kraig Roberts, and Steve Keen) can threaten to topple powerful institutions like the Fed and the NYT. The Gods themselves covet such powers. Fascinating.
I think it was Archimedes.
On the rare occasions that I read a newspaper anymore, I am always disgusted by the slanted fictions in the articles I read. I do not feel the least bit sorry the printed news is losing readership; they deserve it. 99% of my news source is the internet now. You just have to know when it is worthwhile reality, and when it is fictional BS. It seems conservatives and liberals both try to sway opinions with propaganda.
“It seems conservatives and liberals both try to sway opinions with propaganda.”
Not seems, it is absolute fact.
Both wings have their conspiracy theories. Both wings have their scripted talking points. Both wings have their cheer leading. Where is the truth buried in the midst of all that?
The two wings of the one bird of prey
I’m guessing Sliver’s previous success has gone to his head. Maybe now he thinks he can influence the results with his predictions.
“…Silver is brilliant. His calls on prior elections have been amazing.”
Once you make it in the polling business as independent you can contract you services to political operatives. Silver may have deal with some anti-Trump people. Or maybe he just doesn’t want Trump to be the nominee for whatever reason. But, what about his reputation if he’s off? As things get close to the election, Silver come get back to reality and cement his (good) reputation for the next one. And there are many races besides the Presidency.
Per Occam, Silver is exactly as brilliant (Relative to his peers. All professional statisticians are well above average at the stuff, compared to Joe Blow and Toothless Tony) as any other two time lottery winner.
Elections don’t occur often enough for anyone to deduce any more than that from his calls. And Vegas has plenty of baseball odds makers with track records just as good. Some better, some worse, but again, nothing particularly meaningful.
In general, popularity contests (like democracy) are a pretty poor process for determining “brilliance.” Or anything else of lasting value, for that matter.
Several “important” polls have a serious construction error with respect to Trump. (The NBC/WSJ Marist Republican Primary National Poll is an offender.)
They reject for inclusion a responder who did not vote in at least one of the last two Republican Primaries. So they reject for inclusion many of the most enthusiastic Trump voters … you know, the “I’m as mad as hell, and I’m not going to take this anymore” Trumpistas who stand in lines for hours and hours to attend a Trump event but who have never voted Republican before and/or have not voted for many years.
How any sane pollster can assume that someone will stand in line all day to see Trump but will not spend ten minutes to vote for him is beyond passing strange.
It is probably a form of bias.
H
It makes no sense for Silver to tarnish his reputation by skewing poll results for a particular outcome. Remember that the final reality check is the actual primary results. The “secret sauce” may be the fact that Trump’s election results sometimes underperforms the pre-primary polling and that is taken into account. In any event, are we really quibbling about 2 percentage points with noisy poll data?
Silver took a 1 percentage point difference in polls and concluded Cruz had a 60% chance of winning Wisconsin.
Then in his POLLs Plus Forecast projected an 81% chance Cruz would win.
Ridiculous. I failed to post the slide but did so earlier here.
http://mishtalk.com/2016/03/23/nate-silvers-incredulous-wisconsin-primary-odds/
“It is probably a form of bias.”
It is a form of bias. The whole is the sum of its parts. It is deliberate bias to exclude those who are voting for the first time, as that skews the result of the poll to be unfavorable toward the person most of those first time voters are signing up to vote for.
The media narrative has gone from Trump is a clown and will fall on his face by X date, to who can stop Trump? I don’t remember an election where the media narrative was, Who can stop (candidates name here) over and over again. Some of these politicians embed themselves in office for 20 years. The media never asks what candidate can stop McCain or Feinstein from being re-elected for the umpteenth time.
Apparently, last night, on a CNN town hall interview, Trump said he would no longer stand by his pledge to support the republican candidate. That would be understandable, considering the republican party had opened all the stops in an attempt to prevent him from being the party nominee.
The republican nominee is supposed to be Goldman Sachs and no one else. The democrat nominee is supposed to be Goldman Sachs and no one else. Ted Cruz got secret loan(s) and Hillary Clinton was paid $675,000 by Goldman Sachs to give secret speeches.
Goldman Sachs referred to their clients as Muppets.
So there it is: Goldman Sucks is the Wizard Behind the Curtain, pulling the levers, pushing the buttons. If The Donald is elected, hopefully he will have a grudge against GS.
Pingback: Links 3/30/16 | naked capitalism
Looks for Silver, after cashing his checks, to pull a Greenspan with a ‘flaw’ statement, similar to: “Flaw in the model that I perceived is the critical functioning structure that defines how the world works, so to speak.”
Bullshit. How Greenspan is not in jail amazes me. Corzine, Clinton, now Silver ruining his reputation for some untraceable loot.
Disgusting.
And the Fourth Estate? Never to be trusted again. Pure Agenda Driven Entertainment.
I read you, ZeroHedge, and George Ure in the morning to get the latest world crazy news. I find many news worthy information that won’t ever make my locl news outlets, and apparently much of the M$M.
My only complaint with ZH is that much of the commenters, say 2 out of 3 posts, are inane & stupid. The worst offender, imo, is the one constantly posting about it’s the jews fault. In. Every. Posted. Article. Like I care about fault now; It’s the consequences that interest me.
Mish, I suspect Silver’s error stems from the fact that he’s using normal filters in abnormal times:
Imagine standing on a street corner in Munich or Berlin, prior to the 1932 election, and eavesdropping on political conversations.
It’s quite likely that many of those chatting would have been expressing political views at large variance with the politically correct polls of the time. They got those views, mostly, listening to a little Austrian house painter/wingnut named Adolf Hitler…on radio.
Hitler didn’t get a majority, just enough to be picked as the best bubble in the urinal by Hindenburg, as allowed by the Weimar Constitution, because he thought he could “handle” Hitler. Right.
Fast forward to today and the names, places and costumes have changed, but the script. looks suspiciously similar to me, right down to the poll numbers.
Plus, he has the web and all the pat answers: It’s the Mexicans/Muslims/Chinese, etc….not us
Mish, I suspect Silver’s error stems from the fact that he’s using normal filters in abnormal times:
Exactly!
His projections on Trump’s odds were based on the “historical” fact that party-supported candidates win nominations. In January, he still had Trump’s odds in the single digits.
He missed a big attitude shift in that people did not care if Trumpo said stupid things, they liked the fact he was the candidate who spoke his mind. I mentioned the angry-white men idea. Mainstream media juts now caught on.
Mish
Nate Silver adjusts the polls only for “house effects” too. I don’t know if he does that in the primary, but in the general his polls only average isn’t a straight weighting on the poll numbers, it is poll numbers + adjustment for how biased previous polls by the same pollster were towards one party (or candidate). IE, he adjusts for polls where the error is always in favor of the republican or democrat, or in favor of one specific candidate.
I understand the adjustments for house effects but that should be in the weights. Also I find it incredible in the case of Wisconsin that he took essentially one poll and weeks ahead of the primary concluded something like a 60% chance for Cruz even though that poll was 36% to 35%. Lots of things don’t add up here.
I was in a rush last night posting that. I wish I had showed that slide. But I did show it once before.
Mish
Mish,
Nate Siver quotes the probability that Cruz wins – not the part of the vote he’s predicted to gain.
Indeed, with a sample size of 439 and a probability of succes of 36/(35+36)=.507 , the binomial law gives a probability of more than 439/2 successes as 61.6%, close enough to Silver’s
Olivier
Hi Mish,
Truly enjoy your blog and was excited to see you picked up on this. I have calc’d the weighted polling-only averages on 538 for the Democratic primary off and on over the last few weeks. The Democratic side also seems to have errors fairly frequently, typically in favor of Clinton. In general, I’ve noticed the following issues with not only his “models” but also the commentary on the site…
– New Primary Polls not added when they come out. In comparison, Real Clear Politics always has new polls added quickly.
– Number of polls used significantly lower than RCP. The argument could be made that they are eliminating “bad” polls, but this could simply be addressed with weighting
– Weighted average off frequently as detailed above
– Silver and other contributors tend to hedge their model results with comments. Model will have a 90% probability but during their live posts of a primary Silver will say the he wouldn’t be surprised if the other candidate does significantly better than polls and may even win. Doing so allows him refer back to that comment if there is an “upset”.
In general, it’s amazing to me that a site that prides itself on data analysis doesn’t seem to update the data or even calculate their projections correctly. They managed to write-off Trump, miss the surge of Bernie Sanders, and continue to highlight their predictions rather than analyzing the results in a way that shed light on political currents.
Never fails, Nate is giving himself an out again…
“natesilver: The polls have tended to underrate Clinton in the South and underrate Bernie in the North. Which would be a good sign for Sanders in Wisconsin. I suspect the polls are having a lot of trouble picking up enough black voters and young voters, which are the two most decisive groups in the Democratic primary.
With all that said, the contrarian devil on my shoulder is saying that since it seems obvious to everyone that Sanders will beat his polls in Wisconsin, maybe we’re heading for a Clinton upset instead.”
Mish,
I’m not a serious mathematician, but is it possible that in addition to the stated weightings, Silver is taking the size of potential error into account in his weightings. That is to say, your weighted averages are based on point estimates. But the Silver estimates may take into account the fact that some polls have much smaller sample sizes and therefore wider potential errors. That might swing the average probabilities by the differences you show.
Many excellent comments to this thread
Thanks to all
Mish
You are not alone in not getting replies from Silver.
If you want a laugh, read some of Ben Casselman’s articles on 538. He’s the “chief economics writer”.
He’s more like Captain Obvious.
I haven’t figured out if he’s a journalist writing about economics or vice versa.
But then again, 538 is pretty much an establishment bunch.
Nate Silver was hailed for correctly predicting all fifty state outcomes after the 2012 elections; but almost no one noted that a crude, across-the-board prediction of ‘no change’–if a state went Democratic in 2008, it would do the same in 2012–would have scored 48/50, which suggests that the excited exclamations of ‘He called all 50!’ were a tad overwrought. I commend Superforecasting, by Tetlock and Gardner, for some good reading about predictions in general. As for the current primary season, in addition to all the information you present, I fell off the Silver bandwagon when he predicted a 99% probability of a Clinton victory in Michigan, and, of course, Sanders won.
There is no secret sauce or conspiracy here. Nate’s model is well documented here:
http://fivethirtyeight.com/features/how-we-are-forecasting-the-2016-presidential-primary-election/
The weights you see are based on :
1. Time decay – The previous poll in wisconsin was conducted on Feb 18-21 – thats a life time in primary season – its a surprise that the poll has any validity at all.
2. House effects – where polls consistently show a bias towards one candidate.
One issue in primary races is that there is just not enough data to extrapolate. Wisconsin for example had 1 poll in a month until recently.
I think you are confusing Nate Silver the journalist with Nate Silver the mathematician. The former is writing articles that express his “opinion” and “gut feel” based on empirical data – and is about at accurate as any other journalist making predictions. The latter is showing you models which are fairly accurate if you account for sparse datasets.
In any case more polls on Wisconsin are painting the same picture now…which is not surprising after Rubio’s drop out
Pingback: Neo-Committee On UnAmerican Activities – Fake Russian News – Military Votes Count