On the eve of the French Presidential election (and doubtless to be repeated in June for the UK General Election), Nate Silver's hypothesis that pollsters are herding -- finding ways to be close to the polling average rather than where a pure application of polling methodology would take them -- is getting a lot of prominence.
There's one aspect of this theory that does not get enough scrutiny: it's not just a claim that pollsters herd close to the average, but they are more likely to do as the election date approaches. Key question: how close is the "end?" Consider for example the application by Léopold Mebazaa of Silver's hypothesis to the French polls. He uses a cutoff date of 25 February to determine whether the polls started to converge, on the basis that there were some big moves around that time (e.g. the Fillon indictment) that altered the race. But Le Monde notes that the analysis does not depend on that date -- you could pick a wide variety of dates and the polls would appear to be converging. That suggests that there's more than one thing going on.
For the purposes of this post, we want to zero in on the choice of date being based on assessment that big things were happening in the campaign around that time. How do we know they were big things? Because the polls moved! Thus the analysis is claiming that the polls became less variable after a date when they had responded a lot to news, which raises questions of when pollsters would be chasing the average and when they wouldn't -- after all, if each pollster knows of a "big news" event, why would be trying to stay close to a previous average?
Thus, the herding claim rests on a circularity that is inherent to the way Silver analyzes polls -- it says nothing about which kinds of news causes a move in the polls. You can analyze poll numbers to the Nth degree, but if you don't have a theory of how the public is filtering news into a poll response, it will only take you so far.
UPDATE: The herding claim doesn't fare well. The polls were converging to a very accurate prediction of the outcome!
There's one aspect of this theory that does not get enough scrutiny: it's not just a claim that pollsters herd close to the average, but they are more likely to do as the election date approaches. Key question: how close is the "end?" Consider for example the application by Léopold Mebazaa of Silver's hypothesis to the French polls. He uses a cutoff date of 25 February to determine whether the polls started to converge, on the basis that there were some big moves around that time (e.g. the Fillon indictment) that altered the race. But Le Monde notes that the analysis does not depend on that date -- you could pick a wide variety of dates and the polls would appear to be converging. That suggests that there's more than one thing going on.
For the purposes of this post, we want to zero in on the choice of date being based on assessment that big things were happening in the campaign around that time. How do we know they were big things? Because the polls moved! Thus the analysis is claiming that the polls became less variable after a date when they had responded a lot to news, which raises questions of when pollsters would be chasing the average and when they wouldn't -- after all, if each pollster knows of a "big news" event, why would be trying to stay close to a previous average?
Thus, the herding claim rests on a circularity that is inherent to the way Silver analyzes polls -- it says nothing about which kinds of news causes a move in the polls. You can analyze poll numbers to the Nth degree, but if you don't have a theory of how the public is filtering news into a poll response, it will only take you so far.
UPDATE: The herding claim doesn't fare well. The polls were converging to a very accurate prediction of the outcome!
No comments:
Post a Comment