On Polls' Methods, Bloggers' Methods, and other nefarious means of prediction
There's a beautiful article in today's National Post entitled "Rae, the Liberal Front-Runner?", which echoes sentiments heard in the blogosphere for a few days. The article kicks the metaphorical stuffing out of professional pollsters' predictions, such as Dryden being the leader, Rae being the leader, and Ignatieff being down in 3rd place.
By contrast, most bloggers' predictions were much more on the mark. Some were flippant guesses. Others were exceedingly well reasoned (e.g. Greg's at democraticspace.com). But virtually all of them got the basics right: Ignatieff in the lead in the high 20's; Rae, Kennedy, and Dion in the 10-20% range, not too far apart; everyone else a distant fourth. They might have got things wrong in terms of the degree (e.g. underestimating Ignatieff and Dion, overestimating Brison), but they were certainly much more correct that the professional pollsters' predictions.
Now, there's nothing wrong necessarily with the pollsters' predictions. The problem is one of interpretation. Ken Dryden may be the most popular and electable of the Liberal leadership candidates; that poll was probably completely correct. But that electability and popularity completely failed to translate into votes on the ground because it ignores reality and how the DSMs actually work.
Had the DSMs simply been a strawpoll of all Liberal members, maybe Dryden would be in the lead (and I'm not even sure of this, as I point out below). But it's not. DSMs involve having a large organizational team to identify and get the vote out, ex officios to build support, the money to send literature to your potential delegates and other voters, and more. Those are among the true measures of success at the DSMs. While candidate popularity is up there too (obviously), it's a necessary, but by no means a sufficient, condition for success.
But even if all those factors were irrelevant and it were more of a straw poll DSM, I'm still not convinced that Dryden could have come out on top. Individuals' responses in polls such as these often indicate their preference in an abstract ideal world. When push comes to shove, they might vote for a very different candidate, possibly for strategic reasons (i.e. their primary preference isn't for a candidate, it's against a candidate), possibly for perceptions of general electability, or possibly as a result of some kind of herd mentality phenomenon.
In any event, the point of the above is that while polls are useful indicators, those interpreting polls need to be careful not to attach too much significance to them or to read too much into them. Polls may show popularity, but this is only relevant for success in an election to the extent that sheer popularity is relevant.
Now, here's where I have a bone to pick with pollsters. The media and partisan bloggers will always pick up on a certain poll, for either the media value or to support their candidate, respectively. You'd expect some kind of spin or deliberate error to achieve those ends. And even if you didn't expect that, you might chalk up errors to a lack of understanding of what the polls are actually saying. But you should never expect that from professional pollsters. Insofar as some of them have used pre-DSM polling data to predict convention outcomes, they should feel ashamed of themselves, as that's just poor reasoning and use of statistics.
So, that's it for bashing pollsters and pointing out their limitations. Why did bloggers get it right?
Quite, frankly, bloggers got it right because they focused on process.
Most bloggers, in trying to determine DSM support, considered the factors which might lead to success, and came up with some formula based on these factors. These factors included membership sign-ups, ex-officio support, financial contributions, polls, and more. The weightings were somewhat arbitrary, but in the absence of PhD level research and econometric analysis, they were the best that could be expected.
Thus, the bloggers were successful because they asked, "what does it take to do well at the DSMs?", rather than just assuming it was a straight-up popularity contest.
This importance ascribed to process is something that professional pollsters and the media should pay attention to. Until they do, all the complex survey methodology, boostrapping, and asymptotic theory in the world can't save them.
By contrast, most bloggers' predictions were much more on the mark. Some were flippant guesses. Others were exceedingly well reasoned (e.g. Greg's at democraticspace.com). But virtually all of them got the basics right: Ignatieff in the lead in the high 20's; Rae, Kennedy, and Dion in the 10-20% range, not too far apart; everyone else a distant fourth. They might have got things wrong in terms of the degree (e.g. underestimating Ignatieff and Dion, overestimating Brison), but they were certainly much more correct that the professional pollsters' predictions.
Now, there's nothing wrong necessarily with the pollsters' predictions. The problem is one of interpretation. Ken Dryden may be the most popular and electable of the Liberal leadership candidates; that poll was probably completely correct. But that electability and popularity completely failed to translate into votes on the ground because it ignores reality and how the DSMs actually work.
Had the DSMs simply been a strawpoll of all Liberal members, maybe Dryden would be in the lead (and I'm not even sure of this, as I point out below). But it's not. DSMs involve having a large organizational team to identify and get the vote out, ex officios to build support, the money to send literature to your potential delegates and other voters, and more. Those are among the true measures of success at the DSMs. While candidate popularity is up there too (obviously), it's a necessary, but by no means a sufficient, condition for success.
But even if all those factors were irrelevant and it were more of a straw poll DSM, I'm still not convinced that Dryden could have come out on top. Individuals' responses in polls such as these often indicate their preference in an abstract ideal world. When push comes to shove, they might vote for a very different candidate, possibly for strategic reasons (i.e. their primary preference isn't for a candidate, it's against a candidate), possibly for perceptions of general electability, or possibly as a result of some kind of herd mentality phenomenon.
In any event, the point of the above is that while polls are useful indicators, those interpreting polls need to be careful not to attach too much significance to them or to read too much into them. Polls may show popularity, but this is only relevant for success in an election to the extent that sheer popularity is relevant.
Now, here's where I have a bone to pick with pollsters. The media and partisan bloggers will always pick up on a certain poll, for either the media value or to support their candidate, respectively. You'd expect some kind of spin or deliberate error to achieve those ends. And even if you didn't expect that, you might chalk up errors to a lack of understanding of what the polls are actually saying. But you should never expect that from professional pollsters. Insofar as some of them have used pre-DSM polling data to predict convention outcomes, they should feel ashamed of themselves, as that's just poor reasoning and use of statistics.
So, that's it for bashing pollsters and pointing out their limitations. Why did bloggers get it right?
Quite, frankly, bloggers got it right because they focused on process.
Most bloggers, in trying to determine DSM support, considered the factors which might lead to success, and came up with some formula based on these factors. These factors included membership sign-ups, ex-officio support, financial contributions, polls, and more. The weightings were somewhat arbitrary, but in the absence of PhD level research and econometric analysis, they were the best that could be expected.
Thus, the bloggers were successful because they asked, "what does it take to do well at the DSMs?", rather than just assuming it was a straight-up popularity contest.
This importance ascribed to process is something that professional pollsters and the media should pay attention to. Until they do, all the complex survey methodology, boostrapping, and asymptotic theory in the world can't save them.
posted by Anonymous Liberal at 8:35 AM
0 Comments:
Post a Comment
<< Home