Here’s why you should view polling results on ballot questions differently than those on races between candidates

NINE BALLOT QUESTIONS have turned voters’ ballots into booklets this fall, with questions ranging from marijuana policy to voting rules to zoning reform. Not since 1994 have Massachusetts voters had the opportunity (or burden) to decide so many questions. In recent months, we’ve seen some or all of the nine questions appearing in pre-election polls, including the poll that CommonWealth Beacon released earlier this month.

So what to make of these polls? At one level, 8 of the 9 questions have shown a clear edge for the “yes” side at this stage, which would seem to suggest they are favored to pass. If these were polls of candidate races, it would be reasonable to conclude that the candidates favored by more respondents have a better chance of winning than their opponents.

While we’ve been conditioned for years to read “horse race” polling in that way, we should not apply the same lens used in candidate races to ballot question polls. Doing so gives a far too rosy a picture of each question’s likelihood of passing.

That’s because undecided voters break late and toward the “no” side when it comes to ballot questions, meaning an ostensible lead in the polls for the yes side is more tenuous than it might appear, especially when that lead involves less than majority support. The current “lead” for the yes side is actually far less predictive than just looking at the percentage who say they will vote “yes.” Viewed this way, the outcomes for questions 2 through 7 are much less clear based on polling from MPG and Suffolk University. Each shows an ostensible lead for the “yes” side, but “yes” vote also are hovering in the danger zone of the 40s or low 50s.

Political consultants and others have noticed and noted this dynamic, and cautioned ballot question proponents against becoming overconfident unless polling shows their side is favored by a healthy majority, though that advice is often lost when ballot question polls are discussed in public.

The precise statistics to support this view, however, have not been available before now. To provide a more rigorous test of this bit of conventional wisdom, we plumbed the depths of the Massachusetts polling archives, compiling a database of publicly released poll results from every ballot question survey we could find from the last eight cycles of even-year elections (2010-2024).

Our search turned up 129 instances of ballot questions included in a public survey. In building the list, if the same poll had 2 or 3 (or 9) questions on it, each result is entered separately.

Using this newly created database, we put the insider knowledge to the test, with very clear results. First and foremost, the level of the “yes” vote in a poll is indeed much more predictive of the ultimate result than the “yes” versus “no” gap.

The results of ballot question polling typically include three main numbers: the percentage who say they would vote support the question or vote “yes,” the percentage who say they would vote “no,” and the percentage of undecided voters.

In our analysis of these three responses in ballot question polling over these eight election cycles, the percentage who say they will vote “yes” is the most closely related to the actual election results. Put another way, if a group of polls finds about 45 percent say they will vote yes, the actual yes vote is generally close to that estimate, regardless of whether the pre-election poll showed the yes side leading or trailing the no side.

One way to look at this is measuring the size of the “error” in each poll, or how different the poll was from what ended up happening. Using that approach, as shown in the table below, polls taken at any point in the pre-election cycle were, on average, 4.6 points from the final “yes” vote, but a whopping 11.6 points away from the final yes-versus-no gap. This includes all polls done anytime from January of an election year up through Election Day. The gap improves somewhat as a predictor of the actual outcome as Election Day draws nearer but never comes close to the “yes” percentage in terms of accuracy.

These error figures also allow us to look at the polls to see if they are skewed, or biased, in a specific direction. Accuracy tells us how far the polls were from what they were trying to measure. Bias tells us if they are consistently off in one direction, or if the direction of the errors are more or less random.

Applying this lens shows the margin in ballot question polls tends to be both large and biased toward the “yes” side. In other words, analyzing the gap between the “yes” and “no” side would give you an overly rosy view of the “yes” side’s prospects by an average of 8.5 points.

The reasons for this bias is that voters who report being undecided earlier in the election cycle move overwhelmingly to the “no” side when they actually cast their ballots. So polls undercount the “no” voters who just haven’t made their minds up yet. This bias persists in polls taken right up until Election Day, though it does get smaller in the last few weeks of a campaign.

This late break is because voting “no” on a ballot question almost always preserves the status quo, making no change to current law. Ballot questions often cover arcane or esoteric topics, which can be difficult or impossible for the average voter to become expert on with the time they allot to deciding how to vote. For these voters, “no” often becomes the default choice, and “no” campaigns are often built around introducing a variety of specific doubts – just enough to make voters nervous about the proposed change.

So what does this mean about this year and the record-tying nine ballot questions voters are being asked to decide? Looking at the MPG and Suffolk polls shows which questions seem to be on the bubble and which offer a clearer picture of what outcome is most likely.

We’re now right in the middle of the time when voters are making up their mind. If history is any guide, we’ll see more and more “no” voters as Election Day approaches.

Steve Koczela is president of the MassINC Polling Group. Janice Hagerman is a research associate at MPG.

Courtesy of Commonwealth Beacon