You are on the Natcen site

Click here for Scotcen

natcen map

You are on the Natcen site

Click here for Scotcen

natcen map

The Butterfly Effect: How Ballot Paper Designs Could Decide Elections

Posted on 04 November 2016 by Matt Shapley, Intern
Tags: QDT, Questionnaire Development and Testing Hub, cognitive interviewing, voting

“Like a jerk.” After voting in the 2000 American presidential election, this was the feeling of one voter in Palm Beach County, Florida when realising he had voted for Pat Buchanan, rather than preferred candidate, Al Gore. He was not alone. Within just one Palm Beach County precinct, 47 from over 1,700 voters indicated they had made the same mistake and, importantly, Buchanan received 3,400+ votes across the county where over 400,000 were cast.

Of course, normally, miscast or spoilt ballot papers do not decide elections. But after meticulous and lengthy recounts, George W. Bush was elected with a miniscule 154 vote margin over Al Gore in Florida which delivered victory overall, across the 50 states. One must ask: How many of Buchanan’s 3,400+ votes were intended for Gore? To what extent was the democratic process hindered? 


The choice of a ‘Butterfly’ ballot paper design used in the county was blamed for the miscast votes, featuring a response format for selecting candidates that was unclear for many voters. But, of course, poor designs are possible anywhere.

In the UK, the Electoral Commission is responsible for ensuring poor designs do not feature in elections, charged by parliament to support the running of a fair democracy. They asked the Questionnaire Development and Testing Hub (QDT) at NatCen to look at how voters interact with UK ballot papers. In particular, the QDT examined how party identifiers on ballot papers, such as names, slogans, and emblems, could affect a voter’s ability to correctly mark their preferences.

To do so, the QDT used cognitive interviewing methods, which explore all manner of thoughts and thought processes participants have when completing a task. QDT researchers worked with the City University Interaction Lab to use eye-tracking technology as a part of this testing, providing a quantitative element to an otherwise qualitative testing technique.   

In this laboratory setting, interviews simulated local and EU parliamentary elections where participants were shown a campaign flyer from a fictitious political party for which they were subsequently asked to vote. The ballot paper featured both real and fictitious political parties, and identifiers for candidates were designed to bare similarities to others on the ballot.  In particular, the type of similarity – i.e. similar names, slogans, and emblems – or combination of similarities was varied across interviews to isolate the particular confusions which could be caused.


The QDT recorded what people were thinking during the interviews using a combination of ‘think aloud’ and verbal probing techniques. Respondents were asked to verbalise their thoughts as they were completing their assigned task, and were later probed as to what elements of the ballot paper they looked at, how they decided to choose a particular party on the ballot paper, and so on.

To further aid the findings, the QDT analysed the gaze of a sub-group of participants using eye-tracking technology to record the specific areas of a ballot paper participants looked at, and the amount of time they spent looking there. This is illustrated by the ‘heat-maps’ below.

The QDT found that textual elements of a ballot paper - party and candidate names, abbreviations, or slogans - are more likely than pictorial elements to provide confusion for people. Eye-tracking showed that participants, on average, focussed on textual identifiers for 8.82 seconds and on pictorial identifiers for 2.61 seconds in the simulated local election. In the simulated EU election, this increased to 14.18 and 1.91 seconds respectively.

Of all identifiers, party name was found to be the major cause of miscast votes. When asked to vote for the ‘London Conservatives’ cognitive interviews showed that some mistook the party as a regional subsidiary of the Conservative Party and voted for the latter. Others mistook ‘Liberal Democrats’ for the intended ‘Liberals United’, focusing on the ideological word ‘Liberal’ and identifying the first ‘good’ match on their ballot paper for the intended party. And it came back to the issue of text even with pictorial issues; in the EU election one respondent confused the emblems of the ‘Nottingham Conservatives’ and the ‘Nottingham Communities’ parties due to the initials ‘NC’ featured in both, despite the former displaying a torch and the latter having a shaking hands emblem.

Of course, this type of analysis is only possible in laboratory settings, so the extent to which confusion over identifiers occurs in real-life elections is unclear. Moreover, as is normal in qualitative research, the sample was purposive, designed to provide a variety of ways of thinking when using ballot papers, rather than being statistically representative of the target population; the sample featured 69 participants of different previous voting behaviours, genders, ages, educational attainments, social grades, ethnicities, and regions.   

Regardless, this sample allowed for a detailed and comprehensive review of the ballot papers, and the findings reinforced the importance of pictorial and, in particular, textual design choices on ballot papers.

As we saw in the 2000 US election, voter confusion can occur if ballot papers are not designed effectively, and the Electoral Commission should keep these findings in mind when approving ballot papers in the future. Otherwise, who knows when the next butterfly will change the course of history?


Click here to find out more about NatCen's cognitive testing expertise.

comments powered by Disqus
Blog filters
  • 2023
  • 2022
  • 2021
  • 2020
  • 2019
  • 2018
  • 2017
  • 2016
  • 2015
  • 2014
  • 2013
  • 2012
  • 2011
  • 2010
  • Jan
  • Feb
  • Mar
  • Apr
  • May
  • Jun
  • Jul
  • Aug
  • Sep
  • Oct
  • Nov
  • Dec
Clear Filters

Subscribe to the RSS Feed: