Abstract
Most U.S. voters cast hand-marked paper ballots that are counted by optical scanners. Deployed ballot scanners typically utilize simplistic mark-detection methods, based on comparing the measured intensity of target areas to preset thresholds, but this technique is known to sometimes misread “marginal” marks that deviate from ballot instructions. We investigate the feasibility of improving scanner accuracy using supervised learning. We train a convolutional neural network to classify various styles of marks extracted from a large corpus of voted ballots. This approach achieves higher accuracy than a naive intensity threshold while requiring far fewer ballots to undergo manual adjudication. It is robust to imperfect feature extraction, as may be experienced in ballots that lack timing marks, and efficient enough to be performed in real time using contemporary central-count scanner hardware.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bajcsy, A., Li-Baboud, Y.-S., Brady, M.: Systematic measurement of marginal mark types on voting ballots. Technical report (2015). https://nvlpubs.nist.gov/nistpubs/ir/2015/NIST.IR.8069.pdf. NIST
Bowen, D.: Top-to-Bottom Review of voting machines certified for use in California. Technical report (2007). https://www.sos.ca.gov/elections/voting-systems/oversight/top-bottom-review/. California Secretary of State
Bradski, G.: The OpenCV library. Dr. Dobb’s J. Softw. Tools (2000)
Clark, A.: Pillow (PIL Fork) Documentation (2015). https://buildmedia.readthedocs.org/media/pdf/pillow/latest/pillow.pdf
Colorado Secretary of State Elections Division: Voter Intent: Determination of Voter Intent for Colorado Elections (2013). https://www.broomfield.org/DocumentCenter/View/11702/Voter-Intent-Guide
Cordero, A., Ji, T., Tsai, A., Mowery, K., Wagner, D.: Efficient user-guided ballot image verification. In: Electronic Voting Technology/Workshop on Trustworthy Elections. EVT/WOTE (2010)
Curling v. Raffensperger, Civil Action No. 1:17-cv-2989-AT (N.D. Ga. Oct. 11, 2020)
Dominion Voting Systems: AuditMark. https://www.dominionvoting.com/democracy-suite-ems/
He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. CoRR abs/1512.03385 (2015). arXiv:1512.03385
Humboldt County: November 3, 2009 UDEL Election: Official Canvass Precinct Report. https://humboldtgov.org/DocumentCenter/View/3941/November-3-2009-UDEL-Election-Official-Canvass-Precinct-Report-PDF
Hursti, H.: Critical Security Issues with Diebold Optical Scan Design, The Black Box Report (2005)
Jones, D.W.: On optical mark-sense scanning. In: Chaum, D., et al. (eds.) Towards Trustworthy Elections. LNCS, vol. 6000, pp. 175–190. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-12980-3_10
Kamarck, E., Ibreak, Y., Powers, A., Stewart, C.: Voting by mail in a pandemic: a state-by-state scorecard, Brookings Institution (2020). https://www.brookings.edu/research/voting-by-mail-in-a-pandemic-a-state-by-state-scorecard/
Kiayias, A., Michel, L., Russell, A., Shvartsman, A.: Security assessment of the Diebold optical scan voting terminal (2006). https://voter.engr.uconn.edu/voter/wp-content/uploads/uconnreport-os.pdf
Kiayias, A., Michel, L., Russell, A., Shashidhar, N., See, A., Shvartsman, A.: An authentication and ballot layout attack against an optical scan voting terminal. In: USENIX/ACCURATE Electronic Voting Technology Workshop (EVT) (2007)
LeNail, A.: NN-SVG: publication-ready neural network architecture schematics. J. Open Source Softw. 4(33), 747 (2019). https://doi.org/10.21105/joss.00747
Lindeman, M., Stark, P.: A gentle introduction to risk-limiting audits. IEEE Secur. Priv. 10, 42–49 (2012)
McDaniel, P., Blaze, M., Vigna, G.: EVEREST: evaluation and validation of election-related equipment, standards and testing. Technical report (2007). http://siis.cse.psu.edu/everest.html. Ohio Secretary of State
Nagy, G., Lopresti, D., Smith, E.H.B., Wu, Z.: Characterizing challenged Minnesota ballots. In: 18th Document Recognition and Retrieval Conference (2011)
National Academies of Sciences, Engineering, and Medicine: Securing the Vote: Protecting American Democracy. The National Academies Press, Washington, DC (2018). https://www.nap.edu/catalog/25120/securing-the-vote-protecting-american-democracy
National Conference of State Legislatures: Post-Election Audits (2019). http://www.ncsl.org/research/elections-and-campaigns/post-election-audits635926066.aspx
Poulos, J., Hoover, J., Ikonomakis, N., Obradovic, G.: Marginal Marks with Pixel Count, U.S. Patent 9,710,988 B2 (2012)
Pueblo County Elections: Ballot Images, November 2020 Election. https://county.pueblo.org/clerk-and-recorder/ballot-images
Rivest, R.: On the notion of ‘software independence’ in voting systems. Phil. Trans. R. Soc. A 366(1881), 3759–3767 (2008)
State of Georgia: Report of The 21st Century Voting Commission, (2001). https://voterga.files.wordpress.com/2014/11/21st_century_report.pdf
Sultana, F., Sufian, A., Dutta, P.: Advancements in image classification using convolutional neural network. ICRCICN (2018). https://doi.org/10.1109/icrcicn.2018.8718718
Toledo, J.I., Cucurull, J., Puiggalí, J., Fornés, A., Lladós, J.: Document analysis techniques for automatic electoral document processing: a survey. In: Haenni, R., Koenig, R.E., Wikström, D. (eds.) VOTELID 2015. LNCS, vol. 9269, pp. 129–141. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-22270-7_8
U.S. Election Assistance Commission: Central Count Optical Scan Ballots (2008). https://www.eac.gov/sites/default/files/documentlibrary/files/Quick_Start_Guide_-_Central_Count_Optical_Scan_Ballots.pdf
Verified Voting: The Verifier: Polling Place Equipment (2021). https://www.verifiedvoting.org/verifier/
Wang, K., Kim, E., Carlini, N., Motyashov, I., Nguyen, D., Wagner, D.: Operator-assisted tabulation of optical scan ballots. In: Electronic Voting Technology/Workshop on Trustworthy Elections. EVT/WOTE (2012)
Xiu, P., Lopresti, D., Baird, H., Nagy, G., Smith, E.B.: Style-based ballot mark recognition. In: 10th International Conference on Document Analysis and Recognition (2009)
Acknowledgments
The authors thank Marilyn Marks and Harvie Branscomb for assistance acquiring ballot images and the students of EECS 498.5: Election Cybersecurity (Fall 2020) for their suggestions and feedback. We thank the Humboldt County Election Transparency Project and Pueblo County Elections for making ballot images available. We also thank our anonymous reviewers and our shepherd, Catalin Dragan. This material is based upon work supported by the Andrew Carnegie Fellowship, the U.S. National Science Foundation under grant number CNS-1518888, and a gift from Microsoft.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Barretto, S., Chown, W., Meyer, D., Soni, A., Tata, A., Halderman, J.A. (2021). Improving the Accuracy of Ballot Scanners Using Supervised Learning. In: Krimmer, R., et al. Electronic Voting. E-Vote-ID 2021. Lecture Notes in Computer Science(), vol 12900. Springer, Cham. https://doi.org/10.1007/978-3-030-86942-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-86942-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86941-0
Online ISBN: 978-3-030-86942-7
eBook Packages: Computer ScienceComputer Science (R0)