Machine learning research typically focuses on optimization and testing on a few criteria, but deployment in a public policy setting requires more. Technical and non-technical deployment issues get relatively little attention. However, for machine learning models to have real-world benefit and impact, effective deployment is crucial. In this case study, we describe our implementation of a machine learning early intervention system (EIS) for police officers in the Charlotte-Mecklenburg (North Carolina) and Metropolitan Nashville (Tennessee) Police Departments. The EIS identifies officers at high risk of having an adverse incident, such as an unjustified use of force or sustained complaint. We deployed the same code base at both departments, which have different underlying data sources and data structures. Deployment required us to solve several new problems, covering technical implementation, governance of the system, the cost to use the system, and trust in the system. In this paper we describe how we addressed and solved several of these challenges and provide guidance and a framework of important issues to consider for future deployments.

References

[1]

Xavier Amatriain and Justin Basilico. 2012. Netflix Recommendations: Beyond the 5 stars (Part 1). (2012). https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-1--55838468f429 {Online; accessed 10-February-2018}.

Google Scholar

[2]

Samuel Carton, Jennifer Helsby, Kenneth Joseph, Ayesha Mahmud, Youngsoo Park, Joe Walsh, Crystal Cody, CPT Patterson, Lauren Haynes, and Rayid Ghani. 2016. Identifying police officers at risk of adverse events Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 67--76.

Digital Library

Google Scholar

[3]

Crystal Cody, Estella Patterson, Putney Kerr, Jennifer Helsby, Joe Walsh, Rayid Ghani, Samuel Carton, Kenneth Joseph, Ayesha Mahmoud, and Youngsoo Park. 2016. Building Better Early Intervention Systems. Police Chief Magazine (2016).

Google Scholar

[4]

Wikipedia contributors. 2018. 2015 Texas pool party incident -- Wikipedia, The Free Encyclopedia. (2018). https://en.wikipedia.org/wiki/2015_Texas_pool_party_incident {Online; accessed 10-February-2018}.

Google Scholar

[5]

Tristan Crockett, Eric Potash, Jesse London, Matt Bauman, Erika Salomon, Avishek Kumar, Kit Rodolfa, Adolfo De Unanue, Tzu-Yun Lin, Klaus Ackermann, Rayid Ghani, Hannes Koenig, Andrea Navarette, and Benedict Kuester. 2018. Triage. https://github.com/dssg/triage/tree/v2.2.0https://github.com/dssg/triage/tree/v2.2.0. (2018).

Google Scholar

[6]

Jennifer Helsby, Samuel Carton, Kenneth Joseph, Ayesha Mahmud, Youngsoo Park, Andrea Navarrete, Klaus Ackermann, Joe Walsh, Lauren Haynes, Crystal Cody, et almbox. 2016. Early intervention systems: Predicting adverse interactions between police and the public. Criminal Justice Policy Review (2016), 0887403417695380.

Google Scholar

[7]

Mohit Kumar, Rayid Ghani, and Zhu-Song Mei. 2010. Data mining to predict and prevent errors in health insurance claims processing Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 65--74.

Digital Library

Google Scholar

[8]

Tzu-Yun Lin, Tristan Crockett, Adolfo De Unanue, and Rayid Ghani. 2017. Tyra. https://github.com/dssg/tyra/tree/v0.2. (2017).

Google Scholar

[9]

Andrew D Martin, Kevin M Quinn, Theodore W Ruger, and Pauline T Kim. 2004. Competing approaches to predicting supreme court decision making. Perspectives on Politics Vol. 2, 4 (2004), 761--767.

Crossref

Google Scholar

[10]

Netflix. 2009. Leaderboard. (2009). https://www.netflixprize.com/leaderboard.html {Online; accessed 10-February-2018}.

Google Scholar

[11]

Theodore W Ruger, Pauline T Kim, Andrew D Martin, and Kevin M Quinn. 2004. The Supreme Court forecasting project: Legal and political science approaches to predicting Supreme Court decision making. Columbia Law Review (2004), 1150--1210.

Google Scholar

[12]

D. Sculley, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, and Michael Young. 2014. Machine Learning: The High-Interest Credit Card of Technical Debt. (2014).

Google Scholar

[13]

Parikshit Shah, Akshay Soni, and Troy Chevalier. 2017. Online Ranking with Constraints: A Primal-Dual Algorithm and Applications to Web Traffic-Shaping. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 405--414.

Digital Library

Google Scholar

[14]

James Soland. 2013. Predicting high school graduation and college enrollment: Comparing early warning indicator data and teacher intuition. Journal of Education for Students Placed at Risk (JESPAR) Vol. 18, 3--4 (2013), 233--262.

Crossref

Google Scholar

Cited By

View all

Chor KLuo ZRodolfa KGhani R(2025)Exploring Machine Learning to Support Decision-Making for Placement Stabilization and Preservation in Child WelfareJournal of Child and Family Studies10.1007/s10826-024-02993-x34:1(282-297)Online publication date: 7-Jan-2025
https://doi.org/10.1007/s10826-024-02993-x
Hasan Chy MNana Buadi O(2024)Role of Machine Learning in Policy Making and EvaluationInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24OCT687(456-463)Online publication date: 19-Oct-2024
https://doi.org/10.38124/ijisrt/IJISRT24OCT687
Ananthi SB LM SS S(2024)Framework for Platform Independent Machine Learning (ML) Model Execution2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT)10.1109/IDCIoT59759.2024.10467931(728-732)Online publication date: 4-Jan-2024
https://doi.org/10.1109/IDCIoT59759.2024.10467931
Show More Cited By

Recommendations

Studying the Practices of Deploying Machine Learning Projects on Docker
EASE '22: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering

Docker is a containerization service that allows for convenient deployment of websites, databases, applications’ APIs, and machine learning (ML) models with a few lines of code. Studies have recently explored the use of Docker for deploying general ...
Deploying on the Grid with DeployWare
CCGRID '08: Proceedings of the 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid

In this paper, we present DeployWare to address the deployment of distributed and heterogeneous software systems on large scale infrastructures such as grids. Deployment of software systems on grids raises many challenges like 1) the complexity to take ...
Operationalizing machine learning models: a systematic literature review
SE4RAI '22: Proceedings of the 1st Workshop on Software Engineering for Responsible AI

Deploying machine learning (ML) models to production with the same level of rigor and automation as traditional software systems has shown itself to be a non-trivial task, requiring extra care and infrastructure to deal with the additional challenges. ...

Comments

Information & Contributors

Information

Published In

KDD '18: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

July 2018

2925 pages

ISBN:9781450355520

DOI:10.1145/3219819

General Chairs:
Yike Guo
Imperial College London
,
Faisal Farooq
IBM

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '18

Sponsor:

KDD '18: The 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 19 - 23, 2018

London, United Kingdom

Acceptance Rates

KDD '18 Paper Acceptance Rate 107 of 983 submissions, 11%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
1,477
Total Downloads

Downloads (Last 12 months)82
Downloads (Last 6 weeks)4

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Chor KLuo ZRodolfa KGhani R(2025)Exploring Machine Learning to Support Decision-Making for Placement Stabilization and Preservation in Child WelfareJournal of Child and Family Studies10.1007/s10826-024-02993-x34:1(282-297)Online publication date: 7-Jan-2025
https://doi.org/10.1007/s10826-024-02993-x
Hasan Chy MNana Buadi O(2024)Role of Machine Learning in Policy Making and EvaluationInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24OCT687(456-463)Online publication date: 19-Oct-2024
https://doi.org/10.38124/ijisrt/IJISRT24OCT687
Ananthi SB LM SS S(2024)Framework for Platform Independent Machine Learning (ML) Model Execution2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT)10.1109/IDCIoT59759.2024.10467931(728-732)Online publication date: 4-Jan-2024
https://doi.org/10.1109/IDCIoT59759.2024.10467931
O’Connor RBolton MSaeri AChan TPearson R(2024)Artificial intelligence and complex sustainability policy problems: translating promise into practicePolicy Design and Practice10.1080/25741292.2024.23488347:3(308-323)Online publication date: 7-May-2024
https://doi.org/10.1080/25741292.2024.2348834
Parekh POyeleke R(2024)Towards Enhancing Extended Reality for Healthcare Applications with Machine LearningAI Technologies and Virtual Reality10.1007/978-981-99-9018-4_23(313-330)Online publication date: 20-Mar-2024
https://doi.org/10.1007/978-981-99-9018-4_23
Chavan KAwade AMahadik SKalme GDeshpande K(2024)Ur’s Corporator—Innovating Citizen Corporator Interaction Through ML-Enabled Comprehensive FrameworkICT for Intelligent Systems10.1007/978-981-97-6675-8_16(187-198)Online publication date: 29-Oct-2024
https://doi.org/10.1007/978-981-97-6675-8_16
Ramalakshmi KAgarwal AGunasekaran HKhan SMandal N(2023)A Prototype for Machine Learning Model Deployment in Cloud Environment2023 International Conference on New Frontiers in Communication, Automation, Management and Security (ICCAMS)10.1109/ICCAMS60113.2023.10526026(1-4)Online publication date: 27-Oct-2023
https://doi.org/10.1109/ICCAMS60113.2023.10526026
Erden C(2022)Machine Learning Experiment Management With MLFlowEncyclopedia of Data Science and Machine Learning10.4018/978-1-7998-9220-5.ch071(1215-1234)Online publication date: 14-Oct-2022
https://doi.org/10.4018/978-1-7998-9220-5.ch071
Jiang ZHu JSamia AYu X(2022)Predicting Active Sites in Photocatalytic Degradation Process Using an Interpretable Molecular-Image Combined Convolutional Neural NetworkCatalysts10.3390/catal1207074612:7(746)Online publication date: 7-Jul-2022
https://doi.org/10.3390/catal12070746
Nisar HGupta DKumar PMurapaka SRajesh AUpadhyaya A(2022)Algorithmic Rural Road Planning in India: Constrained Capacities and Choices in Public SectorProceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization10.1145/3551624.3555299(1-11)Online publication date: 6-Oct-2022
https://dl.acm.org/doi/10.1145/3551624.3555299
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Recommendations

Studying the Practices of Deploying Machine Learning Projects on Docker

Deploying on the Grid with DeployWare

Operationalizing machine learning models: a systematic literature review

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations