What We Know About Bug Bounty Programs - An Exploratory Systematic Mapping Study

Abstract

This paper presents a systematic mapping study of the research on crowdsourced security vulnerability discovery. The aim is to identify aspects of bug bounty program (BBP) research that relate to product owners, the bug-hunting crowd or vulnerability markets. Based on 72 examined papers, we conclude that research has mainly been focused on the organisation of BBPs from the product owner perspective, but that aspects such as mechanisms of the white vulnerability market and incentives for bug hunting have also been addressed. With the increasing importance of cyber security, BBPs need more attention in order to be understood better. In particular, datasets from more diverse types of companies (e.g. safety-critical systems) should be added, as empirical studies are generally based on convenience sampled public data sets. Also, there is a need for more in-depth, qualitative studies in order to understand what drives bug hunters and product owners towards finding constructive ways of working together.

This research was funded by Swedish funding agency Vinnova, FFI program, HoliSec project (project number 2015-06894).

Appendix A

This appendix maps each publication included in the mapping study to the categories it was included in, product owner (PO), crowd (CR), and market mechanisms (MM).

Ref	Publication	PO	CR	MM
1	“Is There a Cost to Privacy Breaches? An Event Study”, Acquisti, A., Friedman, A., Telang, R	X
2	“Friendly Hackers to the Rescue: How Organizations Perceive Crowdsourced Vulnerability Discovery”, Al-Banna, M., Benatallah, B., Schlagwein, D., Bertino, E., Barukh, M.C	X
3	“Most successful vulnerability discoverers: Motivation and methods”, Algarni, A.M., Malaiya, Y.K		X
4	“Software Vulnerability Markets: Discoverers And Buyers, Algarni, A.M., Malaiya, Y.K		X
5	“Economic Factors of Vulnerability Trade and Exploitation”, Allodi, L			X
6	“Comparing Vulnerability Severity and Exploits Using Case-Control Studies”, Allodi, L., Massacci, F	X
7	“The Economics of Information Security”, Anderson, R., Moore, T	X		X
8	“Windows of vulnerability: a case study analysis”, Arbaugh, W.A., Fithen, W.L., McHugh, J	X
9	“0-Day Vulnerabilities and Cybercrime”, Armin, J., Foti, P. Cremonini, M			X
10	“Economics of software vulnerability disclosure”, Arora, A., Telang, R	X
11	“An Empirical Analysis of Software Vendors’ Patch Release Behavior: Impact of Vulnerability Disclosure”, Arora, A., Krishnan, R., Telang, R., Yang, Y.,	X
12	“Does information security attack frequency increase with vulnerability disclosure? An empirical analysis”, Arora, A., Nandkumar, A., Telang, R.,	X
13	“A Target to the Heart of the First Amendment: Government Endorsement of Responsible Disclosure as Unconstitutional”, Bergman, K			X
14	“Before We Knew It: An Empirical Study of Zero-day Attacks in the Real World”, Bilge, L., Dumitraş, T	X
15	“Vulnerability markets”, Böhme, R			X
16	“A Comparison of Market Approaches to Software Vulnerability Disclosure”, Böhme, R			X
17	“Enter the Hydra: Towards Principled Bug Bounties and Exploit-Resistant Smart Contracts”, Breindenbach, L., Daian, P., Tramer, F., Juels, A.,	X
18	“Efficiency of Vulnerability Disclosure Mechanisms to Disseminate Vulnerability Knowledge”, Cavusoglu, H., Cavusoglu, H. Raghunathan, S	X
19	“Cybersecurity Innovation in Government: A Case Study of U.S. Pentagon’s Vulnerability Reward Program”, Chatfield, A.T., Reddick, C.G	X
20	“Crowdsourced cybersecurity innovation: The case of the Pentagon’s vulnerability reward program”, Chatfield, A.T., Reddick, C.G	X
21	“Network Security: Vulnerabilities and Disclosure Policy”, Choi, Jay Pil; Fershtman, C., Gandal, N	X
22	“Vulnerabilities and their surrounding ethical questions: a code of ethics for the private sector”, De Gregorio, A		X
23	“Markets for zero-day exploits: ethics and implications”, Egelman, S., Herley, C., van Oorschot, P.C			X
24	“Private Ordering Shaping Cybersecurity Policy: The Case of Bug Bounties”, Elazari Bar On, A	X
25	“To Improve Cybersecurity, Think Like a Hacker”, Esteves, J., Ramalho, E., De Haro, G	X
26	“An Empirical Study of Vulnerability Rewards Programs”, Finifter, M., Akhawe, D., Wagner, D	X
27	“Vulnerability Disclosure: The Strange Case of Bret McDanel”, Freeman, E	X
28	“Web science challenges in researching bug bounties”, Fryer, H., Simperl, E., Fryer, H., Simperl, E	X
29	“Revenue Maximizing Markets for Zero-Day Exploits”, Guo, M., Hata, H., Babar, A			X
30	“Cyber vulnerability disclosure policies for the smart grid”, Hahn, A., Govindarasu, M	X
31	“Understanding the Heterogeneity of Contributors in Bug Bounty Programs”, Hata, H., Guo, M., Babar, M. A		X
32	“A study on Web security incidents in China by analyzing vulnerability disclosure platforms”, Huang, C., Liu, J., Fang, Y., Zuo, Z.,		X
33	“Shifting to Mobile: Network-based Empirical Study of Mobile Vulnerability Market”, Huang, K., Zhang, J., Tan, W., Feng, Z		X	X
34	“Defining and Assessing Quantitative Security Risk Measures Using Vulnerability Lifecycle and CVSS Metrics”, Joh, H., Malaiya, Y.K	X
35	“Economic analysis of the market for software vulnerability disclosure”, Kannan, K., Telang R			X
36	“Market for Software Vulnerabilities? Think Again”, Kannan, K., Telang, R			X
39	“Shifts in the Cybersecurity Paradigm: Zero-Day Exploits, Discourse, and Emerging Institutions”, Kuehn, A., Mueller, M			X
40	“Banishing Misaligned Incentives for Validating Reports in Bug-Bounty Platforms”, Laszka, A., Zhao, M., Grossklags, J	X
41	“The Rules of Engagement for Bug Bounty Programs”, Laszka, A., Zhao, M., Malbari, A., Grossklags, J	X
42	“An examination of private intermediaries’ roles in software vulnerabilities disclosure”, Li, P., Rao, H.R			X
43	“Economic solutions to improve cybersecurity of governments and smart cities via vulnerability markets”, Li, Z., Liao, Q			X
44	“Given enough eyeballs, all bugs are shallow? Revisiting Eric Raymond with bug bounty programs”, Maillart, T., Zhao, M., Grossklags, J., Chuang, J	X	X
45	“Software Vulnerability Disclosure and its Impact on Exploitation: An Empirical Study”, Mangalaraj, G.A., Raja, M.K	X
46	“Security-related vulnerability life cycle analysis”, Marconato, G. V., Nicomette, V., Kaâniche, M	X
47	“Hacking Speech: Informational Speech and the First Amendment”, Matwyshyn, A.M			X
48	“Stockpiling Zero-Day Exploits: The Next International Weapons Taboo”, Maxwell, P			X
49	“Are Vulnerability Disclosure Deadlines Justified?”, McQueen, M., Wright, J. L., Wellman, L	X
50	“Vulnerability Severity Scoring and Bounties: Why the Disconnect?”, Munaiah, N., Meneely, A			X
51	“The Attack of the Clones: A Study of the Impact of Shared Code on Vulnerability Patching”, Nappa, A., Johnson, R., Bilge, L., Caballero, J., Dumitras, T	X
52	“To disclose or not? An analysis of software user behavior”, Nizovtsev, D., Thursby, M.,	X	X
53	“An Assessment of Market Methods for Information Security Risk Management”, Pandey, P., Snekkenes, E.A			X
54	“Understanding Hidden Information Security Threats: The Vulnerability Black Market”, Radianti, J., Gonzalez, J.J			X
55	“Are Markets for Vulnerabilities Effective?”, Ransbotham, S., Mitra, S., Ramsey, J			X
56	“Is finding security holes a good idea?”, Rescorla, E	X
57	“Ethical Issues in E-Voting Security Analysis”, Robinson, D.G., Halderman, J.A		X
58	“Exploring the clustering of software vulnerability disclosure notifications across software vendors”, Ruohonen, J., Holvitie, J., Hyrynsalmi, S., Leppänen, V	X
59	“Trading exploits online: A preliminary case study”, Ruohonen, J., Hyrynsalmi, S., Leppänen, V			X
60	“A Bug Bounty Perspective on the Disclosure of Web Vulnerabilities”, Ruohonen, J., Allodi, L			X
61	“Current data security issues for financial services firms”, Sipes, E.K., James, J., Zetoony, D	X
62	“Economic Motivations for Software Bug Bounties”, Sprague, C., Wagner, J			X
63	“Identifying self-inflicted vulnerabilities: The operational implications of technology within U.S. combat systems”, Stevens, R	X
64	“Curbing the Market for Cyber Weapons”, Stockton, P.N.; Golabek-Goldman, M			X
65	“Doing What Is Right with Coordinated Vulnerability Disclosure”, Suárez, R.A., Scott, D	X
66	“Agents of responsibility in software vulnerability processes”, Takanen, A., Vuorijärvi, P., Laakso, M., Röning, J	X
67	“Impact of Software Vulnerability Announcements on the Market Value of Software Vendors - an Empirical Investigation”, Telang, R., Wattal, S	X
68	“Characterizing and Modeling Patching Practices of Industrial Control Systems”, Wang, B., Li, X., de Aguiar, L.P., Menasche, D.S., Shafiq, Z	X
69	“Ethics of the software vulnerabilities and exploits market”, Wolf, M.J., Fresco, N			X
70	“Evaluating CVSS Base Score Using Vulnerability Rewards Programs”, Younis, A., Malaiya, Y., Ray, I	X
71	“An Exploratory Study of White Hat Behaviors in a Web Vulnerability Disclosure Program”, Zhao, M., Grossklags, J., Chen, K	X	X
72	“An Empirical Study of Web Vulnerability Discovery Ecosystems”, Zhao, M., Grossklags, J., Liu, P		X
73	“Devising Effective Policies for Bug-Bounty Platforms and Security Vulnerability Discovery”, Zhao, M., Laszka, A., Grossklags, J	X
74	“Crowdsourced Security Vulnerability Discovery: Modeling and Organizing Bug-Bounty Programs”, Zhao, M., Laszka, A., Maillart, T., Grossklags, J	X	X