Abstract
White hat hackers, also called ethical hackers, who find and report vulnerabilities to bug bounty programs have become a significant part of today’s security ecosystem. While the efforts of white hats contribute to heightened levels of security at the participating organizations, the white hats’ participation needs to be carefully managed to balance risks with anticipated benefits. One way, taken by organizations, to manage bug bounty programs is to create rules that aim to regulate the behavior of white hats, but also bind these organizations to certain actions (e.g., level of bounty payments). To the best of our knowledge, no research exists that studies the content of these program rules and their impact on the effectiveness of bug bounty programs.
We collected and analyzed the rules of 111 bounty programs on a major bug bounty platform, HackerOne. We qualitatively study the contents of these rules to determine a taxonomy of statements governing the expected behavior of white hats and organizations. We also report specific examples of rules to illustrate their reach and diversity across programs. We further engage in a quantitative analysis by pairing the findings of the analysis of the program rules with a second dataset about the performance of the same bug bounty programs, and conducting statistical analyses to evaluate the impact of program rules on program outcomes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We used the Python difflib implementation of the Ratcliff/Obershelp matching algorithm [20] with parameter 0.9.
- 2.
HackerOne’s Privacy Policy (https://www.hackerone.com/privacy/) states as a general policy that “we welcome minors to submit reports to HackerOne.” However, the site is not directed at minors below 13 who would need to have their parents/guardians submit vulnerability reports and to set up an account.
- 3.
HackerOne’s Vulnerability Disclosure Guidelines (https://www.hackerone.com/ disclosure-guidelines/).
- 4.
Since not all programs disclose their average bounty, we have to restrict our analysis to 58 data points in this subsection.
- 5.
A lower value of Alexa rank represents a more popular website. For example, an Alexa rank of 1 indicates the most-visited website.
- 6.
Note that we use data from the entire history of each bug bounty program. We have also tested the models using only data available after the last major rule update of each program. The regression analysis shows the same directionality of effects, but the dataset is much smaller to report a robust analysis.
References
Algarni, A., Malaiya, Y.: Software vulnerability markets: discoverers and buyers. Int. J. Comput. Inf. Sci. Eng. 8(3), 71–81 (2014)
Bacon, D., Chen, Y., Parkes, D., Rao, M.: A market-based approach to software evolution. In: 24th ACM SIGPLAN Conference Companion on Object Oriented Programming, Systems, Languages, and Applications (2009)
Böhme, R.: A comparison of market approaches to software vulnerability disclosure. In: Müller, G. (ed.) ETRICS 2006. LNCS, vol. 3995, pp. 298–311. Springer, Heidelberg (2006). https://doi.org/10.1007/11766155_21
Bozorgi, M., Saul, L., Savage, S., Voelker, G.: Beyond heuristics: learning to classify vulnerabilities and predict exploits. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 105–114 (2010)
Bugcrowd: The state of Bug Bounty, July 2015
Bugcrowd: The state of Bug Bounty, June 2016
Clark, S., Frei, S., Blaze, M., Smith, J.: Familiarity breeds contempt: the honeymoon effect and the role of legacy code in zero-day vulnerabilities. In: Proceedings of the 26th Annual Computer Security Applications Conference (ACSAC), pp. 251–260 (2010)
Edmundson, A., Holtkamp, B., Rivera, E., Finifter, M., Mettler, A., Wagner, D.: An empirical study on the effectiveness of security code review. In: Jürjens, J., Livshits, B., Scandariato, R. (eds.) ESSoS 2013. LNCS, vol. 7781, pp. 197–212. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-36563-8_14
Finifter, M., Akhawe, D., Wagner, D.: An empirical study of vulnerability rewards programs. In: USENIX Security Symposium (2013)
Flesch, R.: A new readability yardstick. J. Appl. Psychol. 1948(32), 221–233 (1948)
Huang, K., Siegel, M., Madnick, S., Li, X., Feng, Z.: Poster: diversity or concentration? Hackers’ strategy for working across multiple bug bounty programs. In: 37th IEEE Symposium on Security and Privacy (S&P) (2016)
Kuehn, A., Mueller, M.: Analyzing bug bounty programs: an institutional perspective on the economics of software vulnerabilities. In: TPRC Conference Paper (2014)
Laszka, A., Zhao, M., Grossklags, J.: Banishing misaligned incentives for validating reports in bug-bounty platforms. In: Askoxylakis, I., Ioannidis, S., Katsikas, S., Meadows, C. (eds.) ESORICS 2016. LNCS, vol. 9879, pp. 161–178. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45741-3_9
Laszka, A., Zhao, M., Grossklags, J.: Devising effective economic policies for bug-bounty platforms and security vulnerability discovery. J. Inf. Policy 7, 372–418 (2017)
Maillart, T., Zhao, M., Grossklags, J., Chuang, J.: Given enough eyeballs, all bugs are shallow? Revisiting Eric Raymond with bug bounty markets. J. Cybersecur. 3, 81–90 (2017)
Mc Laughlin, H.: SMOG grading - a new readability formula. J. Reading 12(8), 639–646 (1969)
Ozment, A.: The likelihood of vulnerability rediscovery and the social utility of vulnerability hunting. In: Workshop on the Economics of Information Security (WEIS) (2005)
Ozment, A., Schechter, S.: Milk or wine: does software security improve with age? In: USENIX Security Symposium (2006)
Ransbotham, S., Mitra, S., Ramsey, J.: Are markets for vulnerabilities effective? MIS Q. 36(1), 43–64 (2012)
Ratcliff, J., Metzener, D.: Pattern-matching: the gestalt approach. Dr Dobbs J. 13(7), 46 (1988)
Rescorla, E.: Is finding security holes a good idea? IEEE Secur. Priv. 3(1), 14–19 (2005)
Senter, R., Smith, E.: Automated readability index. Technical report, DTIC document (1967)
Shahzad, M., Shafiq, M., Liu, A.: A large scale exploratory analysis of software vulnerability life cycles. In: International Conference on Software Engineering (2012)
Zhao, M., Grossklags, J., Chen, K.: An exploratory study of white hat behaviors in a web vulnerability disclosure program. In: 2014 ACM CCS Workshop on Security Information Workers (2014)
Zhao, M., Grossklags, J., Liu, P.: An empirical study of web vulnerability discovery ecosystems. In: 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS) (2015)
Zhao, M., Laszka, A., Maillart, T., Grossklags, J.: Crowdsourced security vulnerability discovery: modeling and organizing bug-bounty programs. In: HCOMP Workshop on Mathematical Foundations of Human Computation (2016)
Acknowledgment
We thank the anonymous reviewers for their comments. The research activities of Jens Grossklags are supported by the German Institute for Trust and Safety on the Internet (DIVSI).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
A Example Bug Bounty Rule Statements
We list example rule statements below:
1. “Please report serious vulnerabilities in our website (https://staging.factlink.com), proxy (https://staging.fct.li), or other components (our annotation library, Wordpress plugin, browser extensions, and gems)” (Factlink)
2. “Please note that Binary.com’s front-end code is open-sourced at [...] - please feel free to report any vulnerabilities found in this code by submitting a pull-request in github” (Binary)
3. “Not in scope: shopify.asia, go.shopify.com and investors.shopify. com are operated by third parties, and are not in scope” (Shopify)
4. “Any Sucuri customer website are out of the scope of this disclosure program” (Sucuri)
5. “Mattel websites and services are owned and operated by Mattel and are explicitly outside the scope of this bug bounty program” (ToyTalk)
6. “Please only use our staging environments for testing, they are otherwise identical to production” (Factlink)
7. “In general, anything which has the potential for financial loss or data breach is of sufficient severity, including: XSS, CSRF, Authentication bypass or privilege escalation, Click jacking, Remote code execution, Obtaining user information, Accounting errors” (Coinbase)
8. “We are generally not interested in DoS vulnerabilities that are perceived by a lack of rate-limiting or captcha. As a web-scale service, our threshold for rate limiting is higher than you would probably expect. Of course, if you think you have found an exception to this rule, please let us know” (Automattic)
9. “Please create a free account and (pen) test away our GhostMail, ChostChat and GhostBox” (Flox)
10. “Please note that automated testing is not permitted! System will ban you permanently if you do” (DigitalSellz); “If you employ automated scanning tools, their requests must be rate limited to not exceed 3 requests per second without prior approval” (Vimeo)
11. “Do not attempt to gain access to another user’s account or confidential information” (Adobe)
12. “You are not allowed to conduct social engineering attacks against our support team” (Coinbase)
13. “While researching, we’d like to ask you to refrain from: [...] Any physical attempts against BitHunt property or data centers” (BitHunt)
14. “In order to encourage responsible disclosure, we promise not to bring legal action against researchers who point out a problem provided they do their best to follow the above” (Openfolio)
15. “You must comply with all applicable laws in connection with your participation in this program. You are also responsible for any applicable taxes associated with any reward you receive” (Twitter)
16. “Yahoo reserves the right to change or modify the terms of this program at any time” (Yahoo)
17. “Yahoo employees and contingent workers, as well as their immediate family members and persons living in the same household, are not eligible to receive bounties or rewards of any kind under the Yahoo Bug Bounty Program, whether hosted by Yahoo or any third party” (Yahoo)
18. “You must be eligible to work within the U.S.; meaning you are a U.S. citizen, a noncitizen national of the U.S., a lawful permanent resident, or an alien authorized to work within the U.S.” (Hack the Army)
19. “You must be 18 years of age or older. Please be an adult when messaging us. We want to work with serious security professionals only” (Envoy)
20. “Share with us the full details of any problem found. Detailed steps on reproducing the bug. If valuable, please include any screen-shots, links you clicked on, pages visited, etc. Provide us with a concrete attack scenario. How will the problem impact Bookfresh or our customers? Put the problem into context” (BookFresh)
21. “Provide us a reasonable amount of time to resolve the issue before any disclosure to the public or a third-party” (ownCloud)
22. “Minimum reward is $100 for security vulnerabilities. The reward depends on the vulnerability severity and will be paid via HackerOne only. Every researcher with accepted vulnerability will be mentioned on http://hackerone.com/algolia/thanks” (Algolia)
23. “We only reward the first reporter of a vulnerability” (DropBox)
24. “Twitter will determine in its discretion whether a reward should be granted and the amount of the reward” (Twitter)
25. “Post on our Hall of Fame. Your very own Informatica Bug Bounty T-Shirt With More Awesome Swag to Come” (Informatica)
26. “Security and privacy are top priorities at Coursera. We believe that no technology is perfect and that working with skilled security researchers across the globe is crucial in identifying weaknesses in our technology” (Coursera)
B Statistics for Programs Grouped by Description Length
C Statistics for Programs Grouped by Clauses
D Further Readability Analysis
Rights and permissions
Copyright information
© 2018 International Financial Cryptography Association
About this paper
Cite this paper
Laszka, A., Zhao, M., Malbari, A., Grossklags, J. (2018). The Rules of Engagement for Bug Bounty Programs. In: Meiklejohn, S., Sako, K. (eds) Financial Cryptography and Data Security. FC 2018. Lecture Notes in Computer Science(), vol 10957. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-58387-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-662-58387-6_8
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-58386-9
Online ISBN: 978-3-662-58387-6
eBook Packages: Computer ScienceComputer Science (R0)