Step One Towards Science of Security

Science of security necessitates conducting methodologically-defensible research and reporting such research comprehensively to enable replication and future research to build upon the reported study. The comprehensiveness of reporting is as important as the research itself in building a science of security. Key principles of science - replication, meta-analysis, and theory building - are affected by the ability to understand the context and findings of published studies. The goal of this paper is to aid the security research community in understanding the state of scientific communication through the analysis of research published at top security conferences. To analyze scientific communication, we use literature on scientific evaluation to develop a set of rubrics as a guide to check the comprehensiveness of papers published in the IEEE Security and Privacy and ACM Computer and Communications Security conferences. Our review found that papers often omit certain types of information from their reports, including research objectives and threats to validity. Our hope is that this effort sheds some light on one of the essential steps towards advancement of the science of security.