Do I really need all this work to find vulnerabilities?

Abstract

Context:

Applying vulnerability detection techniques is one of many tasks using the limited resources of a software project.

Objective:

The goal of this research is to assist managers and other decision-makers in making informed choices about the use of software vulnerability detection techniques through an empirical study of the efficiency and effectiveness of four techniques on a Java-based web application.

Method:

We apply four different categories of vulnerability detection techniques – systematic manual penetration testing (SMPT), exploratory manual penetration testing (EMPT), dynamic application security testing (DAST), and static application security testing (SAST) – to an open-source medical records system.

Results:

We found the most vulnerabilities using SAST. However, EMPT found more severe vulnerabilities. With each technique, we found unique vulnerabilities not found using the other techniques. The efficiency of manual techniques (EMPT, SMPT) was comparable to or better than the efficiency of automated techniques (DAST, SAST) in terms of Vulnerabilities per Hour (VpH).

Conclusions:

The vulnerability detection technique practitioners should select may vary based on the goals and available resources of the project. If the goal of an organization is to find “all” vulnerabilities in a project, they need to use as many techniques as their resources allow.

Appendices

Appendix A: Automated Technique CWEs

Table 10 CWEs covered in the rules implemented by automated techniques

Appendix B: Student Experience Questionnaire

At the beginning of the course, students were asked to fill out a survey about their experience relevant to the course. The four questions asked to students were as follows:

• 1. How much time have you spent working at a professional software organization – including internships – in terms of the # of years and the # of months?

• 2. On a scale from 1 (none) to 5 (fully), how much of the time has your work at a professional software organization involved cybersecurity?

• 3. Which of the follow classes have you already completed?

• 4. Which of the following classes are you currently taking?

Q1 was short answer. For Q2, students selected a single number between 1 and 5. For Q3, the students could check any number of checkboxes corresponding to a list of the security and privacy courses offered at the institution. For Q4, the students selected from the subset of classes from question 4 that were being offered the semester in which the survey was given.

Fifty-nine of the sixty-three students who agreed to let their data be used for the study responded to the survey. Of these 59 responses, four students responses to Q1 provided a numeric value, e.g. “3”, but did not specify whether the numeric value indicated years or months. We considered this invalid and summarize experience from the remaining 55 participants in Section 7.2

Appendix C: Student Assignments

The following are the verbatim assignments for the Course Project that guided the tasks performed by students. We have removed sections of the assignment that are not relevant to this project. Additionally, information that is specific to the tools used, such as UI locations, has also been removed. Text that has been removed is indicated by square brackets [ ].

3.1 C.1 Project Part 1

Throughout the course of this semester, you will perform and document a technical security review of OpenMRS (http://openmrs.org). This open-source systems provides electronic health care functionality for “resource-constrained environments”. While the system has not been designed for deployment within the United States, security and privacy concerns are still a paramount security concern for any patient.

3.1.1 Software:

OpenMRS 2.9.0. There is no need to install OpenMRS. You will use the VCL image CSC515_SoftwareSecurity_Ubuntu.

3.1.2 Deliverables:

Submit a PDF with all deliverables in Gradescope. Only one submission should be performed per team. Do not include your names/IDs/team name on the report to facilitate the peer evaluation of your assignment (see Part 3 of this assignment).

• Security test planning and execution (45 points)

  • Record how much total time (hours and minutes) your team spends to complete this activity (test planning and test execution). Compute a metric of how many true positive defects you found per hour of total effort.

  • Test planning. Create 15 black box test cases to start a repeatable black box test plan for the OpenMSR (Version 2.9). You may find the OWASP Testing Guide and OWASP Proactive Controls helpful references in addition to the references provided throughout the ASVS document.

    For each test case, you must specify:

    • A unique test case id that maps to the ASVS, sticking to Level 1 and Level 2. Provide the name/description of the ASVS control. Only one unique identifier is needed (as opposed to the example in the lecture slides). The ASVS number should be part of the one unique identifier.

    • Detailed and repeatable (the same steps could be done by anyone who reads the instructions) instructions for how to execute the test case

    • Expected results when running the test case. A passing test case would indicate a secure system.

    • Actual results of running the test case.

    • Indicate the CWE (number and name) for the vulnerability you are testing for.

    In choosing your test cases, we are looking for you to demonstrate your understanding of the vulnerability and what it would take to stress the system to see if the vulnerability exists. You may have only one test case per ASVS control.

  • Extra credit (up to 5 points): Create a black box test case that will reveal the vulnerability reported by the static analysis tool (Part 2 of this assignment) for up to 5 vulnerabilities (1 point per vulnerability). Provide the tool output (screen shot of the alert) from each tool.

• Static analysis (45 points)

  • Record how much total time (hours and minutes) your team spends to complete this activity (test planning and test execution). Compute a metric of how many defects you found per hour of total effort.

  • For each of the three tools (below), review the security reports. Based upon these reports:

    • References:

      • Troubleshooting VCL

      • Opening OpenMRS on VCL

    • Randomly choose 10 security alerts and provide a cross-reference back to the originating report(s) where the alert was documented. Explore the code to determine if the alert is a false positive or a true positive. The alerts analyzed MUST be security alerts even though the tools will report “regular quality” alerts – you need to choose security alerts.

    • If the alert is a false positive, explain why. If you have more than 5 false positives, keep choosing alerts until you have 5 true positives while still reporting the false positives (which may make you go above a total of 10).

    • If the alert is a true positive, (1) explain how to fix the vulnerability; (2) map the vulnerability to a CWE; (3) map the vulnerability to the ASVS control.

    • Find the instructions for getting [SAST-3] going on OpenMRS here[hyperlink removed]. [Tool-specific instructions]

    • Find the instructions for getting [SAST-2] going on OpenMRS here[hyperlink removed]. [Tool-specific instructions]

  • Extra credit (up to 5 points): Find 5 instances (1 point per instance) of a potential vulnerability being reported by multiple tools. Provide the tool output (screen shot of the alert) from each tool. Explore the code to determine if the alert is a false positive or a true positive. If the alert is a false positive, explain why. If the alert is a true positive, explain how to fix the vulnerability.

• Peer evaluation (10 points)

  Perform a peer evaluation on another team. Produce a complete report of feedback for the other team using this rubric [to be supplied]. Note: For any part of this course-long project, you may not directly copy materials from other sources. You need to adapt and make unique to OpenMRS. You should provide references to your sources. Copying materials without attribution is plagiarism and will be treated as an academic integrity violation.

3.2 C.2 Project Part 2

The fuzzing should be performed on the VCL Class Image (“CSC 515 Software Security Ubuntu”).

• Black Box Test Cases

  Parts 1 (OWASP ZAP) and 2 ([DAST-2]) ask for you to write a black box test case. We use the same format as was used in Project Part. For each test case, you must specify:

  • A unique test case id that maps to the ASVS, sticking to Level 1 and Level 2. Provide the name/description of the ASVS control. Only one unique identifier is needed (as opposed to the example in the lecture slides). The ASVS number should be part of the one unique identifier.

  • Detailed and repeatable (the same steps could be done by anyone who reads the instructions) instructions for how to execute the test case

  • Expected results when running the test case. A passing test case would indicate a secure system.

  • Actual results of running the test case.

  • Indicate the CWE (number and name) for the vulnerability you are testing for.

• OWASP ZAP (30 points, 3 points for each of the 5 test cases in the two parts)

  Client-side bypassing

  • Record how much total time (hours and minutes) your team spends to complete this activity. Provide:

    • Total time to plan and run the 5 black box test cases.

    • Total number of vulnerabilities found.

  • Plan 5 black box test cases (using format provided in Part 0 above) in which you stop user input in OpenMRS with OWASP ZAP and change the input string to an attack. (Consider using the strings that can be found in the ZAP rulesets, such as jbrofuzz) Use these instructions as a guide.

  • In your test case, be sure to document the page URL, the input field, the initial user input, and the malicious input. Describe what “filler” information is used for the rest of the fields on the page (if necessary).

  • Run the test case and document the results.

  Fuzzing

  • Record how much total time (hours and minutes) your team spends to complete this activity.

    • Do not include time to run ZAP

    • Provide:

      • Total time to work with the ZAP output to identify the 5 vulnerabilities.

      • Total time to plan and run the 5 black box test cases.

  • Use the 5 client-side bypassing testcases (above) for this exercise.

  • Use the jbrofuzz rulesets to perform a fuzzing exercise on OpenMRS with the following vulnerability types: Injection, Buffer Overflow, XSS, and SQL Injection.

  • Take a screen shot of ZAP information on the five test cases.

  • Report the fuzzers you chose for each vulnerability type along with the results, and what you believe the team would need to do to fix any vulnerabilities you find. If you don’t find any vulnerabilities, provide your reasoning as to why that was the case, and describe and what mitigations the team must have in place such that there are no vulnerabilities.

• DAST-2 (25 points)

  [DAST-2] FAQ [hyperlink removed] and [DAST-2] Troubleshooting [hyperlink removed]

  • Record how much total time (hours and minutes) your team spends to complete this activity.

    • Do not include time to run [DAST-2].

    • Provide:

      • Total time to work with the [DAST-2] output to identify the 5 vulnerabilities.

      • Total time to plan and run the 5 black box test cases.

  • Run [DAST-2] on OpenMRS. Run any 5 of your test cases from Project Part 1 to seed the [DAST-2] run. Run [DAST-2] long enough that you feel you have captured enough true positive vulnerabilities that you can complete five test case plans. Note: [DAST-2] will like run out of memory if you run all 5 together. It is best to run each one separately. Also, make sure you capture only the steps for your test cases, not other unnecessary steps.

  • Export your results.

  • Take a screen shot of [DAST-2] information on the five vulnerabilities you will explore further. Write five black box test plans (using format provided in Part 0 above) to expose five vulnerabilities detected by [DAST-2] (which may use a proxy). Hint: Your expected results should be different from the actual results since these test cases should be failing test cases.

• Vulnerable Dependencies (35 points)

  [Assignment Section not Relevant]

• Peer evaluation (10 points)

  Perform a peer evaluation on another team. Produce a complete report of feedback for the other team using this rubric (to be supplied).

3.3 C.3 Project Part 3

The project can be done on the VCL Class Image (“CSC 515 Software Security Ubuntu”).

• Black Box Test Cases

  Parts 1 (Logging), 2 ([Interactive Testing]), and 3 (Test coverage) ask for you to write black box test cases. We use the same format as was used in Project Part 1. For each test case, you must specify:

  • A unique test case id that maps to the ASVS, sticking to Level 1 and Level 2. Provide the name/description of the ASVS control. Only one unique identifier is needed (as opposed to the example in the lecture slides). The ASVS number should be part of the one unique identifier.

  • Detailed and repeatable (the same steps could be done by anyone who reads the instructions) instructions for how to execute the test case

  • Expected results when running the test case. A passing test case would indicate a secure system.

  • Actual results of running the test case.

  • Indicate the CWE (number and name) for the vulnerability you are testing for.

• Logging (25 points)

  Where are the Log files? Check out the OpenMRS FAQ

  • Record how much total time (hours and minutes) your team spends to complete this activity (test planning and test execution). Compute a metric of how many true positive defects you found per hour of total effort.

  • Write 10 black box test cases for ASVS V7 Levels 1 and 2. You can have multiple test cases for the same control testing for logging in multiple areas of the application. What should be logged to support non-repudiation/accountability should be in your expected results.

  • Run the test. Find and document the location of OpenMRS’s transaction logs.

  • Write what is logged in the actual results column. The test case should fail if non-repudiation/accountability is not supported (see the 6 Ws on page 3 of the lecture notes).

  • Comment on the adequacy of OpenMRS’s logging overall based upon these 10 test cases.

• Interactive Application Security Testing (25 points)

  [Assignment Section not Relevant]

• Test Coverage (25 points)

  This test coverage relates to all work you have done in Project Parts 1, 2, and 3.

  • 1. Compute your black box test coverage for each section of the ASVS (i.e. V1, V2, etc.) which includes the black box tests you write for Part 2 (Seeker) for Level 1 and Level 2 controls. You get credit for a control (e.g. V1.1) if you have a test case for it. If you have more than one test case for a control, you do not get extra credit –coverage is binary. Coverage is computed as # of test cases / # of requirements.

  • 2. (15 points, 3 points each) Write 5 more black box tests to increase your coverage of controls you did not have a test case for.

  • 3. (5 points) Recompute your test coverage. Report as below. Record how much total time (hours and minutes) your team spends to complete this activity (test planning and test execution). Compute a metric of how many true positive defects you found per hour of total effort.

  • 4. (5 points) Reflect on the controls you have lower coverage for. Are these controls particularly hard to test, we didn’t cover in class, you just didn’t get to it, etc.

  Control	# of test cases	# of L1 and L2 controls	Coverage
  V1.1: Secure development lifecycle	?	7	?/7
  ...
  Total

• Vulnerability Discovery Comparison (15 points)

  • 1. (5 points) Compare the five vulnerability detection techniques you have used this semester by first completing the table below.

    • A: total number # of true positives for this detection type for all activities (Project Parts 1-3)

    • B: total time spent on all for all activities (Project Parts 1-3)

    • Efficiency is A/B

    • Exploitability: give a relative rating of the ability for this technique to find exploitable vulnerabilities

    • Provide the CWE number for all the true positive vulnerabilities detected by this technique. (This information will help you address the “wide range of vulnerability types” question below.)

    Technique	# of true positive vulnerabilities discovered	Total time (hours)	Efficiency: # vulnerabilities / total time	Detecting Exploitable vulnerabilities? (High/Med/Low)	Unique CWE numbers
    Manual black box
    Static analysis
    Dynamic analysis
    Interactive testing

  • 2. (10 points) Use this data to re-answer the question that was on the midterm (that people generally didn’t do too well on). Being able to understand the tradeoffs between the techniques is a major learning objective of the class.

    As efficiently and effectively as possible, companies want to detect a wide range of exploitable vulnerabilities (both implementation bugs and design flaws). Based upon your experience with these techniques, compare their ability to efficiently and effectively detect a wide range of types of exploitable vulnerabilities.

• Peer evaluation (10 points)

  Perform a peer evaluation on another team. Produce a complete report of feedback for the other team using this rubric [to be supplied].

3.4 C.4 Project Part 4

• Protection Poker (20 points)

  [Assignment Section not Relevant]

• Vulnerability Fixes (35 points)

  [Assignment Section not Relevant]

• Exploratory Penetration Testing (35 points)

  Each team member is to perform 3 hours of exploratory penetration testing on OpenMRS. This testing is to be done opportunistically, based upon your general knowledge of OpenMRS but without a test plan, as is done by professional penetration testers. DO NOT USE YOUR OLD BLACK BOX TESTS FROM PRIOR MODULES. Use a screen video/voice screen recorder to record your penetration testing actions. Speak aloud as you work to describe your actions, such as, “I see the input field for logging in. I’m going to see if 1 = 1 works for a password.” or “I see a parameter in the URL, I’m going to see what happens if I change the URL.” You should be speaking around once/minute to narrate what you are attempting. You don’t have to do all 3 hours in one session, but you should have 3 hours of annotated video to document your penetration testing. There’s lots of screen recorders available – if you know of a free one and can suggest it to your classmates, please post on Piazza.

  Pause the recording every time you have a true positive vulnerability. Note how long you have been working so a log of your work and the time between vulnerability discovery is created (For example, Vulnerability #1 was found at 1 hour and 12 minutes, Vulnerability #2 was found at 1 hour and 30 minutes, etc.) If you work in multiple sessions, the elapsed time will pick up where you left off the prior session – like if you do one session for 1 hour 15 minutes, the second session begins at 1 hour 16 minutes. Take a screen shot and number each true positive vulnerability . Record your actions such that this vulnerability could be replicated by someone else via a black box test case. Record the CWE for your true positive vulnerability. Record your work as in the following table. The reference info for video traceability is to aid a reviewer in watching you find the vulnerability. If you have one video, the “time” should aid in finding the appropriate part of the video. If you have multiple videos, please specify which video and what time on that video.

  Vulnerability #

  Elapsed Time

  Ref Info for Video Traceability

  CWE

  Commentary

  Replication instructions via a black box test and the screenshots for each true positive vulnerability should appear below the table, labeled with the vulnerability number. Since you are not recording all your steps, the replication instructions may not work completely since you may change the state of the software somewhere along the line – document what you can via a black box test and say the actual results don’t match your screenshot.

  After you are complete, compute an efficiency metric (true positive vulnerability/hour) metric for each student. Submit a table:

  	# vuln	Time	Efficiency
  Name 1
  Name 2
  Name 3
  Name 4
  Total

  Copy the efficiency table you turned in for Project Part 3 #4. Add an additional line for Penetration testing. Compare and comment on this efficiency rate with the other vulnerability discovery techniques in the table you input in #4 of Project Part 3.

  • Each person on the team should submit one or more videos by uploading it/them to your own google drive and providing a link to the video(s), sharing the video with anyone who has the link and an NCSU login (which will allow peer evaluation and grading). The video(s) should be approximately 3 hours in length.

  • A person who does not submit a video can not be awarded the points for this part of the project while the rest of the team can.

  • It is possible to work for 3 hours and find 0 vulnerabilities – real penetration tests constantly work more than 3 hours without finding anything. That’s part of the reason for documenting your work via video.

  • For those team members who do submit videos, the grade will be an overall team grade.

  Submission: The team submits one file with the links to the team member’s files.

• Peer Evaluation (10 points)

  Perform a peer evaluation on another team. Produce a complete report of feedback for the other team using this rubric [to be supplied].

Appendix D: Equipment Specifications

In this appendix we provide additional details of the equipment used in our case study. As noted in Section 11, a key resource used in this project was the school’s Virtual Computing Lab²⁷ (VCL), which provided virtual machine (VM) instances. Researchers used VCL when applying EMPT, SMPT, and DAST as part of data collection for RQ1. All student tasks were performed using VCL for RQ1 and RQ2. Researchers created a system image including the SUT (OpenMRS) as well as SAST and DAST tools. The base image was assigned 4-cores, 8G RAM, and 40G disk space. An instance of this image could be checked out by students and researchers and accessed remotely through a terminal using ssh or graphically using Remote Desktop Protocol (RDP). Researchers also used two expanded instances of the base image with 16 CPUs, 32GB RAM, and 80G disk space. For client-server tools, a server was setup in a separate VCL instance by researchers with assistance from the teaching staff of the course. The server UI was accessible from VCL instances of the base image, while the server instance itself was only accessible to researchers and teaching staff. The server instance had 4 cores, 8G RAM, and 60G disk space disk space, and contained the server software for SAST-1 used to answer RQ2. All VCL instances in this study used the Ubuntu operating system.

The VCL alone was used for data collection for RQ2. However, the base VCL images were small, and the remote connection to VCL could lag. Researchers used two used additional resources as needed for RQ1 data collection. First, we created a VM in VirtualBox using the same operating system (Ubuntu 18.04 LTS) and OpenMRS version (Version 2.9) as the VCL images. This VM was used by researchers for SMPT and EMPT data collection, particularly when reviewing the output of each technique where instances of the SUT were needed on an ad hoc basis. The VM was assigned 2 CPUs, 4GB RAM, and 32G disk space and could be copied and shared amongst researchers to run locally. Researchers increased the size of the VM as needed, up to 8 CPUs and 16GB RAM when the host system could support the VM size. A second VM was created in VirtualBox with the same specifications and operating system, but with the server software for Sonarqube installed. We also used a desktop machine with 24 CPUs, 32G RAM, and 500G disk space. The desktop was running the Ubuntu operating system. This machine was accessible through the terminal via ssh and graphically using x2go²⁸. For RQ1 data collection we ran the SAST-1 server software directly on this machine. The desktop was also used to run VirtualBox VMs for resource-intensive activity such as running Sonarqube and DAST-2.

While equipment constraints impacted both SAST and DAST, available equipment and intended use also impacts how SAST tools are setup. SAST tools can be setup and configured according to different architectures. The SAST tools used in this study could be setup as client-server tools where the SUT code is scanned on the “client” machine, and information is sent to a “server”. The analyst then reviews the results through the server. For some tools, the automated analysis and rules are applied on the client, while for other tools the automated analysis and rules are applied on the server. The SAST tools used in this study also included an optional plugin for Integrated Development Environments (IDEs) such as Eclipse²⁹. The plugin allows developers to initialize SAST analysis and in some cases view alerts from the tool within the IDE itself. Some tools can be run without a server using only IDE plugins. Other tools require a server. Similar to the previous work by Austin et al. (Austin and Williams 2011; Austin et al. 2013), we found that the server GUI was easier to use when aggregating and analyzing all system vulnerabilities for RQ1. Consequently, a client-server configuration was used with SAST-2 and Sonarqube to answer RQ1. SAST-2 and SAST-3 were more easily configured to use locally within an IDE, as was done in for the class with RQ2 and RQ3.

Appendix E: All CWEs Table

Table 11 shows the CWE for high and medium severity vulnerabilities found. Table 12 provides the same information for low severity vulnerabilities. The first column of the table indicates the CWE number. The CWEs are organized based on the OWASP Top Ten Categories. The second column of the table indicates which, if any, of the OWASP Top Ten the vulnerability maps to. Columns three and four of the table are the number of vulnerabilities found by the techniques SMPT and EMPT. Columns five through eight break down the vulnerabilities found by DAST and SAST by tool (ZAP, DA-2, Sonar, and SA-2). Column nine of Table 11 shows the total number of vulnerabilities found of each CWE type. The Total column is not the same as the sum of the previous six columns. Some vulnerabilities were found using more than one technique. Similarly, 20 Vulnerabilities were associated with more than one CWE; therefore the total vulnerabilities for each technique as shown in Table 5 may be lower than the sum of each column in Table 11.

Table 11 CWEs associated with more severe Vulnerabilities

Table 12 Low Severity Vulnerability CWEs