Software Security Maturity in Public Organisations

Abstract

Software security is about building software that will be secure even when it is attacked. This paper presents results from a survey evaluating software security practices in software development lifecycles in 20 public organisations in Norway using the practices and activities of the Building Security In Maturity Model (BSIMM). The findings suggest that public organisations in Norway excel at Compliance and Policy activities when developing their own code, but that there is a large potential for improvement with respect to Metrics, Penetration testing, and Training of developers in secure software development.

A Questionnaire

The questionnaire is taken from the BSIMM activity descriptions [3], with only minor textual modifications. Note that the official BSIMM study does not rely on questionnaires. As mentioned before, BSIMM ranks activities on three levels, but we decided not to show this to the respondents; we also re-arranged some activities, placing variations on the same activity together even though they are on different levels in the BSIMM description.

1.1 A.1 Governance

Strategy and Metrics

• We publish our process for addressing software security; containing goals, roles, responsibilities and activities.

• We have a secure software evangelist role to promote software security internally.

• We educate our executives about the consequences of inadequate software security.

• We have identified gate locations in our secure software development process where we make go/no go decisions with respect to software security.

• We enforce the identified gate locations in our secure software development process where we make go/no go decisions with respect to software security, and track exceptions.

• We have a process of accepting security risk and documenting accountability. In this process we assign a responsible manager for signing off on the state of all software prior to release.

• The software security group publishes data internally on the state of software security within the organisation.

• In addition to the software security group, we have also identified members of the development teams that have a special interest in software security, and have a process for involving them in the software security work.

• We have identified metrics that measure software security initiative progress and success.

• The software security group has a centralized tracking application to chart the progress of all software.

• The software security group advertises the software security initiative outside the organization (for example by writing articles, holding talks in conferences, etc.).

Policy and Compliance

• The software security group has an overview of the regulations that our software has to comply with.

• We have a software security policy to meet regulatory needs and customer demands.

• The software security group is responsible for identifying all legislation related to personally identifiable information (for example personopplysningsloven).

• We have identified all the personally identifiable information stored by each of our systems and data repositories.

• All identified risks have to be mitigated or accepted by a responsible manager.

• We can demonstrate compliance with regulations that we have to comply with.

• We make sure that all vendor contracts are compatible with our software security policy.

• We promote executive awareness of compliance and privacy obligations.

• We have all the documentation necessary for demonstrating the organisation’s compliance with regulations we have to comply with (for ex. written policy, lists of controls, artifacts from software development).

• When managing our third party vendors, we impose our software security policies on them.

• Information from the secure software development process is routinely fed back into the policy creation process.

Education and Guidance

• We have a security awareness training program.

• We offer role-specific security courses (for example on specific tools, technology stacks, bug parade).

• The security awareness training content/material is tailored to our history of security incidents.

• We deliver on-demand individual security training.

• We encourage security learning outside of the software security group by offering specific training and events.

• We provide security training for new employees to enhance the security culture.

• We use the security training to identify individuals that have a particular interest in security.

• We have a reward system for encouraging learning about security.

• We provide security training for vendors and/or outsourced workers.

• We host external software security events.

• We require an annual software security refresher course.

• The software security group has defined office hours for helping the rest of the organization.

1.2 A.2 Construction/Intelligence

Attack Models

• We build and maintain a top N possible attacks list.

• We have a data classification scheme and an inventory of attacks so we can prioritize applications by the data handled by them.

• We maintain a list of likely attacker profiles.

• We collect and publish attack stories.

• The software security group keeps up to date by learning about new types of attacks/vulnerabilities.

• We have an internal forum to discuss attacks.

• We link abuse cases to each attacker profile.

• We have a list of technology-specific abuse cases.

• We have an engineering team that develops new attack methods.

• We have automated the attack methods developed by our engineers.

Security Features and Design

• Our software security group builds and publishes a library of security features.

• Security is a regular part of our organization’s software architecture discussion.

• The software security group facilitates the use of secure-by-design middleware frameworks/common libraries.

• The software security group is directly involved in the design of security solutions.

• We have a review board to approve and maintain secure design patterns.

• We require the use of approved security features and frameworks.

• We find and publish mature design patterns from the organization.

Standards and Requirements

• The software security group create standards that explain the accepted way to carry out specific security centric operations.

• We have a portal where all security related documents are easily accessible.

• The software security group assists the software development team in translating compliance constraints (for instance from legislation) into application specific security requirements.

• We use secure coding standards in our software development.

• We have a standards review board to formalize the process used to develop security standards.

• We use a limited number of standard technology stacks.

• We have a template SLA text for use in contracts with vendors and providers, to help prevent compliance and privacy problems.

• We have procedures to communicate and promote our security standards to vendors.

• We have a list of all open source components used in our software.

• We manage the risks related to using open source components.

1.3 A.3 Verification/Touchpoints

Design Review/Architecture Analysis

• We perform security feature review.

• We perform design review for high-risk applications.

• We have a software security group that leads review efforts.

• We use a risk questionnaire to rank applications in terms of the risk they are exposed to.

• We have a defined process to do architecture analysis.

• We have a standardized format for describring architecture that also covers data flow.

• The software security group is available to support architecture analysis when needed.

• The software architects lead design review efforts to detect and correct security flaws.

• Failures identified during architecture analysis are used to update the standard architecture patterns.

Code Review

• We create a list with top N software security defects list.

• The software security group does ad-hoc code reviews.

• We use automated tools (such as static analysis) along with manual review to detect software security defects.

• We make code review mandatory for all projects before release.

• The software security defects found during code review are tracked in a centralized repository.

• We enforce coding standards to improve software security.

• We have mentors for code review tools for making most efficient use of the tools.

• We use automated tools with tailored rules to improve efficiency and reduce false positives.

• We combine assessment results so that multiple analysis techniques feed into one reporting and remediation process.

• When a software defect is found we have tools to search for that defect also in the whole codebase.

• We perform automated code review on all code to detect malicious code.

Security Testing

• We perform adversarial tests with edge and boundary values.

• We create our tests based on existing security requirements and security features.

• We integrate black box security tools into the testing process (including protocol fuzzing).

• We share security test results with QA.

• We include security tests in QA automation.

• We perform fuzz testing customized to application APIs.

• We base the security tests on the security risks analysis.

• We use code coverage tools to ensure that security tests cover all parts of the code.

• We write tests cases based on abuse cases provided by the software security group.

1.4 A.4 Deployment

Configuration and Vulnerability Management

• The software security group has procedures for incident response, in collaboration with the incident response team (if it exists).

• We are able to make quick changes in the software when under attack.

• We perform drills to ensure that incident response capabilities minimize the impact of an attack.

• We identify software defects found in operations (for ex. by intrusion detection systems) and feed back to development.

• We track software defects found during operations until they are closed.

• We maintain a matrix of all installed applications in order to identify all places that need to be updated when a piece of code needs to be changed.

• When a software defect is found in a piece of code during operations we have a process to search for that defect also in the whole codebase.

• We do software security process improvement based on the analysis of cause of software defects found in operations.

• We have a system for paying rewards to individuals who report security flaws in our software.

Software Environment

• We monitor the input to software we run in order to spot attacks on our software.

• We use accepted good practice mechanisms for host/network security.

• The software security group creates and publishes installation guides to ensure that our software is configured securely.

• We create digital signatures for all binaries that we deliver.

• We use code protection such as obfuscation to make reverse engineering harder.

• We monitor the behavior of our software looking for misbehavior and signs of attacks.

Penetration Testing

• We use external penetration testers on our software.

• Defects found in penetration testing are inserted in our bug tracking system and flagged as security defects.

• We use penetration testing tools internally.

• The penetration testers have access to all available information about our software (for example: the source code, design documents, architecture analysis results and code review results).

• We periodically perform penetration tests on all our software.

• We use external penetration testers to do deep-dive analysis for critical projects to complement internal competence.

• The software security group has created customized penetration testing tools and scripts for our organization.