Evolution of Security from Waterfall to DevSecOps
Waterfall and Security
The Waterfall model had been the main methodology in software development for several decades. In this approach, project activities were done in linear sequential phases. Progress flowed in one direction (like a waterfall) through the phases of requirements, design, development, testing, deployment, and maintenance. At that time, little attention was paid to Application Security (AppSec). Instead, customized security controls being part of a Security Management System were used to ensure that security risks were cost effectively managed (see detailssee details). In Capability Maturity Model Integration (CMMI), security was considered as a quality attribute expected in the final software products, together with safety and ergonomics (see detailssee details).
Under the approach taken at this time, the security teams collaborated with the software development teams twice throughout the entire lifecycle of the program: (a) during the requirements and design phase, and (b) during the security check prior to the planned software release. After the requirements and architecture of the software product were defined by the project team, this project was submitted to the AppSec team for review. The AppSec team review was conducted according to a traditional methodology, usually including some form of threat modeling and data flow exercises to ensure the privacy and security of data. The review resulted in a documented list of security requirements to be incorporated into the software.
The security requirements were provided to the development team. Developers created a high-level design document answering such types of questions as: “How can a service composition architecture be secured so as to allow it to remain flexible for future changes and recomposition?”; “What are the primary security mechanisms and industry standards that typically need to be established to provide common security measures that can be used by all services?” or “If a service must rely on shared security resources how can performance degradation be mitigated?”.
Coming from the other side, developers were taking practical measures, e.g. single sign-on, XML encryption, interactions with Public Key Infrastructure to be implemented in the software product for the next several months. Indeed, traceability of security requirements into high-level design, and mapping the latter with practical steps made towards protecting the software product was a challenging endeavor.
Once the development teams have released a software version that satisfies the documented security requirements and stakeholders’ expectations, the AppSec team can start implementation of their second security phase. This phase involves running a suite of penetration tests to identify any security vulnerabilities in the release candidate product. Duration of this would vary from a couple of weeks to a couple of months. The result of this stage of the AppSec team's work, in most cases, was a list of discovered vulnerabilities that must be fixed before launching the software in production. Hopefully, the vulnerabilities would not impact the high-level design and the root cause of the findings would not be vague security requirements in the first place. So, the security check being made late in the development cycle often led to a huge rework of the code followed by re-testing of a product. The location of the security gate and rework paths are shown in the following figure:
The AppSec team really got involved just in the latest stages of the software development lifecycle. The development team was forced to fix security issues just before the end of the project. This was a risk to the project and often resulted in delays in the overall project schedule. Often, the AppSec team had no time to re-run all the penetration tests to verify that all security issues were addressed in the next build. A root cause of it lies in a fact that penetration tests were executed by external teams of “hackers” and were not properly documented, not to mention the absence of automation.
Security in Agile Times
The Agile methodology was invented to overcome the drawbacks of the linear sequential approach of the Waterfall methodology. The Agile Manifesto, released in 2001 by a group of software development visionaries, defines a set of twelve principles that promote quality, provide for continuous delivery, and incorporate regular feedback loops and collaboration between individuals and teams.
The challenge for the new Agile methodology was to stop using the rigid sequential Waterfall process and to create a flexible development environment. Such an agile environment made it possible to change the development process throughout the whole software development lifecycle in the form of iterations based on new technical and business requirements. Another principle of Agile is “Deliver working software frequently.” Those days, “frequently” would mean more than once a quarter or even a few times a month. Working software is the primary measure of progress.
The Agile methodology strongly encourages collaboration among cross-functional teams. In this way, employees from different teams can meet the requirements of continuous planning, development, integration and testing. The development core of Agile can outrun requirements setting, architecture runway, and quality. We must be sensitive to this dynamic, given that Agile would often outrun Security, too.
Security is meant to be included into Agile’s Definition of Done (DoD); e.g. manual penetration testing is performed, no blockers have been found, payment cards information is masked in all user interfaces, etc. However, security-related items were never clearly formulated in Agile. Also, because of Agile's tendency to skip documentation by all means, they were not properly linked to software design or protection mechanisms.
The Agile methodology retained the two-phase approach to software security used in the Waterfall methodology. At the first phase a set of security-related activities are defined during the early stages of the program lifecycle. The next phase is software security testing, which is performed before launching the software into production, often several months after the previous phase. Thus, like the Waterfall approach, the Agile approach to software security does not really solve software security problems.
Security Test Pyramid
In 2009 Mike Cohn introduces the concept of the test pyramid in the book "Succeeding with Agile". The pyramid represents three levels of testing: unit tests form the base of the pyramid; Ul-driven tests at the top of the pyramid; and sandwiched between are service tests. The higher up the pyramid you go, the fewer tests you have, but the cost of each is greater. A comprehensive test strategy includes tests from all layers of the pyramid, starting with a higher number of granular tests at the bottom of the pyramid and ending with a small number of broader tests at the top.
Timing of testing was different too: continuous execution (every build) was assumed only for bottom in the pyramid tests: unit and smoke tests. Tests in the middle of the pyramid, such as regression and functional tests, were run daily or weekly. And top tests, called system on our picture, which by their nature were either long running or manual, were executed weekly or once per sprint. For example, performance, load test and penetration tests belong there.
Although manual security testing has been a core component of traditional software delivery for a number of years, as organizations adopt new ways of working, automated testing has become a necessity. In Agile, the use of manual penetration tests creates bottlenecks in the delivery cycle; nevertheless, security teams persist in prioritizing manual testing over automated testing, failing to keep up with Agile core principles. Unfortunately, in this scenario of opposing priorities, delivery teams often bypass manual security testing completely to avoid impacting delivery schedules.
Security and DevOps
The introduction of the Agile approach has greatly accelerated the software development process. However, not all obstacles have been removed. One of the unresolved problems remained the interaction between the development (Dev) team and the operations (Ops) team. The process of transferring software from the Dev team to the Ops team could be time-consuming and, if a problem occurred, often triggered a dispute between the teams whose environment is not properly configured.
The answer to this problem was the appearance of the DevOps methodology in the late 2000s. DevOps combined the work of Dev and Ops teams into a single process. A culture of collaboration between these teams was conceptually invented and implemented. The DevOps concept claims that software development, testing and deployment should take place regularly and without delays. This approach led to the appearance of the continuous integration (CI) / continuous delivery (CD) concept as part of DevOps, as shown in the figure below:
DevSecOps
The DevOps concept solved the problems related to the integration and automation of all software development, testing and deployment processes. However, DevOps never decided how to solve the software security problem. The key is that the embedding of software security to DevOps is not carried out by the Dev team and not by the Ops team that each perform its own tasks, but by a special security (Sec) team. This led to a shift from the DevOps concept to the DevSecOps concept.
DevSecOps is based on the following key principles:
- Secure-by-Design
- Process
- Maturity Model
- Security Shift-left paradigm
- Metrics
- Automation
- Everything as a Code
- Use of AI
- Security Education
Only applications made by a team taking secure design seriously from day zero could be resilient and protect customers’ data privacy and ensure their sensitive assets remained untouched by groups of hackers. Best secure design and development practices should be in place along with trusted and protected software libraries, open-source components and frameworks.
Maturity model is necessary for understanding of existing state of DevSecOps practices comparing to industry leading standards and further continuous improvements.
Shifting security left means providing the ability and responsibility to implement secure software engineering practices at the very beginning of and throughout the CI/CD pipeline and protecting the development perimeter from the injection of malicious open-source components.
DevSecOps practices encourage automation to achieve scale according to the Everything-as-a-Code concept, but security has traditionally been manual, process-heavy and gate-driven — the antithesis of automation, transparency and speed.
AI provides insightful analytics into the entire secure software engineering process for all parties involved based on DevSecOps data continuously collected and aggregated in a data warehouse.
To build a successful DevSecOps within a software organization would mean to educate everybody involved into creation of the protected software products and services, starting from marketing and product managers formulating business requirements, then, the Dev team applying security into a code, then the Ops team monitoring security incidents in a Security Operations Center (SOC), then the Sec team of cyber experts ready to investigate security incidents in the field. Indeed, the Sec team orchestration of all the efforts is part of the puzzle too. Security champions are playing the role of security culture conductors and are responsible for application security tasks within software delivery teams.
Tools Which Made an Impact on Transition
We are of the opinion that evolution of the software industry is driven by visionaries and practicians. You could not expect a theoretical research would drive the change in the ways how software is built, distributed and operated. We put together top-12 technological timeline of most impactful tools which made a significant contribution into transition from Waterfall to DevOps.
- 2001, Rally Software, Agile project management tools
- 2002, Jira by Atlassian, Issue tracking at that time
- 2004, Confluence by Atlassian, wiki team collaboration
- 2006, TeamCity, CI server
- 2008, GitHub, Version control and SCM
- 2008, Bitbucket, Code review
- 2008, JFrog Artifactory, Repository Manager
- 2011, Jenkins, open source automation server
- 2011, GitLab, DevOps lifecycle tool
- 2013, Docker, Containers
- 2014, Kubernetes, Container Orchestration
- 2015, Sonatype Nexus OSS, Repository Manager
Often, visionaries, who didn’t find a proper tool or a methodology on the market to support their needs, created a software product for their own use and one day discovered billions of users following them.