Editor's Note: Today’s post is from Eric Johnson. Eric is a Senior Security Consultant at Cypress Data Defense, and the Application Security Curriculum Product Manager at SANS. In this post, Eric introduces Secure DevOps and some key DevOps concepts.
This month, our STH.Developer Software Development Lifecycle (SDLC) training module was selected for the video of the month. The SDLC topic reviews the challenges that software development teams face when building security into their lifecycle. In case you missed it, we walked through securing a traditional Waterfall development lifecycle in a previous post. So, by selecting DevOps as September’s video of the month, we are continuing our SDLC journey. Before watching the training module, let’s review some key concepts in DevOps that will help you understand its role in software development.
DevOps addresses the division that typically exists between the “Development” and “Operations” teams. In a pre-DevOps world, development teams were responsible for writing code. Period. The code changes were queued up for the quality assurance team, who ran the software through their testing plans. Then, another handoff was made to the system admins / operations team responsible for deploying code. This strict division of responsibilities causes a few problems:
- Each handoff creates overhead, slowing down the time to market.
- The deployment and release phases are far removed from the development team, increasing the number of failed deployments.
The goal of DevOps is to break down the separation between these various teams, and permit / encourage them to work together. Developers, quality assurance testers, system admins, database administrators, and application security experts all become the DevOps team. They work side-by-side, minimizing the handoffs, to develop, test, deploy, and support their applications. This allows everyone to remain close to the end-to-end process and achieve a common goal: creating reliable software, as quickly as possible, with a high probability of success.
Those of us that have been in software development for a long time remember how painful deploying code was in the past. We committed our code changes, wrote a script to update the database schema, manually updated settings on the development server, and wrote up a step-by-step list of instructions for the operations team to follow. Then, waited the required lead-time for quality assurance and security testing. Finally, weeks or months later on deployment night, the operations team walked through the steps and sent the email notifying us that the deployment was complete. We then stayed up half the night troubleshooting our applications, figuring out where it went wrong. It was painful!
The problem with this process is the manual deployment of changes from development through production. Continuous Delivery addresses this by automating the entire deployment process including server configuration, code changes, database updates, and security testing. Tools, such as Chef and Puppet, can be used to apply changes to all environments in a consistent and repeatable manner. Testing frameworks, such as Gauntlt or BDD-Security, are configured to automate quality assurance and security test plans. Taking the time to automate the deployment and release phases of the lifecycle minimizes the number of failed deployments, allowing the DevOps team to focus on more important tasks.
Now that we have automated the promotion of our changes using Continuous Delivery, let’s discuss how often we schedule a deployment. Many organizations stack up changes for weeks or months at a time, and release them in batches to production. But what happens if a release causes an outage? Or a high-risk vulnerability is discovered in production? In the past, the ability to react quickly and deploy patches has been limited. Continuous Deployment builds on Continuous Delivery by providing the development team a self-service interface to schedule deployments. After committing source code changes, a release is immediately scheduled, picked up by the deployment tool, and pushed into the Continuous Delivery pipeline. It should be noted that using Continuous Deployment has many security considerations, which are covered in the video below.
In addition to the feedback from the automated test results during delivery, DevOps continues its reliance on automation by monitoring production for important events, performance metrics, and exceptions. Once a normal baseline is established, thresholds for notification and incident response can be put in place. Continuous Feedback is achieved by collecting this data, analyzing the results, and generating reports for the entire DevOps team. Again, working together, the DevOps team is in a position to react and fix problems quickly.
Video of the Month
Now that we’ve covered the basics, check out the DevOps module on the STH.Developer Security Awareness Training video of the month page: https://www.securingthehuman.org/resources/votm If you’re hungry for more, SANS Analyst and lead author of the SANS Application Security Survey series, Jim Bird, is hard at work writing a new Secure DevOps live course. You can find a number of his Agile and DevOps blogs at http://swreflections.blogspot.com/.
Bio: Eric Johnson (Twitter: @emjohn20) is the Application Security Curriculum Product Manager at SANS, the lead author and instructor for DEV544 Secure Coding in .NET, and an instructor for DEV541 Secure Coding in Java/JEE. A senior security consultant at Cypress Data Defense, Eric's experience includes web and mobile application penetration testing, secure code review, risk assessment, static source code analysis, security research, and developing security tools. He currently holds the CISSP, GWAPT, GSSP-.NET, and GSSP-Java certifications.