As SANS prepares for the 2nd Annual Secure DevOps Summit, Co-Chairs Frank Kim and Eric Johnson are tackling some of the common questions they get from security professionals who want to understand how to inject security into the DevOps pipeline, leverage leading DevOps practices, and secure DevOps technologies and cloud services.
If you are considering attending the Secure DevOps Summit or have already registered, this post is a good warmup, as these questions will be covered in-depth at the Summit on October 22nd-23rd in Denver, CO.
1. What do security professionals need to do to implement Secure DevOps processes?
SecDevOps requires security and compliance teams to integrate their processes, policies, and requirements into the same workflows being used by Development & Operations. This does not happen without the Security team understanding the current development and operations engineering process, technologies, and tools.
The Continuous Integration (CI) tool, is one of the most important pieces in the DevOps process, manages the workflow, executes steps, and integrates the entire toolchain together. Common examples include Jenkins, Visual Studio Team Services (VSTS), Travis, Bamboo, TeamCity.
Leveraging the CI tool, SecDevOps teams will start to integrate important security touch points such as:
- Pre-Commit Security Hooks - Code checkers to scan for secrets (private keys, system passwords, cloud access keys, etc.) before committing code to version control.
- Static Source Code Scanning - Scanning source code files (infrastructure templates, application source code) for security vulnerabilities in the build pipeline.
- Security Unit Tests - Custom security unit tests written by the security and engineering teams to provide coverage for abuser stories and negative test cases.
- Dynamic Application Security Testing - Scanning a running application for common security vulnerabilities (OWASP Top 10) and misconfigurations and enforcing acceptance criterial using tools such as Gauntlt / BDD Security.
- Secrets Management - Automating the provisioning and storage of secrets using tools such as Vault, Conjure, AWS KMS, Azure Key Vault.
- Continuous Monitoring - Monitoring the production environment for vulnerabilities, anomalies, and in progress attacks using tools such as statsd, graphite, graphana, etc.
2. What Secure DevOps tools should I use?
The authors of the SANS Institute's DEV540 Secure DevOps & Cloud Application Security course put together a Secure DevOps Toolchain poster. Our SecDevOps Toolchain breaks the SecDevOps process down into 5 key phases with a detailed list of associated tools.
- Pre-Commit: Security controls that can take place before code is checked into version control.
- Commit: Fast, automated security checks that are invoked by continuous integration tools during the build.
- Acceptance: Automated security acceptance tests, functional tests, and out of band security scans that are invoked by continuous integration tools as artifacts are delivered to testing / staging environments.
- Production: Security smoke tests and configuration checks that are invoked by continuous integration tools as artifacts are delivered to the production environment.
- Operations: Continuous security monitoring, testing, auditing, and compliance checks running in production and providing feedback to SecDevOps teams.
3. How important is culture in an organization moving towards SecDevOps?
John Wills, Damon Edwards, and Jez Humble came up with the CALMS to understand and describe DevOps:
- Culture — People come first
- Automation — Rely on tools for efficiency + repeatability
- Lean — Apply Lean engineering practices to continuously improve
- Measurement — Use data to drive decisions and improvements
- Sharing — Share ideas, information and goals across organizational silos
There is a reason that Culture is first on the list. DevOps creates an open, diverse working environment that enables working together to solve problems, empower teams, fail fast, and continuously improve.
Here is the biggest challenge integrating the "Sec" into "DevOps": traditional security cultures are always ready to say "no", fail to share information across the organization, and do not tolerate failure.
This is also evidenced in how CISOs lead their organizations and interact with development teams and the rest of the business. There are three different types of CISOs: the dictator, the collaborator, and the partner.
By saying "no" the dictator doesn't fully consider how security introduces unnecessary friction into existing development and business processes. For SecDevOps to be successful we need to move to be more of a collaborator by understanding engineering practices, process, and tools. Even better, security leaders can become partners by understanding business goals to earn a seat at the executive table.
4. How do you measure the success of SecDevOps?
SecDevOps teams often measure success by monitoring change frequency, deployment success / failure rates, and the mean time to recover (MTTR) from a failure.
For the "Sec" component, assume that a vulnerability assessment or penetration test uncovers a critical patch missing from a system. How long does it take the feedback from the security team to enter the operations workflow, build a new gold image, run automated acceptance tests to validate the changes, and roll the gold image into production? Mean time to recover (MTTR) is a critical metric for measuring success.
Measuring the # of vulnerabilities discovered via automated testing in the pipeline versus vulnerabilities escaping to production. This shows the effectiveness of the pipeline and improvements over time.
Tracking the window of exposure, or how long vulnerabilities remain open in production, provides important metrics for management to see progress.
These metrics also help handle managerial issues as you ask above because can show the process working and helping make the organization more secure.
5. What does "Security as code" mean?
A prerequisite for automation requires all steps to be written into code and checked into version control. Development engineering teams have been writing code since the beginning. Modern operations teams are now writing "infrastructure as code" using tools like Chef, Puppet, and Ansible to create and configure cloud infrastructure, on-premise infrastructure, gold images, and network devices.
Security as code takes this approach a step further by converting manual security and compliance steps into automated, repeatable scripts that can be executed inside a CI pipeline. Security tools are quickly evolving to have APIs and command line interfaces to support "security as code" instead of manually configuring a scanner and pressing a button.
In a SecDevOps world, security tools that cannot be automated through an API or from the command line are not worth purchasing.
Summary
Many security teams jump in too quickly and disrupt the engineering workflows. It is important to tackle one step at a time and start slowly. Security teams must ensure new security steps are working properly, evaluate results, fine tune scanners, and minimize false positives before even considering disrupting the workflow.
To learn more about DevOps and Cloud Security, check out DEV540: Secure DevOps and Cloud Application Security!
Eric Johnson is a co-founder and principal security engineer at Puma Security focusing on modern static analysis product development and DevSecOps automation. His experience includes application security automation, cloud security reviews, static source code analysis, web and mobile application penetration testing, secure development lifecycle consulting, and secure code review assessments.
Previously, Eric spent 5 years as a principal security consultant at an information security consulting firm helping companies deliver secure products to their customers, and another 10 years as an information security engineer at a large US financial institution performing source code audits.
As a Certified Instructor with the SANS Institute, Eric authors information security courses on DevSecOps, cloud security, secure coding, and defending mobile apps. He serves on the advisory board for the SANS Security Awareness Developer training program, delivers security training around the world, and presents security research at conferences including SANS, BlackHat, OWASP, BSides, JavaOne, UberConf, and ISSA.
Eric completed a bachelor's degree in computer engineering and a masters degree in information assurance at Iowa State University, and currently holds the CISSP, GWAPT, GSSP-.NET, and GSSP-Java certifications.
Frank Kim is the founder of ThinkSec, a security consulting and CISO advisory firm. Previously, as CISO at the SANS Institute, Frank led the information risk function for the most trusted source of computer security training and certification in the world. With the SANS Institute, Frank continues to lead the management and software security curricula, helping to develop the next generation of security leaders.
Frank was also executive director of cybersecurity at Kaiser Permanente where he built an innovative security program to meet the unique needs of the nation's largest not-for-profit health plan and integrated health care provider with annual revenue of $60 billion, 10 million members, and 175,000 employees.
Frank holds degrees from the University of California at Berkeley and is the author and instructor of popular courses on strategic planning, leadership, application security, and DevOps. For more, visit frankkim.net.
