homepage
Open menu Go one level top
  • Train and Certify
    • Get Started in Cyber
    • Courses & Certifications
    • Training Roadmap
    • Search For Training
    • Online Training
    • OnDemand
    • Live Training
    • Summits
    • Cyber Ranges
    • College Degrees & Certificates
    • NICE Framework
    • DoDD 8140
    • Specials
  • Manage Your Team
    • Overview
    • Security Awareness Training
    • Voucher Program
    • Private Training
    • Workforce Development
    • Skill Assessments
    • Hiring Opportunities
  • Resources
    • Overview
    • Reading Room
    • Webcasts
    • Newsletters
    • Blog
    • Tip of The Day
    • Posters
    • Top 25 Programming Errors
    • The Critical Security Controls
    • Security Policy Project
    • Critical Vulnerability Recaps
    • Affiliate Directory
  • Focus Areas
    • Blue Team Operations
    • Cloud Security
    • Digital Forensics & Incident Response
    • Industrial Control Systems
    • Leadership
    • Offensive Operations
  • Get Involved
    • Overview
    • SANS Community
    • CyberTalent
    • Work Study
    • Instructor Development
    • Sponsorship Opportunities
    • COINS
  • About
    • About SANS
    • Why SANS?
    • Instructors
    • Cybersecurity Innovation Awards
    • Contact
    • Frequently Asked Questions
    • Customer Reviews
    • Press Room
  • Log In
  • Join
  • Contact Us
  • SANS Sites
    • GIAC Security Certifications
    • Internet Storm Center
    • SANS Technology Institute
    • Security Awareness Training
  • Search
  1. Home >
  2. Blog >
  3. Ask the Expert - Nick Galbreath
SANS DevSecOps

Ask the Expert - Nick Galbreath

June 4, 2012

Nick Galbreath is director of engineering at Etsy, overseeing groups handling fraud, security, authentication and internal tools. Over the last 18 years, Nick has held leadership positions at a number of social and e-­commerce companies, including Right Media, UPromise, Friendster, and Open Market, and has consulted for many more. He is the author of "Cryptography for Internet and Database Applications" (Wiley), and was awarded a number of patents in the area of social networking. Today he shares his insights on Continuous Deployment and security.

1. In Continuous Deployment, developers push software to production several times a day. Please explain how this reduces risk in development and operations.

Before we talk about risk, let's talk about what Continuous Deployment requires. It needs both a mechanism and a policy for success. The mechanism is being able to deploy quickly without some manual process (i.e. that guy who magically deploys code for you). Every organization needs this now. The policy is how, what and when you choose to deploy. This is highly company and risk dependent, and you'll need to adjust this to your needs. You are unlikely to want to deploy, say, database schema changes as often as CSS changes due to the risk involved.

However one important part of the policy is emphasizing "small changes." For complex Web-based, data-driven systems, there is no amount of QA in your development environment that can guarantee that the code won't cause an operational or security issue. None! So given that, would you rather have a big bang change with dozens of authors and hundreds of files being modified each with multiple change sets? Good luck rolling that back and debugging when things go wrong! Or would you rather have a few very small changes that can be code-reviewed "by inspection" and is easy to undo, each with an audit trail. To me the latter is much less risky for both site availability and security.

Also note that while there is risk in making changes, there is also risk in not making changes. Not being able to push out timely security fixes and patches leaves your organization exposed during the (long) release cycle. With a Continuous Deployment process, exercised multiple times per day, you know you can push a security fix when the need arises.

2. Many Appsec professionals are concerned that Agile development teams build software too fast to be secure. Continuous Deployment seems to accelerate this even more. How do security controls and checks fit into Continuous Deployment, and what controls and checks need to be done differently to keep up with the pace?

I've seen no study saying that agile methods are any better or worse for security than any other methodology — security can be ignored equally well in all methodologies! Waterfall with full SDLC has been producing security flaws for decades, so I'm not sure that is the gold standard. I suspect that all methodologies, properly implemented, have about the same change-to-bug ratio in a release (i.e. number of lines of code changed to number of new bugs created in a single code deployment).

Continuous Deployment is about movement of code from development to production. It doesn't mean you don't do review for security, architecture and operations and don't write tests and don't do peer code reviews. You still need to do all that. Continuous Deployment also means holding developers more responsible for their code quality, because developers can immediately see the results of their work. There is little to no buffer between them and their code running.

As mentioned to make this work, developers need to learn an "incremental programming" style involving pushing lots of small changes. This is not natural to most developers and takes a while to learn. A lot of the changes in a proper Continuous Deployment process do nothing. This dark code isn't executed in production. It just sits there. Why? Since we know if something does go wrong on deployment we can rule out these dark changes, focusing our efforts on code that actually does something. In the meantime, that dark code is now in source control for all to see and review, and we know has compiled and passed basic automated QA tests. For code that is active, there are a number of strategies to manage risk. Lighting up the dark code is normally done under a configuration flag, so it's easy to identify the change and to easily turn off if something occurs. Features are "ramped up", by first testing internally, then exposing to 1% of the site's users, then 10% and so on to make sure everything is working right before going to 100%. If a problem is found, it can just as easily be ramped down.

So far I talked a lot of risk and development. As for security and Continuous Deployment, the summary is "secure by default" and "alert on when it's not." It's easier said than done. The following aren't really specific to Continuous Deployment, but perhaps they are more important:

  • Isolate, insulate or ban functions that are easy to misuse. Cryptographic functions are a prime example of this. Most of these APIs are too low-level, so they'll need wrapping into something that mere mortals can use. Then alert on the raw function. If new crypto starts showing up in the code base, it probably needs review (and ideally this code was already reviewed in your software development life cycle).
  • Segment out sensitive code and alert on changes to it. Your password storage mechanism shouldn't be changing that often. If it is, find out why.
  • Use static analysis, even for dynamic languages. Prevent those silly bugs before they go out into production. In C, those silly bugs are frequently security vulnerabilities. Static analysis for dynamic languages such as PHP is becoming more popular as well (e.g. see http://slidesha.re/KzTfLy). testing easy for developers and automatic for deployment. Set up a continuous integration stack such as Jenkins and have it run every time. This also means spending time on making your tests run fast. If they are slow, no-one will want to run them.
  • Make untrusted user inputs secure by default. Right now, every web platform gives you the raw user input. Unless you explicitly escape it, it's insecure. This is impossible to manage. If possible flip it around so inputs are escaped, and if you need the raw data, you have to explicitly un-escape the data. In other words, the inputs are secure by default. This is complicated and probably deserves its own paper.

3. Where do teams that want to move faster and use Continuous Deployment need to start if they want to do this in a safe and secure way? What tools and practices do they need to have in place to succeed?

To get started with Continuous Deployment, the very first thing to do is to "put a button on it." Get your release engineer (or whoever deploys code) to start automating the release process so it's a "one button" process or simple shell script that anyone can do. The release engineer should not view this as job-threatening as it actually makes him more valuable. Then work on making the time it takes to deploy shorter and shorter. If it hasn't been done already, this process is likely to cause a whole cascade of site operations improvements: automation, standardization, monitoring of your stack. All of which help improve security. Then work on how fast you can upgrade key servers (e.g. Apache HTTPD) or even the operating system. You want all of this to be painless, since if it's not, you'll postpone it, or worse not do it all and miss a key security patch.

A basic requirement is to log all changes: what is the change, who made the change, and who pushed out the change to production. You'll probably want some conventions around when people can push and on peer review. You'll also want to make visible site operations and when site changes are happening. You'll want to visibly correlate changes to problems.

The previous section answered what is needed to make Continuous Deployment work securely before code goes out. Now it's time to see what happens after code is deployed. Again, none of these are unique to Continuous Deployment:

  • Make security visible. Graph potential SQLi and XSS attacks. These are normally quite common, and turns security into a visible event for all to see. It's a fantastic tool for security education, as well as knowing when someone new is probing you.
  • Use attacker-driven testing. Use previous results to guide how you do manual or semi-automated testing. You'll find attackers don't scan everything but focus on particular regions of your website. Maybe they found something.
  • Monitor core dumps. Your server shouldn't be core dumping very often, if ever. If it is, maybe you need to patch or upgrade. Or maybe someone found a buffer overflow.
  • Monitor "server 500" errors. Can you reproduce them? Probably it's just a QA problem but maybe it's someone scanning your system and they found something.
  • Monitor database SQL syntax errors. SQLi breaches don't happen in a vacuum. An attacker needs to spend a lot of time probing your application for the appropriate entry point. These often end up generating SQL syntax errors.
  • Look for other anomalies. New really long URLs coming in? New parameters showing up in a query string? Maybe a code review needs to happen.

It takes a while to get the technology, the code, and the culture ready for secure Continuous Deployment. Perhaps the most important effect of Continuous Deployment is the transition of operations and security teams typically known for blocking change and "saying no" into service organizations to enable change.

Share:
TwitterLinkedInFacebook
Copy url Url was copied to clipboard
Subscribe to SANS Newsletters
Join the SANS Community to receive the latest curated cybersecurity news, vulnerabilities, and mitigations, training opportunities, plus our webcast schedule.
United States
Canada
United Kingdom
Spain
Belgium
Denmark
Norway
Netherlands
Australia
India
Japan
Singapore
Afghanistan
Aland Islands
Albania
Algeria
American Samoa
Andorra
Angola
Anguilla
Antarctica
Antigua and Barbuda
Argentina
Armenia
Aruba
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belize
Benin
Bermuda
Bhutan
Bolivia
Bonaire, Sint Eustatius, and Saba
Bosnia And Herzegovina
Botswana
Bouvet Island
Brazil
British Indian Ocean Territory
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Christmas Island
Cocos (Keeling) Islands
Colombia
Comoros
Cook Islands
Costa Rica
Croatia (Local Name: Hrvatska)
Curacao
Cyprus
Czech Republic
Democratic Republic of the Congo
Djibouti
Dominica
Dominican Republic
East Timor
East Timor
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Ethiopia
Falkland Islands (Malvinas)
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
French Southern Territories
Gabon
Gambia
Georgia
Germany
Ghana
Gibraltar
Greece
Greenland
Grenada
Guadeloupe
Guam
Guatemala
Guernsey
Guinea
Guinea-Bissau
Guyana
Haiti
Heard And McDonald Islands
Honduras
Hong Kong
Hungary
Iceland
Indonesia
Iraq
Ireland
Isle of Man
Israel
Italy
Jamaica
Jersey
Jordan
Kazakhstan
Kenya
Kingdom of Saudi Arabia
Kiribati
Korea, Republic Of
Kosovo
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon
Lesotho
Liberia
Liechtenstein
Lithuania
Luxembourg
Macau
Macedonia
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Marshall Islands
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Micronesia, Federated States Of
Moldova, Republic Of
Monaco
Mongolia
Montenegro
Montserrat
Morocco
Mozambique
Myanmar
Namibia
Nauru
Nepal
Netherlands Antilles
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
Norfolk Island
Northern Mariana Islands
Oman
Pakistan
Palau
Palestine
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Pitcairn
Poland
Portugal
Puerto Rico
Qatar
Reunion
Romania
Russian Federation
Rwanda
Saint Bartholemy
Saint Kitts And Nevis
Saint Lucia
Saint Martin
Saint Vincent And The Grenadines
Samoa
San Marino
Sao Tome And Principe
Senegal
Serbia
Seychelles
Sierra Leone
Sint Maarten
Slovakia (Slovak Republic)
Slovenia
Solomon Islands
South Africa
South Georgia and the South Sandwich Islands
South Sudan
Sri Lanka
St. Helena
St. Pierre And Miquelon
Suriname
Svalbard And Jan Mayen Islands
Swaziland
Sweden
Switzerland
Taiwan
Tajikistan
Tanzania
Thailand
Togo
Tokelau
Tonga
Trinidad And Tobago
Tunisia
Turkey
Turkmenistan
Turks And Caicos Islands
Tuvalu
Uganda
Ukraine
United Arab Emirates
United States Minor Outlying Islands
Uruguay
Uzbekistan
Vanuatu
Vatican City
Venezuela
Vietnam
Virgin Islands (British)
Virgin Islands (U.S.)
Wallis And Futuna Islands
Western Sahara
Yemen
Yugoslavia
Zambia
Zimbabwe

Tags:
  • DevSecOps

Related Content

Blog
InstructorSpotlight_370x370.png
Cloud Security, DevSecOps
November 20, 2020
Instructor Spotlight: Brandon Evans, SEC510 Lead Author
Get to know SANS Certified Instructor and SANS Cloud Ace, Brandon Evans.
BrandonEvans_Headshot_370x370.png
Brandon Evans
read more
Blog
DevSecOps
October 14, 2018
Exploring the DevSecOps Toolchain
The authors of the SANS Institute's DEV540 Secure DevOps & Cloud Application Security course created the Cloud Security and DevSecOps Best Practices poster to help security teams create a methodology for integrating security into the DevOps workflow. As you can see, the poster breaks DevOps down...
Eric_Johnson_370x370.png
Eric Johnson
read more
Blog
DevSecOps
September 13, 2018
Your Secure DevOps Questions Answered
As SANS prepares for the 2nd Annual Secure DevOps Summit, Co-Chairs Frank Kim and Eric Johnson are tackling some of the common questions they get from security professionals who want to understand how to inject security into the DevOps pipeline, leverage leading DevOps practices, and secure DevOps...
Eric_Johnson_370x370.png
Eric Johnson
read more
  • Register to Learn
  • Courses
  • Certifications
  • Degree Programs
  • Cyber Ranges
  • Job Tools
  • Security Policy Project
  • Posters
  • The Critical Security Controls
  • Focus Areas
  • Blue Team Operations
  • Cloud Security
  • Cybersecurity Leadership
  • Digital Forensics
  • Industrial Control Systems
  • Offensive Operations
Subscribe to SANS Newsletters
Join the SANS Community to receive the latest curated cybersecurity news, vulnerabilities, and mitigations, training opportunities, plus our webcast schedule.
United States
Canada
United Kingdom
Spain
Belgium
Denmark
Norway
Netherlands
Australia
India
Japan
Singapore
Afghanistan
Aland Islands
Albania
Algeria
American Samoa
Andorra
Angola
Anguilla
Antarctica
Antigua and Barbuda
Argentina
Armenia
Aruba
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belize
Benin
Bermuda
Bhutan
Bolivia
Bonaire, Sint Eustatius, and Saba
Bosnia And Herzegovina
Botswana
Bouvet Island
Brazil
British Indian Ocean Territory
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Christmas Island
Cocos (Keeling) Islands
Colombia
Comoros
Cook Islands
Costa Rica
Croatia (Local Name: Hrvatska)
Curacao
Cyprus
Czech Republic
Democratic Republic of the Congo
Djibouti
Dominica
Dominican Republic
East Timor
East Timor
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Ethiopia
Falkland Islands (Malvinas)
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
French Southern Territories
Gabon
Gambia
Georgia
Germany
Ghana
Gibraltar
Greece
Greenland
Grenada
Guadeloupe
Guam
Guatemala
Guernsey
Guinea
Guinea-Bissau
Guyana
Haiti
Heard And McDonald Islands
Honduras
Hong Kong
Hungary
Iceland
Indonesia
Iraq
Ireland
Isle of Man
Israel
Italy
Jamaica
Jersey
Jordan
Kazakhstan
Kenya
Kingdom of Saudi Arabia
Kiribati
Korea, Republic Of
Kosovo
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon
Lesotho
Liberia
Liechtenstein
Lithuania
Luxembourg
Macau
Macedonia
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Marshall Islands
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Micronesia, Federated States Of
Moldova, Republic Of
Monaco
Mongolia
Montenegro
Montserrat
Morocco
Mozambique
Myanmar
Namibia
Nauru
Nepal
Netherlands Antilles
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
Norfolk Island
Northern Mariana Islands
Oman
Pakistan
Palau
Palestine
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Pitcairn
Poland
Portugal
Puerto Rico
Qatar
Reunion
Romania
Russian Federation
Rwanda
Saint Bartholemy
Saint Kitts And Nevis
Saint Lucia
Saint Martin
Saint Vincent And The Grenadines
Samoa
San Marino
Sao Tome And Principe
Senegal
Serbia
Seychelles
Sierra Leone
Sint Maarten
Slovakia (Slovak Republic)
Slovenia
Solomon Islands
South Africa
South Georgia and the South Sandwich Islands
South Sudan
Sri Lanka
St. Helena
St. Pierre And Miquelon
Suriname
Svalbard And Jan Mayen Islands
Swaziland
Sweden
Switzerland
Taiwan
Tajikistan
Tanzania
Thailand
Togo
Tokelau
Tonga
Trinidad And Tobago
Tunisia
Turkey
Turkmenistan
Turks And Caicos Islands
Tuvalu
Uganda
Ukraine
United Arab Emirates
United States Minor Outlying Islands
Uruguay
Uzbekistan
Vanuatu
Vatican City
Venezuela
Vietnam
Virgin Islands (British)
Virgin Islands (U.S.)
Wallis And Futuna Islands
Western Sahara
Yemen
Yugoslavia
Zambia
Zimbabwe
  • © 2021 SANS™ Institute
  • Privacy Policy
  • Contact
  • Twitter
  • Facebook
  • Youtube
  • LinkedIn