homepage
Open menu
Go one level top
  • Train and Certify
    Train and Certify

    Immediately apply the skills and techniques learned in SANS courses, ranges, and summits

    • Overview
    • Courses
      • Overview
      • Full Course List
      • By Focus Areas
        • Cloud Security
        • Cyber Defense
        • Cybersecurity and IT Essentials
        • DFIR
        • Industrial Control Systems
        • Offensive Operations
        • Management, Legal, and Audit
      • By Skill Levels
        • New to Cyber
        • Essentials
        • Advanced
        • Expert
      • Training Formats
        • OnDemand
        • In-Person
        • Live Online
      • Course Demos
    • Training Roadmaps
      • Skills Roadmap
      • Focus Area Job Roles
        • Cyber Defence Job Roles
        • Offensive Operations Job Roles
        • DFIR Job Roles
        • Cloud Job Roles
        • ICS Job Roles
        • Leadership Job Roles
      • NICE Framework
        • Security Provisionals
        • Operate and Maintain
        • Oversee and Govern
        • Protect and Defend
        • Analyze
        • Collect and Operate
        • Investigate
        • Industrial Control Systems
      • European Skills Framework
    • GIAC Certifications
    • Training Events & Summits
      • Events Overview
      • Event Locations
        • Asia
        • Australia & New Zealand
        • Latin America
        • Mainland Europe
        • Middle East & Africa
        • Scandinavia
        • United Kingdom & Ireland
        • United States & Canada
      • Summits
    • OnDemand
    • Get Started in Cyber
      • Overview
      • Degree and Certificate Programs
      • Scholarships
    • Cyber Ranges
  • Manage Your Team
    Manage Your Team

    Build a world-class cyber team with our workforce development programs

    • Overview
    • Why Work with SANS
    • Group Purchasing
    • Build Your Team
      • Team Development
      • Assessments
      • Private Training
      • Hire Cyber Professionals
      • By Industry
        • Health Care
        • Industrial Control Systems Security
        • Military
    • Leadership Training
  • Security Awareness
    Security Awareness

    Increase your staff’s cyber awareness, help them change their behaviors, and reduce your organizational risk

    • Overview
    • Products & Services
      • Security Awareness Training
        • EndUser Training
        • Phishing Platform
      • Specialized
        • Developer Training
        • ICS Engineer Training
        • NERC CIP Training
        • IT Administrator
      • Risk Assessments
        • Knowledge Assessment
        • Culture Assessment
        • Behavioral Risk Assessment
    • OUCH! Newsletter
    • Career Development
      • Overview
      • Training & Courses
      • Professional Credential
    • Blog
    • Partners
    • Reports & Case Studies
  • Resources
    Resources

    Enhance your skills with access to thousands of free resources, 150+ instructor-developed tools, and the latest cybersecurity news and analysis

    • Overview
    • Webcasts
    • Free Cybersecurity Events
      • Free Events Overview
      • Summits
      • Solutions Forums
      • Community Nights
    • Content
      • Newsletters
        • NewsBites
        • @RISK
        • OUCH! Newsletter
      • Blog
      • Podcasts
      • Summit Presentations
      • Posters & Cheat Sheets
    • Research
      • White Papers
      • Security Policies
    • Tools
    • Focus Areas
      • Cyber Defense
      • Cloud Security
      • Digital Forensics & Incident Response
      • Industrial Control Systems
      • Cyber Security Leadership
      • Offensive Operations
  • Get Involved
    Get Involved

    Help keep the cyber community one step ahead of threats. Join the SANS community or begin your journey of becoming a SANS Certified Instructor today.

    • Overview
    • Join the Community
    • Work Study
    • Teach for SANS
    • CISO Network
    • Partnerships
    • Sponsorship Opportunities
  • About
    About

    Learn more about how SANS empowers and educates current and future cybersecurity practitioners with knowledge and skills

    • SANS
      • Overview
      • Our Founder
      • Awards
    • Instructors
      • Our Instructors
      • Full Instructor List
    • Mission
      • Our Mission
      • Diversity
      • Scholarships
    • Contact
      • Contact Customer Service
      • Contact Sales
      • Press & Media Enquiries
    • Frequent Asked Questions
    • Customer Reviews
    • Press
    • Careers
  • Contact Sales
  • SANS Sites
    • GIAC Security Certifications
    • Internet Storm Center
    • SANS Technology Institute
    • Security Awareness Training
  • Search
  • Log In
  • Join
    • Account Dashboard
    • Log Out
  1. Home >
  2. Blog >
  3. Ask the Expert - Nick Galbreath
SANS DevSecOps

Ask the Expert - Nick Galbreath

June 4, 2012

Nick Galbreath is director of engineering at Etsy, overseeing groups handling fraud, security, authentication and internal tools. Over the last 18 years, Nick has held leadership positions at a number of social and e-­commerce companies, including Right Media, UPromise, Friendster, and Open Market, and has consulted for many more. He is the author of "Cryptography for Internet and Database Applications" (Wiley), and was awarded a number of patents in the area of social networking. Today he shares his insights on Continuous Deployment and security.

1. In Continuous Deployment, developers push software to production several times a day. Please explain how this reduces risk in development and operations.

Before we talk about risk, let's talk about what Continuous Deployment requires. It needs both a mechanism and a policy for success. The mechanism is being able to deploy quickly without some manual process (i.e. that guy who magically deploys code for you). Every organization needs this now. The policy is how, what and when you choose to deploy. This is highly company and risk dependent, and you'll need to adjust this to your needs. You are unlikely to want to deploy, say, database schema changes as often as CSS changes due to the risk involved.

However one important part of the policy is emphasizing "small changes." For complex Web-based, data-driven systems, there is no amount of QA in your development environment that can guarantee that the code won't cause an operational or security issue. None! So given that, would you rather have a big bang change with dozens of authors and hundreds of files being modified each with multiple change sets? Good luck rolling that back and debugging when things go wrong! Or would you rather have a few very small changes that can be code-reviewed "by inspection" and is easy to undo, each with an audit trail. To me the latter is much less risky for both site availability and security.

Also note that while there is risk in making changes, there is also risk in not making changes. Not being able to push out timely security fixes and patches leaves your organization exposed during the (long) release cycle. With a Continuous Deployment process, exercised multiple times per day, you know you can push a security fix when the need arises.

2. Many Appsec professionals are concerned that Agile development teams build software too fast to be secure. Continuous Deployment seems to accelerate this even more. How do security controls and checks fit into Continuous Deployment, and what controls and checks need to be done differently to keep up with the pace?

I've seen no study saying that agile methods are any better or worse for security than any other methodology — security can be ignored equally well in all methodologies! Waterfall with full SDLC has been producing security flaws for decades, so I'm not sure that is the gold standard. I suspect that all methodologies, properly implemented, have about the same change-to-bug ratio in a release (i.e. number of lines of code changed to number of new bugs created in a single code deployment).

Continuous Deployment is about movement of code from development to production. It doesn't mean you don't do review for security, architecture and operations and don't write tests and don't do peer code reviews. You still need to do all that. Continuous Deployment also means holding developers more responsible for their code quality, because developers can immediately see the results of their work. There is little to no buffer between them and their code running.

As mentioned to make this work, developers need to learn an "incremental programming" style involving pushing lots of small changes. This is not natural to most developers and takes a while to learn. A lot of the changes in a proper Continuous Deployment process do nothing. This dark code isn't executed in production. It just sits there. Why? Since we know if something does go wrong on deployment we can rule out these dark changes, focusing our efforts on code that actually does something. In the meantime, that dark code is now in source control for all to see and review, and we know has compiled and passed basic automated QA tests. For code that is active, there are a number of strategies to manage risk. Lighting up the dark code is normally done under a configuration flag, so it's easy to identify the change and to easily turn off if something occurs. Features are "ramped up", by first testing internally, then exposing to 1% of the site's users, then 10% and so on to make sure everything is working right before going to 100%. If a problem is found, it can just as easily be ramped down.

So far I talked a lot of risk and development. As for security and Continuous Deployment, the summary is "secure by default" and "alert on when it's not." It's easier said than done. The following aren't really specific to Continuous Deployment, but perhaps they are more important:

  • Isolate, insulate or ban functions that are easy to misuse. Cryptographic functions are a prime example of this. Most of these APIs are too low-level, so they'll need wrapping into something that mere mortals can use. Then alert on the raw function. If new crypto starts showing up in the code base, it probably needs review (and ideally this code was already reviewed in your software development life cycle).
  • Segment out sensitive code and alert on changes to it. Your password storage mechanism shouldn't be changing that often. If it is, find out why.
  • Use static analysis, even for dynamic languages. Prevent those silly bugs before they go out into production. In C, those silly bugs are frequently security vulnerabilities. Static analysis for dynamic languages such as PHP is becoming more popular as well (e.g. see http://slidesha.re/KzTfLy). testing easy for developers and automatic for deployment. Set up a continuous integration stack such as Jenkins and have it run every time. This also means spending time on making your tests run fast. If they are slow, no-one will want to run them.
  • Make untrusted user inputs secure by default. Right now, every web platform gives you the raw user input. Unless you explicitly escape it, it's insecure. This is impossible to manage. If possible flip it around so inputs are escaped, and if you need the raw data, you have to explicitly un-escape the data. In other words, the inputs are secure by default. This is complicated and probably deserves its own paper.

3. Where do teams that want to move faster and use Continuous Deployment need to start if they want to do this in a safe and secure way? What tools and practices do they need to have in place to succeed?

To get started with Continuous Deployment, the very first thing to do is to "put a button on it." Get your release engineer (or whoever deploys code) to start automating the release process so it's a "one button" process or simple shell script that anyone can do. The release engineer should not view this as job-threatening as it actually makes him more valuable. Then work on making the time it takes to deploy shorter and shorter. If it hasn't been done already, this process is likely to cause a whole cascade of site operations improvements: automation, standardization, monitoring of your stack. All of which help improve security. Then work on how fast you can upgrade key servers (e.g. Apache HTTPD) or even the operating system. You want all of this to be painless, since if it's not, you'll postpone it, or worse not do it all and miss a key security patch.

A basic requirement is to log all changes: what is the change, who made the change, and who pushed out the change to production. You'll probably want some conventions around when people can push and on peer review. You'll also want to make visible site operations and when site changes are happening. You'll want to visibly correlate changes to problems.

The previous section answered what is needed to make Continuous Deployment work securely before code goes out. Now it's time to see what happens after code is deployed. Again, none of these are unique to Continuous Deployment:

  • Make security visible. Graph potential SQLi and XSS attacks. These are normally quite common, and turns security into a visible event for all to see. It's a fantastic tool for security education, as well as knowing when someone new is probing you.
  • Use attacker-driven testing. Use previous results to guide how you do manual or semi-automated testing. You'll find attackers don't scan everything but focus on particular regions of your website. Maybe they found something.
  • Monitor core dumps. Your server shouldn't be core dumping very often, if ever. If it is, maybe you need to patch or upgrade. Or maybe someone found a buffer overflow.
  • Monitor "server 500" errors. Can you reproduce them? Probably it's just a QA problem but maybe it's someone scanning your system and they found something.
  • Monitor database SQL syntax errors. SQLi breaches don't happen in a vacuum. An attacker needs to spend a lot of time probing your application for the appropriate entry point. These often end up generating SQL syntax errors.
  • Look for other anomalies. New really long URLs coming in? New parameters showing up in a query string? Maybe a code review needs to happen.

It takes a while to get the technology, the code, and the culture ready for secure Continuous Deployment. Perhaps the most important effect of Continuous Deployment is the transition of operations and security teams typically known for blocking change and "saying no" into service organizations to enable change.

Share:
TwitterLinkedInFacebook
Copy url Url was copied to clipboard
Subscribe to SANS Newsletters
Receive curated news, vulnerabilities, & security awareness tips
United States
Canada
United Kingdom
Spain
Belgium
Denmark
Norway
Netherlands
Australia
India
Japan
Singapore
Afghanistan
Aland Islands
Albania
Algeria
American Samoa
Andorra
Angola
Anguilla
Antarctica
Antigua and Barbuda
Argentina
Armenia
Aruba
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belize
Benin
Bermuda
Bhutan
Bolivia
Bonaire, Sint Eustatius, and Saba
Bosnia And Herzegovina
Botswana
Bouvet Island
Brazil
British Indian Ocean Territory
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Christmas Island
Cocos (Keeling) Islands
Colombia
Comoros
Cook Islands
Costa Rica
Croatia (Local Name: Hrvatska)
Curacao
Cyprus
Czech Republic
Democratic Republic of the Congo
Djibouti
Dominica
Dominican Republic
East Timor
East Timor
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Ethiopia
Falkland Islands (Malvinas)
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
French Southern Territories
Gabon
Gambia
Georgia
Germany
Ghana
Gibraltar
Greece
Greenland
Grenada
Guadeloupe
Guam
Guatemala
Guernsey
Guinea
Guinea-Bissau
Guyana
Haiti
Heard And McDonald Islands
Honduras
Hong Kong
Hungary
Iceland
Indonesia
Iraq
Ireland
Isle of Man
Israel
Italy
Jamaica
Jersey
Jordan
Kazakhstan
Kenya
Kiribati
Korea, Republic Of
Kosovo
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon
Lesotho
Liberia
Liechtenstein
Lithuania
Luxembourg
Macau
Macedonia
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Marshall Islands
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Micronesia, Federated States Of
Moldova, Republic Of
Monaco
Mongolia
Montenegro
Montserrat
Morocco
Mozambique
Myanmar
Namibia
Nauru
Nepal
Netherlands Antilles
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
Norfolk Island
Northern Mariana Islands
Oman
Pakistan
Palau
Palestine
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Pitcairn
Poland
Portugal
Puerto Rico
Qatar
Reunion
Romania
Russian Federation
Rwanda
Saint Bartholemy
Saint Kitts And Nevis
Saint Lucia
Saint Martin
Saint Vincent And The Grenadines
Samoa
San Marino
Sao Tome And Principe
Saudi Arabia
Senegal
Serbia
Seychelles
Sierra Leone
Sint Maarten
Slovakia
Slovenia
Solomon Islands
South Africa
South Georgia and the South Sandwich Islands
South Sudan
Sri Lanka
St. Helena
St. Pierre And Miquelon
Suriname
Svalbard And Jan Mayen Islands
Swaziland
Sweden
Switzerland
Taiwan
Tajikistan
Tanzania
Thailand
Togo
Tokelau
Tonga
Trinidad And Tobago
Tunisia
Turkey
Turkmenistan
Turks And Caicos Islands
Tuvalu
Uganda
Ukraine
United Arab Emirates
United States Minor Outlying Islands
Uruguay
Uzbekistan
Vanuatu
Vatican City
Venezuela
Vietnam
Virgin Islands (British)
Virgin Islands (U.S.)
Wallis And Futuna Islands
Western Sahara
Yemen
Yugoslavia
Zambia
Zimbabwe

By providing this information, you agree to the processing of your personal data by SANS as described in our Privacy Policy.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Tags:
  • DevSecOps

Related Content

Blog
DevSecOops_Thumbnail_FINAL.png
DevSecOps
February 10, 2023
From DevOops! to DevSecOps
Part one of a four-part blog series about what’s needed to shake, shimmy, and shift left
Profile_Picture.jpeg
Stacy Dunn
read more
Blog
powershell_option_340x340.jpg
Cyber Defense, DevSecOps, Digital Forensics and Incident Response, Cybersecurity and IT Essentials, Penetration Testing and Red Teaming
July 15, 2022
Month of PowerShell: Working with PowerShell Log Files
In this article we'll look at how we can leverage PowerShell's object-passing pipeline to parse and retrieve data from an IIS web server log file.
370x370_Joshua-Wright.jpg
Joshua Wright
read more
Blog
DevSecOps
May 4, 2015
2015 State of Application Security: Closing the Gap
The 2015 SANS State of Application Security Analyst Paper and webcasts are complete. This year, Jim Bird, the lead author of the SANS Application Security Survey series, Frank Kim, and I all participated in writing the questions, analyzing the results, drafting the paper, and preparing the webcast...
Eric_Johnson_370x370.png
Eric Johnson
read more
  • Register to Learn
  • Courses
  • Certifications
  • Degree Programs
  • Cyber Ranges
  • Job Tools
  • Security Policy Project
  • Posters & Cheat Sheets
  • White Papers
  • Focus Areas
  • Cyber Defense
  • Cloud Security
  • Cybersecurity Leadership
  • Digital Forensics
  • Industrial Control Systems
  • Offensive Operations
Subscribe to SANS Newsletters
Receive curated news, vulnerabilities, & security awareness tips
United States
Canada
United Kingdom
Spain
Belgium
Denmark
Norway
Netherlands
Australia
India
Japan
Singapore
Afghanistan
Aland Islands
Albania
Algeria
American Samoa
Andorra
Angola
Anguilla
Antarctica
Antigua and Barbuda
Argentina
Armenia
Aruba
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belize
Benin
Bermuda
Bhutan
Bolivia
Bonaire, Sint Eustatius, and Saba
Bosnia And Herzegovina
Botswana
Bouvet Island
Brazil
British Indian Ocean Territory
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Christmas Island
Cocos (Keeling) Islands
Colombia
Comoros
Cook Islands
Costa Rica
Croatia (Local Name: Hrvatska)
Curacao
Cyprus
Czech Republic
Democratic Republic of the Congo
Djibouti
Dominica
Dominican Republic
East Timor
East Timor
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Ethiopia
Falkland Islands (Malvinas)
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
French Southern Territories
Gabon
Gambia
Georgia
Germany
Ghana
Gibraltar
Greece
Greenland
Grenada
Guadeloupe
Guam
Guatemala
Guernsey
Guinea
Guinea-Bissau
Guyana
Haiti
Heard And McDonald Islands
Honduras
Hong Kong
Hungary
Iceland
Indonesia
Iraq
Ireland
Isle of Man
Israel
Italy
Jamaica
Jersey
Jordan
Kazakhstan
Kenya
Kiribati
Korea, Republic Of
Kosovo
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon
Lesotho
Liberia
Liechtenstein
Lithuania
Luxembourg
Macau
Macedonia
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Marshall Islands
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Micronesia, Federated States Of
Moldova, Republic Of
Monaco
Mongolia
Montenegro
Montserrat
Morocco
Mozambique
Myanmar
Namibia
Nauru
Nepal
Netherlands Antilles
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
Norfolk Island
Northern Mariana Islands
Oman
Pakistan
Palau
Palestine
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Pitcairn
Poland
Portugal
Puerto Rico
Qatar
Reunion
Romania
Russian Federation
Rwanda
Saint Bartholemy
Saint Kitts And Nevis
Saint Lucia
Saint Martin
Saint Vincent And The Grenadines
Samoa
San Marino
Sao Tome And Principe
Saudi Arabia
Senegal
Serbia
Seychelles
Sierra Leone
Sint Maarten
Slovakia
Slovenia
Solomon Islands
South Africa
South Georgia and the South Sandwich Islands
South Sudan
Sri Lanka
St. Helena
St. Pierre And Miquelon
Suriname
Svalbard And Jan Mayen Islands
Swaziland
Sweden
Switzerland
Taiwan
Tajikistan
Tanzania
Thailand
Togo
Tokelau
Tonga
Trinidad And Tobago
Tunisia
Turkey
Turkmenistan
Turks And Caicos Islands
Tuvalu
Uganda
Ukraine
United Arab Emirates
United States Minor Outlying Islands
Uruguay
Uzbekistan
Vanuatu
Vatican City
Venezuela
Vietnam
Virgin Islands (British)
Virgin Islands (U.S.)
Wallis And Futuna Islands
Western Sahara
Yemen
Yugoslavia
Zambia
Zimbabwe

By providing this information, you agree to the processing of your personal data by SANS as described in our Privacy Policy.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
  • © 2023 SANS™ Institute
  • Privacy Policy
  • Contact
  • Careers
  • Twitter
  • Facebook
  • Youtube
  • LinkedIn