Tags:
Stephen J, who is a member of our software security mailing list, asked a while back, "Do you have any recommendations on static source code scanners?" James Jardine and I started talking and came up with the following tips.
There are so many commercial static analysis tools from vendors like Armorize, Checkmarx, Coverity, Fortify (HP), Klocwork, IBM, and Veracode that it's hard to recommend a specific product. Instead we'd like to focus on seven tips that can help you maximize your selection.
1) Test before you buy
This probably sounds obvious but, assuming you haven't purchased anything yet, definitely do a bake off and have the vendor run the code against your actual apps. Do *not* simply run the tool on a vendor supplied sample app as the quality of the results, surprisingly, can vary quite a bit across different tools and code bases. Just keep in mind that some vendors will try to avoid this so they can better control the test or, simply, to keep you from getting a free scan.
2) Determine your budget
It's really important to understand your budget. The cost of commercial tools varies greatly between vendors and you should be aware of how much you are willing to spend. This includes the cost of the tool, ongoing maintenance, and even things like additional RAM for your build servers (yes, it can take a lot of RAM). And don't forgot to budget for the time people spend setting up, configuring, and reviewing results from the tool.
3) Prepare to spend a lot of time on it
Usually, anything worth doing takes time. Test-driven development and writing thorough unit tests don't happen on their own. Similarly, using a static analysis tool to find security issues is an investment. It needs both management and developer buy-in to succeed when time consuming issues like false positives, tweaking rules, and product upgrades come up.
4) Cloud or Internal
Are you ok with uploading your source code, or binaries, to a third party site to do the analysis? This is only a question you and your company can answer, but it is an important one. Not only do you have to consider sharing your code, but if the solution is in the cloud, what about integration with your build process. Do you want to have a solution that can be built into your nightly build process? Tools that are housed internally usually lend themselves better to customization. Do you have the staff and capabilities to extend the tool so that it works better for your applications?
5) What type of support do you need?
Some tools come with much more support than others. One tool may have support personnel available to go over the results of your scan with you, so you understand them, whereas others may not have this type of support. Depending on your in-house skill set this could be a big deciding factor.
6) Check references
Ask other colleagues (not the vendors) who use the tool and get their feedback. These are the ones that will give you honest answers. Keep in mind that even though a specific vendor works well for one application or company it doesn't mean that it's the right fit for your situation.
7) Research the products
Make sure that the solution you are considering is the best fit for you. You will definitely want to make sure that the vendor/product you choose supports your development platform and your development process.
Every application is different and it's impossible for an outsider to recommend the correct static analysis tool without having intimate knowledge of your application and processes. Given the investment in time and money, choosing a vendor and product is a difficult yet important decision. Do your due diligence and research before making a choice.
Take some time to lay out your current landscape and answer some of these questions:
- What language(s) do you develop in?
- What source control system do you use?
- What other types of technologies are used in your application (Ajax, jQuery, Silverlight, Flash, mobile, etc)?
- What skill sets do you have in house when it comes to application security?
- Do you have people that can quickly learn how to operate the scanner? Can they understand the output?
- What type of support do you want from the vendor?
- Are you willing to submit your code to an external site, or does your policy require that it not be shared outside the company boundaries?
- What is your budget for a static analyzer?
- How will the technology work with the people and processes that you have in house?
- Hopefully, all these questions don't change your opinion about needing a static analysis tool and make it a little easier to actually select a tool to use.
Thanks for reading!