As the lead of the Product Security Assurance team at EMC, I am often asked to talk about our software security practices. While previously we have shared our practices with industry presentations, SAFECode papers, etc., I thought now is as good a time as ever to write a blog post to discuss software security evolution at EMC.
Today, I want to provide insight into how EMC began our journey in this domain, what our strategy has been and what we consider best practices for teams at EMC to adopt as part of the process of building products. In the future I will provide more details on each of these practices.
Any good story starts with when, where and how it began. When you look around the industry, you have some clear watershed moments of when and how companies started taking software security more seriously. The famous Bill Gates memo from January 2002 is one of those moments in the software security evolution. About the same time, EMC started to transition from hardware into a software company, with acquisitions of Documentum, Legato and VMware. Our story on software security began as part of this transition.
How do we respond to security vulnerabilities?
The first step we took was to create a policy and process to respond to security vulnerabilities. We researched, talked to other software vendors, became smarter on industry expectations related to vulnerability response, created and internally published our process for vulnerability response in 2004. This was very timely as the first public vulnerability on EMC products was reported to us by a researcher in 2005; we were ready to respond. We already had set clear expectations for EMC engineering teams on their roles and responsibilities for dealing with security vulnerabilities in their products in a timely manner. You can learn more about our vulnerability response process here.
How do we integrate security best practices in the product lifecycle?
A good vulnerability response process is just one side of the coin. We also made sure we were doing everything we can to not introduce these vulnerabilities in our products. A very large cross section of our customer base at the time was in regulated markets such as finance and healthcare. What this meant was EMC products had to meet our customers’ compliance requirements. Also, since EMC is an amalgamation of different technology types, any prescription we put together on product security needed to be applicable to products built using a myriad of technology.
We decided to create a broad but prescriptive set of requirements for product security based on the most common software security mistakes and also the regulatory compliance needs of our customers. The first version of our Product Security Policy (PSP) was created in 2005. We treat the PSP and the underlying standards as a product with a clear roadmap and make changes to it based on the evolving threat landscape and are currently on revision 3 of the policy and are working on version 4.
We took our PSP and in 2007 created clear, underlying activities to be adopted. At different points in the development lifecycle our teams must meet the requirements of our PSP. We rolled this out in the structure of EMC’s Security Development Lifecycle (SDL), which was modeled in some ways on the Microsoft SDL. We learned from Microsoft and adapted their SDL to the culture, requirements and DNA of EMC. Some of the elements of our SDL include:
- Comprehensive, repeatable and measureable secure product lifecycle practices that go across requirements, design, development, testing and sustainment and inclusion of a root cause analysis for all vulnerabilities reported to us post GA/release
- Secure Design requirements to help minimize security vulnerabilities or abuse of functionality across Authentication, Authorization, Accountability, Cryptography and Design Principles
- Secure Implementation requirements that cover secure software engineering practices that minimize the risk of occurrence of security vulnerabilities in product code such as validation issues, various injection vulnerabilities, web and C/C++ coding requirements
How do we ensure our products are following the SDL?
Performing a mandatory Pre-GA security assessment and risk review of every product is baked into EMC’s Enterprise Product Lifecycle. This aids the EMC product teams with identifying risks as they develop, analyzing risks, determining the priority in which to apply resources to treatment of risks and ensuring appropriate treatment is taken to correct or avoid the impact of risk.
We have policy, standards and requirements, how do we get them adopted by the engineering workforce?
We also quickly realized as we started rolling out our policy that to create a security-aware culture across the entire engineering community required an educated engineering workforce. We put in place a role-based security engineering curriculum to train our engineering workforce on the elements of our SDL, job-specific security best practices, common security vulnerabilities that they should be aware of, language based secure coding practices and how to design and test products to prevent the introduction of vulnerabilities in products.
Build or leverage?
People also ask us if we have built internal tools and resources to make our standards easier to adopt by EMC product teams. The answer is yes and no. Yes, we have tools, training and other resources that we rollout to EMC product teams (e.g., we have internal portals/wikis that cover everything that an engineer may need on interpreting the world of software security), but we haven’t focused on creating proprietary or internal-only tools.
When we rolled out the SDL there were valuable industry references and commercial tools available and we integrated them into our practices. We use static analysis tools, vulnerability analysis tools, dynamic testing tools, third party penetration testing, etc. throughout our lifecycle. We use MITRE’s Common Weakness Enumeration (CWE), OWASP top 10 and CWE/ SANS Top 25 Most Dangerous Software Errors as a reference and for validation. We contribute heavily to industry efforts such as the CWE/ SANS Top 25 project, Building Security in Maturity Model, and SAFECode.
We also had the luxury to learn from what was available and have found areas of innovation. As an example, we created our own methodology for threat modeling that we call “threat modeling for the commoners.” Instead of threat modeling ourselves to death, we created a Threat Library to use as a baseline from where we can move forward. This helps developers understand threat modeling in a scalable manner that is easy to comprehend and allows them to think beyond the threats we present to them from our library. Also, our prescriptive design standard calls out when a security function needs to be performed and we do not wait for threat modeling to be initiated to detect whether it has been performed or not. Our approach was published by IEEE Privacy & Security Magazine:“Developer-Driven Threat Modeling: Lessons Learned in the Trenches”. We now are taking our Threat Modeling success forward into a library-based approach to security testing.
Beyond tools required in any good secure software practice we also took an approach after the RSA acquisition in 2006 to build a set of software, standards, specifications, and designs for common software security elements such as authentication, authorization, audit and accountability, cryptography, and key management using state-of-the-art RSA technology. Our engineering teams integrate these into their products for better integration with customer security architectures. These have been built as toolkits by a Common Security Engineering team to enable centralized development of security functions which helps us reduce mistakes commonly made by non-security experts who build security functionality from scratch.
How do we measure success?
At the end of the day we look for the engineering workforce to embrace our standards and policies. At EMC, we depend on security advisors, influencers and champions to help carry the mission of security. We support those with a personal interest in security and collaborate with them. Developers often know the importance of securing products and are empowered to take the lead. All standards and policies aside, it is this ownership that is the most important measure of success; without it my team and our mission cannot be successful.
We used four cornerstones to build our program: People, Policy, Process & Technology. My current job is to make sure we strike the right balance as we continue to build products our customers can trust.