Have you heard about the shortage of cybersecurity skills? It’s likely you have, as it’s hard to miss the headlines. According to Cybersecurity Magazine, “there will be 3.5 million unfilled cybersecurity jobs globally by 2021.” Let’s unpack this a bit. What is cybersecurity, apart from being a buzzword overused by some and hated by others? It is a wide collection of loosely related domains — offensive and defensive, deeply technical and policy-related, preventive and corrective.

If we look under the hood of skills shortage, will we find that some domains are harder hit than others? Inevitably so. Software development is one of the areas most starved of security attention. Even though it is a known fact that vulnerable applications are one of the easiest routes for attackers, somehow the industry still gets away with the idea of security as an optional add-on rather than a basic requirement for an acceptable application.

Know Your Options

What are the options for an organization that does not want to contribute to the sad statistics and wants to take security in software development seriously? The range of options runs from training everyone in application security to an adequate degree to setting up a team of full-time security specialists that will somehow support the rest of the development organization. Let’s take a closer look.

The first option, often described as “security is everybody’s job,” is a noble aspiration. Is it realistic for most organizations, though? Even with significant initial investment in training and some ways to maintain and update the skills, the ethos of security will inevitably become diluted with new hires and mostly with the other day-to-day duties of roles such as software developers, QA engineers, operations, product and project management.

The other option, the dedicated team, is yet another silo and organizations have enough of these as it is. Also, remember the skill shortage problem? You will struggle to hire enough of these specialists to support all the development teams sufficiently. On average, organizations report that they manage to have one application security specialist for 100 application developers.

An In-Between Solution

Luckily, we don’t need to pretend there are only these two extreme choices. There is a spectrum between these two points, and organizations can find something that suits their circumstances. The in-between space also calls for T-shaped people — someone who has wide-ranging skills, plus one in-depth technical specialty. (As an aside, this concept evolved to call for pi-shaped people, and even as one brilliant IBMer put it, Cthulhu-shaped people!) Apart from being a striking image, what are these shapes good for when we are trying to solve the problem of security in software development?

Let’s walk through one such in-between solution. Acme Corporation has 20 agile development teams, some degree of automation in their pipeline to production and very worrying pen test results on their hands — full of high-risk issues in their applications. The first goal is to fix the live environment, but after a round of fighting fires, everyone comes out with first degree burns and a firm resolution not to repeat the experience going forward.

The organization made a strategic decision to invest in a secure development life cycle. They explore some training options and, for immediate impact, some hiring. They could hire one full-time security engineer for each team, in the spirit of having an agile cross-functional team. But it is expensive and time-consuming to find and hire 20 specialists. Not to mention the issue of efficiency, or whether or not there is enough security work to keep them occupied full time. And assuming they manage to improve the situation, what will it do for the company’s morale to start firing them for doing their jobs too well?

Enter: The Security Champions Approach

Acme Corporation explores another option — the HR team writes a job description for a ninja-Jedi-superhero security engineer to save them from their troubles. Surely one good person can be found if the money is right. They get some interest, but the interviews go nowhere — potential hires hear about the task of single-handedly keeping 20 teams afloat with all the different technology stacks involved, and lose interest. One of them, though, mentions the concept of security champions on the way out. Cthulhu to the rescue! Acme can have someone on each of their development teams with application security as one of their in-depth specialties.

Crucially, this won’t be their only skill, so it is not necessary to ensure there is enough security work to occupy these champions with nothing but security tasks full time; they can do other tasks in the true spirit of an agile team. In fact, it is crucial for the success of the security champions approach that they do other tasks on the team. This gives them expertise in the context of a specific application, enabling better security decisions. Equally important is the other constraint: Security champions cannot do all the security work on the team. Part of the role is to gradually raise the security skills of everyone else, so pairing and sharing are important.

Since we’ve covered what security champions shouldn’t be doing, let’s cover what they should be doing. They should:

• Inject security into definition of done (DoD), acceptance criteria, coding standards and test cases. This doesn’t mean no one else on the team can suggest these, but the security champion will be the one to raise alarm if security criteria is missing.
• Lead threat modeling sessions.
• Review the requirements for security gaps.
• Coordinate with external or internal pen testers and ensure the results feed into the next iteration.
• Lead introduction of security tools into CI/CD toolchains.
• Lead root cause analysis sessions on security issues found post-release or late in the cycle, and use the outcomes for continuous improvement of toolchains, test suites, development life cycle, etc.
• Keep an eye on security news: new attack techniques, advisories relevant to the technology stack in use, current best practices, etc.
• If there is a centralized, specialized security team, be a link between them and the developers.
• Contribute to cross-teams security knowledge repository.

To ensure the program is successful, Acme Corporation followed the following principles:

• After running the initial secure development training, the trainers were asked to identify potential candidates. However, everyone had a chance to volunteer as a champion.
• The volunteers were assured that these duties will be accounted for in their performance review, and not just expected of them as extra on top of “normal” work.
• To keep the motivation and knowledge exchange, the security champions guild was set up, with some budget for cookies at meetings and continuous learning.
• The importance of security was consistently communicated from the top and a clear escalation route was established in case security champions had concerns about security being consistently deprioritized.

Does the problem faced by Acme Corporation seem familiar to your organization? Keep in mind that it is not likely to get better on its own: New attack techniques on applications are published every day, and unfortunately, some of the old techniques are a recurring theme in OWASP Top-10. Combine with the inevitable transition from gated releases to continuous deployment, and the need to take action and change the current ways of work is clearly urgent.

OWASP has a Security Champions Playbook for introducing security champions program, which is very much in line with IBM’s point of view on the subject. We have helped hundreds of teams perform successful rollouts, both internally and for clients across different industries.