How can you tell if security awareness training is working?

‘If you don’t know where you’re heading, you’ll likely end up someplace else,’

– American baseball legend, Yogi Berra.

Do you know where your security training programme is headed? Will you be able to tell when you’ve arrived?

Measuring the effectiveness of security awareness training can be difficult. In a field where things change daily and best practice is constantly evolving, some say security is more journey than destination anyway.

But even processes in constant motion can be observed and measured. You just need to decide what’s important to quantify, the indicators to look for, how to track them, and for how long.

Metrics for measuring training effectiveness often include the following:

• Employee feedback
• Simulated phishing exercises
• Counting security incidents before and after training
• Regular participation

These can tell you something, but not enough. Selecting a set of metrics is only a start. You need to …

Have a strategy

Having a cybersecurity strategy with agreed objectives is really step 1,2, an 3 for measuring the effectiveness of security awareness training. You have to know what the threats are to the business and the information assets that you need to defend. You have to be clear on the policies that need to be complied with or better understood, what behaviours you want to promote, and what a reasonable target outcome should be in 48 hours, one month, one quarter and so on. The starting point is to decide what success looks like.

Ask the right questions

To formulate or update a security strategy you have to have a clear picture of the current cyber state of play in your workplace. We would argue that one of the objectives of security awareness training is to promote a security-aware culture.

In an organisational context culture means knowledge, behaviours, and attitudes. To understand those in detail we would ask the following:

• What or how much do your employees know about staying safe online?
• How much importance do people place on security, and what do they think about current policies and procedures?
• How confident are employees in their abilities where online security is concerned?
• How do members of staff actually behave when confronted by an attempted breach?

If you’re interested in finding out more about creating a positive security culture, our MD, Eddie Whittingham, wrote an e-book that you might find useful, entitled ‘Making Cyber Security Sexy: How to get your employees to care about cyber-security’ (not for the faint hearted!).

Choose your metrics

Measure engagement — People’s engagement should be monitored effectively through online security awareness training. Rather than testing an employee’s IQ, organisations should try to establish engagement with the subject matter.  The facts are that security awareness isn’t complicated – and therefore testing via quizzes or sneaky questions doesn’t mean they’ll be alert when they next see a phishing email.  The real challenge here is keeping your users engaged with the ever-changing landscape.  Instead, judge users by regular, consistent engagement – resulting in their heightened security awareness, rather than just intelligence.  It’s the ongoing awareness of, say, preventing a malicious visitor to piggyback through a security gate on your ID card swipe, that will be key – not whether they’ve answered 9/10 questions correctly.

Measure behaviour — Measuring behaviour is best achieved using simulations of actual breach attempts at a breach. Monitoring how people respond to simulations gives you a real-world metric of security behaviour.

Measure attitudes — Measuring something as intangible as attitudes is hard – but not impossible.   Considering how engaged users are with your security awareness programme is a good indicator of their attitude towards security generally, especially when combined with the measured behaviour detailed above.  Don’t forget to look wider than just the individual too – it’s often clear to see department attitudes towards your risks.

Attitudes can be further analysed through anonymous surveys, which can give you an idea of why people take risky actions like clicking links in a pop up ad or leaving their desks while still logged in to the network.

Set a timescale

To get an accurate read of how effective security awareness training is measurements need to be taken on or just before the day training commences.  This is typically helped along by a bench-marking exercise prior to commencing your training.  As the programme unfolds you can then take or review measurements at regular intervals.  This might mean quarterly simulated phishing exercises or monthly office walks.

It’s important that security awareness isn’t just seen as a quick fix, either.  Building a positive security awareness culture takes time – and it also needs nurturing to help maintain it.  Providing regular, fun and bite-size security awareness training can help your programme maintain effectiveness over time – key to helping reduce your risk in the long term.

Such timelines, regular measurements and reviewing user engagement, will tell you where and under what circumstances employees require more support.

Isn’t it just about counting the number of incidents?

If you can say there were ten incidents in the six months before security awareness training began and only three in the six months after, that would be wonderful.

The problem is that stopping threats is a daily challenge and that number fails to take into account the potential severity of a breach. Waiting ‘til quarter end to find out that your programme didn’t deliver the desired outcomes – is waiting too long.

The Ponemon Institute describes the key dimensions of information security effectiveness along these lines:

• Uptime: being able to bear a cyber-attacks without significant disruption to normal operations.
• Compliance: being in compliance with all relevant regulations and laws.
• Threat containment: being able to quickly detect and block external threats like phishing and malvertising
• Cost efficiency: being able to manage cyber investments at sustainable levels of investment
• Preventability: being able to prevent or quickly detect insider threats.
• Policy enforcement: being able to monitor and ensure that policies and procedures are being followed

Using Ponemon’s calculus, the higher the score in these areas, the stronger your security posture and the more likely you are to minimise the impact of a breach and lower its costs – or avoid breaches altogether. That’s an objective measure of security training effectiveness.

What’s interesting about the six dimensions is how closely related they are to security-aware behaviours rather than information technology. Ponemon studies show again and again that when the behaviours of a security aware culture become embedded in everyday workplace activity, the incidence and cost of non-compliance plummets.

One thing we can measure easily is failure

Research from Forrester shows that that only a quarter of workers know what to do when a breach occurs, while up to seven percent say openly that they ignore or try and work around security policies.

It’s hitting objectives in these areas that will really prove if your security awareness training has been effective – or not.

The question is, can you really afford to learn the hard way by measuring failure?  The best step is to start building a positive security culture, today.

Want to learn more about measuring the effectiveness of security awareness training?  Sign up for a free demo and find out how we’re already helping organisations just like yours.