One of the advantages of working with SANS is I have an incredible wealth of experience and knowledge to tap into, SANS Instructors.  These are some of the most trained and experienced security professionals in the world.  As such I'm often picking the minds of these poor souls, seeing which ideas I can suck out of them.  David Hoeltzer was recently one of my victims, I was asking him for ideas on how to measure the impact of security awareness.  This is what he had to say. Incident handling and auditing reports are Ccd to the awareness manager. Awareness manager uses these to compile list of issues that are likely awareness problems rather than ineffective controls. Based on these, he creates a simple 10 question multiple choice quiz.  No trick questions.  The right answer is there, along with the most common actual findings from the environment. The quiz is handed out at the very beginning of awareness training.  They have 5 minutes.  The quiz is not collected and not graded. During training, all of these issues are covered.  Nothing obvious, but they are covered. At the end of training the attendees are asked to take exactly the same quiz.  This one is collected as evidence that they "got" what we were saying.  They should all be 100%. Two outcomes that are positive here, one metrics wise, one awareness wise:
  1. When someone changes his answer he realizes that he actually learned something in the awareness training (unbelievable!!!)
  2. We should find that our audit and incident metrics show a decline in occurrences following the training.  Even better, if it's a large organization, we should see that recidivism is dramatically reduced and that exceptions and incidents are continuing to occur in the population that hasn't been trained.
There are some great ideas in here I never thought of.  First, the idea of copying the Security Awareness officer on incident and auditing reports.  This allows the awareness team to track which incidents are human related, identify trends and perhaps dig in a bit more.  I also like David's quiz idea, providing metrics not to the organization but to the actual users.  In other words the quiz before training and after training demonstrates to people that they are learning.  If any organizations are doing something similar I would love to hear how it has worked for you, email me at