Training Evaluation Rubrics

In my book, Creating Training for Law Enforcement, a key point I emphasize is that training materials are living documents that need regular updating. The two most important documents to regularly maintain and update are the Instructor Guide (IG) and your assessment materials. Before you can really know if your IG and assessments are actually delivering what you expect, you need to put it all through something called “validation.”

Validation is putting your materials through stress testing to make sure they are accomplishing their goals – your IG to make sure the timing, delivery, activities, and content meet the goals of the course and assessments that they are measuring what you intend to measure.

This is something I learned in the military, and I am a firm proponent of all courses being validated before they are presented to the public. All of my courses are validated before I promote them heavily, as all training, especially law enforcement training, should be.

There has been a lot of interest in the Training Evaluation Program I am starting. It has taken a lot longer than I had hoped to get to the point of validating the first course with my initial rubric, but I trust the process. I finished the first iteration of the rubric in January of 2025, reviewed it several times and made sure I had criteria and objectives firmly in place.

I then had six people I trust go through it and provide their feedback. I ended up making a number of changes and some of the feedback actually caused me to consider the basic philosophy of what I am ultimately trying to achieve here. It was all good stuff. I am fortunate to have people I trust to tell me the truth and attack the weakness of my materials honestly. I wouldn’t have it any other way.

So with confidence I had a pretty dialed-in rubric already, I attended Valor Threat Protection’s course “Lone Protector – Solo Active Shooter Response” (the review is here). The CEO, Jamie Gissel, volunteered his course as one of the validation points. I attended his course without him having reviewed or seen the first version of the rubric because I wanted a blind review. Future validations will have the benefit of seeing the results of this first validation, Jamie did not.

This was intentional – I didn’t want to “poison the well” if the rubric was way off and have him adjust training to a rubric that was flawed, in case it was.

And it definitely was.

This is why validation is necessary – every plan is perfect until the first shot is fired.

The result of the first rubric (image 1) was way off what my experience said it should be. I have attended hundreds of courses in multiple industries and have reviewed hundreds of courses, over 60 for a national certification program.

Objectively, the course is outstanding; well-designed, well-executed, excellent instructors. Objectively, the rubric failed.

I assumed that the course would score high when I filled out the rubric but was stunned when I saw that the results didn’t equal the expectation. This first version was divided into two sections – Course and Instructor. I evaluate them separately because instructor skills are not specifically tied to course creation skills. You can have a great instructor with a trash course and vice versa.

There are three sheets in this rubric – one for the course, one for Jamie Gissel and one for Mike Campofernando of Grayman Defense Consulting who partners with Valor for this course. Then there is a pivot table for all the math.

The course scored a 3.6 out of five for stars, Jamie and Mike both scored 5.0 for stars. There was something very wrong.

Fortunately, Jamie is an abject professional who wanted a real objective evaluation of his course to find places to improve. He took the score as it was, but I reassured him the rubric was broken.

What I realized as I was seeing the scores populate was that I had erroneously combined two parts into one with the Course score – I had documentation mixed with the design of the course. I also had elements of scenario-based training and testing mixed, and I needed more N/A options. His course is unique and made it clear that my rubric was conflating a lot of standards into areas where they were inappropriately applied.

The reason documentation dragged down the course score is simple – overwhelmingly, most LE instructors are not trained on how to create quality course materials, how to document, or even why it is exceptionally important to document a lot of the parts of training. Yet, there are a lot of well designed courses that are documented poorly. That doesn’t mean it is a mediocre class.

The new version has three sections now – Documentation, Design, and Instructor.

The next version was a lot more accurate with what I expected where the course should be, but as I was filling it out using the scores from the first version, I saw additional things missing, and still some elements that needed to be rearranged. (image 2)

I added a column for N/A elements – like if a course uses scenario-based testing instead of a written test, the written test criteria needs to be excepted out of the possible score and not count against an actual score. There are times where a course doesn’t need to have slide decks, so they shouldn’t count against the class, and so on.

All three sections have different criteria for scenario-based activities – some are documentation-related (are they well-scripted, are there roleplayer cards), some are design (are they realistic, are they feasible) and some are instructor-based (did they demonstrate first, did they provide feedback).

And then there were some things missing that needed adding – like safety. If the design or the instructor is unsafe, it is an auto-failure. If they violate two very important case laws, it reduces the score by 25% for each one, for a total of a 50% deduction in points.

And there were some items that needed further expansion – for example, was research, observation, and analysis provided, in both the documentation, design, and instructor sheets.

The result was a third version and the final version  – for this first validation. (image 3) With the extensive changes we made to the first version, it requires three more validations. Hopefully, the next validation will have minor tweaks, if any.

This is why validating is so important. We have to stress test our training materials before we put people through them. They need to be validated for accomplishing the elements they need.

Jamie’s questions and concerns also helped me hone explanations of this whole program. When he saw he received five stars as an instructor, he was at first concerned – he doesn’t believe he is a perfect instructor. Neither do I, nor do I believe anyone is or will be. It helped me define what five stars means – it isn’t a measure of perfection, it is a measure of expertise.

A five-star rating only means he is an advanced and very professional instructor. The rubric has to measure objective standards, not personality. You can be an advanced instructor and also be someone I wouldn’t want to share a dinner with. Likewise, you could be a mediocre instructor that I would consider a great friend. We have to remove personality from rating instructors – one person’s poison is another person’s punch.

The five-star ratings are based on a cumulative set of criteria from basic to complex; from concrete to abstract. They get points if something is present, not if it is something I like or dislike. As the knowledge and skills present themselves, they either get points or they don’t.

The final score with the new rubric is something that both Jamie and I agree is far more reflective of what he has with this class. Documentation – 3.6; Design – 4.9; the instructor scores changed a little, but they are high enough that they both remained at a 5 stars each. He feels he learned a lot about it, especially with collaborating on this rubric, and sees what he can focus on for further developing this course.

I have said in a previous post – the test and standards exposes one’s true character – those with character who get scores they don’t like, knuckle down, fix it, and improve. Those with poor character get angry at the test and complain.

This Training Evaluation program will be exposing character and shedding light on courses and instructors. Exactly as it is meant to do.