"Can you run me a training report? I need to know where we stand with our training effectiveness program!" For a training manager, this is a double-edged sword. While it is very encouraging that upper management is taking an interest in the training program, focusing merely on the percentage of completion does not measure training effectiveness.
Yes, it is a starting point. Because without the records, it's only a rumor that training happened at all. But relying solely on the metric as a measure of training effectiveness, executive management creates a false sense of security. All it does is confirm that 100% of the training events were documented. Enter a checkmark in the box. But checkmark two needs to occur as well. Can the employees do their assignments according to written procedures? The very procedures they signed for? This is the essence of training effectiveness.
Measuring effectiveness of training requires a "performance" assessment. Examples of employees' performance show up in batch records and executed forms every day. In addition, deviations and CAPAs are areas where employees' ability to perform procedures can be evaluated. During a regulatory inspection, investigators will conduct this "paper" performance audit and observe live real-time performance during the "walking" tour of the facilities. Will the regulators conclude 100% effectively trained?
For starters, it's more than a "Read & Understand" signature. Effective training is really a degree of the transfer of learning back to the job. Are the "100% trained" employees correctly following the procedures one month, three months, or one year later? Or do repeat deviations and training-related CAPAs indicate re-training remediation as corrective action? If it does, this indicates that there is a flaw somewhere in the training process. When enough examples point to a pattern or a trend, training is not effective despite 100% documented signatures.
Concluding that the root cause was poor training without drilling down further does not correct the problem nor does it prevent another occurrence from happening again. A training session involves more than the actual training delivery event. Training is a process and like any other process it has inputs, value added steps and outputs. And all processes exist within organizational context and other management systems. So, an examination of all four components needs to be executed to determine the source of the flaw and where correction should occur.
The cause and effect diagram is a popular problem-solving tool and when used with the 4-M checklist (man, machine, methods, and materials) can provide a structured approach to analyzing all aspects of the training process. (1) Whatever tool is used, questions that concentrate on the training process and the workplace environment need to be asked. For example,
Effectiveness of training can also be affected by the "use it or lose it" phenomenon. If an employee can use the training immediately after, learning transfer increases significantly. The cause analysis needs to include an activities timeline. See Figure 1 - Employee Timeline of Key Activities. The timeline can help determine how much time elapsed from the original date of training to the date the deviation/error occurred and any retraining events as well as SOP revision events. Lack of opportunity to use the training in a timely manner or the lack of additional repetitive practice sessions monitored by a qualified trainer or supervisor can also be a contributing factor. Aligning other organizational events into the timeline can provide additional insight into the barriers of training transfer. For example, a significant change in the procedure went into effect with minimal revision change training (a memo only).Figure 1
If the cause analysis points to the employee, an interview type of discussion is in order. Ask specific questions about the error. Get past the "I don't know why" response by asking about each step of the procedure. Was the entire procedure performed incorrectly or just the last step? Why were they able to perform some steps correctly and the other steps incorrectly? In many interviews such as these, new information comes to light that exposes areas of confusion and ambiguity with the procedure. What starts out as operator error can result in procedure error instead.
If the analysis suggests complications with the qualified trainer (QT), conduct an exploratory interview. Ask the individuals if they want to continue being QTs. Often in these discussions, QTs reveal that they never wanted to be a QT or no longer wish to be a QT and are "not allowed" to opt-out so they let their training performance deteriorate hoping to be removed from the program. Another common issue with QTs is varying delivery of training or inconsistent use of training materials. Without identifying and agreeing on the critical steps from procedures, it is up to the QT to decide what s/he wants to emphasize.
And a lot of the times, the cause analysis will point to organizational barriers. Mary L. Broad and John W. Newstrom (2) in their groundbreaking book, Transfer of Training, uncovered the top three barriers to transfer during their study of training transfer:
The actions of management after training can actually make or break training success. Providing timely and specific feedback with regards to the training is one such technique that doesn't cost any money. Yet, it requires care, thought, and a desire to ensure that employees are successful. Another strategy is to allow for deliberate practice sessions or mini-refresher sessions to reinforce the skills and ensure a successful final performance demonstration (qualification) rather than focus on expediting the qualification event.
So what is the metric? How do we measure transfer of training/learning? It begins with measuring improved job performance. Data is already being collected and used to report on efficiencies, productivity and other key performing indicators. Use what is available. Another powerful set of data are repeat deviations and training-related CAPAs. A decrease in these indicates that employees are making fewer mistakes. Fewer errors are a measure of improved job performance. Training reports can still include "% complete." But now it necessitates effectiveness metrics for improved procedure performance: checkmark two.
Effective training relies on a well-designed training process that utilizes multiple inputs resulting in qualified employees who produce outcomes of value for the organization. The ultimate outcome of training is the development of employees' expertise. The degree of successful transfer back to the work setting is the true measure of effective training, not 100% complete training matrix requirements.
Vivian Bringslimark has more than 20 years of experience as a field practitioner within the Life Sciences community, serving in various professional and management positions as well as consulting internally and externally on diverse assignments. She currently serves as president of HPIS Consulting Inc., partnering with clients to analyze true root causes of human performance gaps and implementing appropriate solutions that align with stated business outcomes to bring about more long-term and predictable performance, resulting in yearly goal achievement and operational excellence. She worked previously for Parexel Consulting and is frequently sought as a guest speaker for training conferences. Reach Vivian at (203)-270-6519 or firstname.lastname@example.org. Visit her website at www.hpisconsulting.com.