Tuesday, October 16, 2018
Official Blog of the One and Only Kirkpatrick® Minimize

Welcome to the Kirkpatrick Partners blog, where we strive to share the most pertinent training and evaluation related tips and thoughts each week. Please check back often, or simply subscribe to our RSS feed to be sure you never miss an update.

The Academy of Program/Project and Engineering Leadership (APPEL) shares their success using the Kirkpatrick Model to evaluate their program effectiveness for The National Aeronautics and Space Administration (NASA). Read more in this guest feature reprinted with permission.

APPEL Goes Gold

APPEL supports NASA by providing development and training for the agency’s technical workforce. To reinforce the value of its offerings, the Academy recently brought home the gold. 

APPEL takes the design, development, and delivery of courses extremely seriously, as they are critical components of training. Just as important is confirming the value of the training. The Academy accomplishes this by evaluating the effectiveness of its offerings. Evaluation allows APPEL to assess whether its programs meet learner needs and objectives, whether knowledge and skills are imparted as intended, whether desired changes in performance are achieved, and what organizational benefits result from the training.

APPEL evaluates the effectiveness of all of its courses to confirm their value to NASA’s workforce. Here, participants in the course Project Management for non-Technical Managers, held in September 2016 at NASA Headquarters, explore the tools and techniques that are critical to planning and managing any project successfully. Credit: NASA/Dan Daly

To ensure the validity of its course evaluation measures, the Academy’s evaluation strategy is aligned with the Kirkpatrick Model: the foremost model in the world for analyzing and evaluating the results of training and educational programs. As part of this effort, members of the APPEL team—including the Academy’s entire Curriculum team—are Kirkpatrick Certified Professionals – Gold Level.

The Kirkpatrick Model measures training effectiveness across four levels:

Level 1: Reaction

  • Evaluations at this level assess the degree to which participants find the training favorable, engaging, and relevant to their jobs. For example: Did the participants find the training relevant? Did they like and enjoy the training?

Level 2: Learning

  • Level 2 evaluations focus on the degree to which participants acquire the intended knowledge, skills, attitude, confidence, and commitment based on their participation in the training. For example: Did the participant’s knowledge increase as a result of the course?

Level 3: Behavior

  • Evaluations at the third level measure the degree to which participants apply what they learned in the training after returning to their jobs. APPEL assesses the application of learnings from the course 45 days after training completion.

Level 4: Results

  • Level 4 evaluations examine the degree to which targeted outcomes occur as a result of the training and subsequent support and reinforcement. For a business or organization, key performance indicators might be quantifiable aspects of organizational performance, such as a reduction in number of complaints or increase in quality ratings. For NASA, APPEL looks at the collective contribution and impact of specific courses on the agency’s ultimate goal: mission success.

“APPEL is doing quality work that is in alignment with the recommendations of the U.S. Office of Personnel Management’s Training Evaluation Field Guide. Through the efforts of organizations like APPEL, the U.S. government is lighting the way for all organizations with their leadership position in creating and demonstrating program value,” said Dr. Jim Kirkpatrick, co-owner of Kirkpatrick Partners.

Course participants emphasize the value of APPEL courses to their professional development and ability to support the NASA mission. According to one individual who attended APPEL’s Assertiveness Training for Technical Professionals, “The assertiveness course follows me on an almost daily basis. First, it helped me feel better that I’m not alone in terms of being reluctant to speak up. I remind myself to be more assertive every day by asking myself, “What’s the worst that could happen if I speak up?” It’s having an impact on my personal life and my work life—and that’s particularly important because a great deal of my sense of self-worth comes from my work. Because I’m asking more questions at work, we are finding more problems and bringing them to light earlier.”

Data reinforces the feedback from participants, confirming that APPEL courses have a positive impact on employee job performance. For example, at the start of an APPEL course, on average 57% of attendees can perform key course objectives with assistance. By the end of the course, that number increased to 91%. Additionally, when surveyed 45 days after completing an APPEL course, more than 75% of respondents report they are successfully applying the critical behaviors learned in the course.


“By implementing the Kirkpatrick enhanced evaluation approach, we are able to demonstrate the impact of APPEL courses on job performance and illustrate how APPEL contributes to mission success at NASA,” said APPEL Director and NASA Chief Knowledge Officer Roger Forsgren. Furthermore, aligning with the Kirkpatrick Model ensures the Academy retains elite status among industry peers.

APPEL will continue to implement the Kirkpatrick evaluation model throughout fiscal year (FY) 2018. Level 1 and 2 evaluations will be applied to all courses, while Level 3 and 4 evaluations will be conducted across 11 courses in APPEL’s new Building and Leading NASA Teams curriculum category.

Join the Discussion

Do you have aspirations of having your entire team become Kirkpatrick certified? How will you use the Kirkpatrick Model to help demonstrate your program's effectiveness? Let us know. Here are some ways to join the discussion. 

Log in and leave a comment

Comment on this feature on Facebook
Start a LinkedIn discussion

Don't Miss a Feature

Register with us to receive Kirkpatrick Quick Tips delivered to your inbox each Wednesday. 

Additional Resources

Kirkpatrick Four Levels® Evaluation Certification Program - Bronze Level

Kirkpatrick's Four Levels of Training Evaluation

Creating ROE: The End Is the Beginning

Kirkpatrick Foundational Principles

Actions: Permalink | Comments (0) |


There are currently no comments, be the first to post one. Please log-in to comment.
Join the Kirkpatrick Community Minimize

Register with us for FREE samples of our Blended Evaluation Plan® form.

(If you have already registered with us, simply log in and visit the Resources tab to access the samples - no need to re-register.)


You'll also receive immediate access to our Resource Library, and weekly updates from Kirkpatrick Partners.

Click here for more information about the materials available in our Resource Library and to see a sample of our weekly newsletter.
Deadlines & Specials Minimize
Early Bird Savings & Registration Deadlines

Kirkpatrick program deadlines and specials

Bronze Certification
November 2, 9, 16, 30 and December 7
Online (from the U.S.)

November 6-7, 2018
Washington, D.C.


Silver Certification
November 8, 2018
Washington, D.C.
December 4, 6, 13, 2018
Online (from the U.S.)
Save $300 Until October 19

calendarClick here for a complete calendar of Kirkpatrick events

Latest News Minimize

More 2019 Kirkpatrick Certification Program Dates Announced 
October 8, 2018

Additional program dates have been added to our 2019 event calendar with a few more coming soon. Click here to find out more.

Kirkpatrick Book Named to Must-Read List for Trainers
September 24, 2018

The best-selling book, Kirkpatrick's Four Levels of Training Evaluation, has been named to a list of must-reads for training professionals. Click here for more.

Kirkpatrick's Four Levels of Training Evaluation Hit #1 on Amazon
November 6, 2016

What a success our book, Kirkpatrick's Four Levels of Training Evaluation, has been! It hit #1 in training books on amazon.com yesterday! Thank you, thank you, thank you to those of you who purchased it, reviewed it, clicked "yes" that you found reviews helpful, or simply visited the page.

It's consistently ranked in the top 25 of the Training books section on Amazon. Discover a complete blueprint for implementing the model in a way that truly maximizes business results. And delve into Jim and Wendy's new findings that, together with Don Kirkpatrick's work, create the New World Kirkpatrick Model, a powerful methodology that melds people with metrics.

Use the New World Model to better train people, improve the way you work, and, ultimately, contribute to your organization's most crucial goals without breaking the budget.  Click here to learn more.

Copyright 2009 - 2018 Kirkpatrick Partners, LLC. All rights reserved.