Effective level 2 questions: Know-ing versus Know-how

Effective level 2 questions: Know-ing versus Know-how
Share This Post:

When assessing knowledge retention and application, crafting effective test questions can be a pivotal aspect of any training program. In Level 2 evaluation, the distinction between questions that merely test recall and those that assess practical job application can be profound. In this blog, we delve into the art of writing Level 2 evaluation multiple-choice test questions that not only gauge what participants know but also their ability to apply that knowledge in real-world scenarios. By understanding the nuances of constructing job application-focused test questions, trainers and educators can gather invaluable insights into the effectiveness of their training programs. Also, join us as we explore the anatomy of a question and uncover guidelines for creating valid and scientifically sound test questions that truly measure the depth of learning and its practicality.

Don’t miss this intriguing
webinar from HRDQ-U

Don’t miss this intriguing webinar from HRDQ-U

Creating Level 2 Quizzes and Tests That Actually Measure Something

Recall vs. Job application

Writing Level 2 evaluation multiple-choice test questions might seem easy enough. However, there is a big difference between writing test questions that test for what participants KNOW (the mere recall of facts) versus those that test for KNOW-HOW (job application).

“It’s all in how you write the questions.” -Ken Phillips

Four examples of test questions

Look over the following four examples and note the difference.

Example #1

What does the letter “I” stand for in the ADDIE model?
A. Initiate
B. Implement
C. Improve
D. Investigate

Example #2

You sent a detailed design document to a group of SMEs to solicit their feedback regarding a program design. This is an example of what element in the ADDIE model?
A. Analysis
B. Design
C. Development
D. Implementation
E. Evaluation

Example #3

What are the four elements of a High Performing Team?
A. Powerful Goals, Frequent Team Parties, Aligned Goals, and Engaged Team Members
B. Powerful Purpose, Aligned Goals, Inclusive Leadership, and Engaged Team Members
C. Powerful Purpose, Aligned Goals, Including Leadership and Engaging Meetings
D. Powerful Purpose, Directive Leadership, Aligned Goals, and Engaged Team Members

Example #4

A supervisor discovered a compliance problem with their team that could substantially negatively impact patients if not reported and corrected immediately. The best leadership style for this situation is:
A. Visionary/Authoritative
B. Directive/Coercive
C. Affiliative
D. Democratic
E. Pacesetting
F. Coaching

The benefits of application questions

As you can see, examples #1 and #3 are recall test questions, and examples #2 and #4 are job application questions. The benefit of job application test questions over recall questions is that they not only provide data regarding whether participants learned the material taught in a training program, but they also provide information regarding whether participants KNOW HOW TO APPLY the new knowledge they learned.

Knowing this can be especially crucial when also conducting a Level 3 evaluation. For example, suppose the Level 3 evaluation results indicate that participants aren’t applying what they learned in a training program. In that case, you’ll have Level 2 test question data showing whether participants learned the information taught in the training program and whether they knew how to apply it.

Consequently, if participants aren’t applying the new information learned, you will know it isn’t because they didn’t learn it and don’t know how to use it. You’ll know that something got in the way AFTER the training that inhibited or prevented the participants from applying what they learned, most likely related to their work environment.

So, which is better? Recall or Job application

In short, the most significant improvement you can make when creating Level 2 multiple-choice test questions is to write job application-focused test questions instead of recall questions. The following information describes the main elements of a multiple-choice test question and guidelines for writing valid, scientifically sound multiple-choice job application-focused test questions.

The anatomy of a question

anatomy of level 2 training questions

Guidelines for writing job application-focused test questions

1. Select a learning objective from your target training program.

2. Think about what learners need to know and do to meet the objective.

3. Identify real-life situations that reflect the actual application of the program content covered by the objective.

4. Choose one of the situations and write the stem (the question or problem) in 1 to 3 sentences.

Some possible item shells for writing the stem include:
• “What is the most effective method for…?”
• “What would happen if…?”
• “What is the most critical step in the process?”
• “What is the most appropriate solution to this problem?”
• “What is the next step to solving this problem?”
• “What is needed to solve this problem?”

5. Identify plausible distracters. Think of things someone might say or do who thought they knew the content associated with this learning objective but didn’t understand.

While writing job application-focused test questions is more complicated than writing recall questions, the data you collect will provide much more valuable insight and make the additional effort worthwhile.

Level 2 job application questions

In conclusion, the power of well-crafted Level 2 evaluation multiple-choice test questions cannot be overstated. These questions serve as a litmus test for not only assessing whether participants have grasped the material but also whether they possess the essential skill of applying their newfound knowledge in practical settings. By following the guidelines outlined in this blog, educators and trainers can shift their focus from mere recall questions to job application-focused inquiries, thus obtaining richer and more actionable data.

Author
Headshot of Ken Phillips
Ken Phillips

Ken Phillips is the founder and CEO of Phillips Associates  and the creator of the Predictive Learning Analytics™️ evaluation methodology. He is also a measurement and evaluation master, having spoken and gotten rave reviews—at the ATD International Conference on measuring and evaluating learning issues every year since 2008. He also has presented at the Annual Training Conference and Expo every year since 2013 on similar topics.

Ken has pooled his measurement and evaluation knowledge and experience into a series of presentations explicitly designed for L&D professionals. The presentations are highly engaging, practical, and filled with relevant content most L&D professionals haven’t previously heard. In short, they are not a rehash of traditional measurement and evaluation theory—but fresh ideas and solutions.

Email Ken with any questions at ken@phillipsassociates.com  ⇗

Follow Ken on LinkedIn ⇗

Recommended Webinar
Creating Level 2 Quizzes and Tests That Actually Measure Something

In this informative, highly engaging webinar, you will learn tips and best practices on how to write Level 2 quizzes and tests that produce valued data. Emphasis is also given to writing training questions that measure job application, not mere recall of facts.

Creating level 2 quizzes and training questions
Related Learning

Discover more HRDQ-U performance management webinars

Access more valuable HRDQ-U content by Ken

Explore HRDQ Store’s performance management training resources ⇗

 

Related Products

Behavioral Interviews Customizable Course

Behavioral Interviews Customizable Course - HRDQ - related to training questions

This is a collection of downloadable and customizable courses for virtual and classroom instructor-led training, plus self-study learning. Each title includes digital document files for facilitator and participant materials provided in unlocked Microsoft Office format. You can download, customize, and deliver training today!

Get 10% Off Today!
Use Code KENP10

More HRDQ-U Blog Posts

The ROI Methodology®, the most recognized approach to ROI evaluation, provides organizations with a process that can cut across organizational

High-performing teams are essential for the success and growth of any business.

Related Topics
Career development
Career Development
Business coaching webinar
Coaching
Creativity and innovation skills training
Creativity and Innovation
Webinar customer service
Customer Service
Effective level 2 questions: Know-ing versus Know-how
Decision Making
Diversity and inclusion webinars
Diversity and Inclusion
Effective level 2 questions: Know-ing versus Know-how
Leadership
PM webinars
Project Management
Log In