When assessing knowledge retention and application, crafting effective test questions can be a pivotal aspect of any training program. In Level 2 evaluation, the distinction between questions that merely test recall and those that assess practical job application can be profound. In this blog, we delve into the art of writing Level 2 evaluation multiple-choice test questions that not only gauge what participants know but also their ability to apply that knowledge in real-world scenarios. By understanding the nuances of constructing job application-focused test questions, trainers and educators can gather invaluable insights into the effectiveness of their training programs. Also, join us as we explore the anatomy of a question and uncover guidelines for creating valid and scientifically sound test questions that truly measure the depth of learning and its practicality.
Don’t miss this intriguing webinar from HRDQ-U
Don’t miss this intriguing webinar from HRDQ-U
Creating Level 2 Quizzes and Tests That Actually Measure Something
Recall vs. Job Application
Writing Level 2 evaluation multiple-choice test questions might seem easy enough. However, there is a big difference between writing test questions that test for what participants KNOW (the mere recall of facts) versus those that test for KNOW-HOW (job application).
Four Examples of Test Questions
Look over the following four examples and note the differences.
Example #1
What does the letter “I” stand for in the ADDIE model?
A. Initiate
B. Implement
C. Improve
D. Investigate
Example #2
You sent a detailed design document to a group of SMEs to solicit their feedback regarding a program design. This is an example of what element in the ADDIE model? A. Analysis
B. Design
C. Development
D. Implementation
E. Evaluation
Example #3
What are the four elements of a high-performing team?
A. Powerful Goals, Frequent Team Parties, Aligned Goals, and Engaged Team Members
B. Powerful Purpose, Aligned Goals, Inclusive Leadership, and Engaged Team Members
C. Powerful Purpose, Aligned Goals, Including Leadership and Engaging Meetings
D. Powerful Purpose, Directive Leadership, Aligned Goals, and Engaged Team Members
Example #4
A supervisor discovered a compliance problem with their team that could substantially negatively impact patients if not reported and corrected immediately. The best leadership style for this situation is:
A. Visionary/Authoritative
B. Directive/Coercive
C. Affiliative
D. Democratic
E. Pacesetting
F. Coaching
The Benefits of Application Questions
As you can see, examples #1 and #3 are recall test questions, and examples #2 and #4 are job application questions. The benefit of job application test questions over recall questions is that they not only provide data regarding whether participants learned the material taught in a training program, but they also provide information regarding whether participants KNOW HOW TO APPLY the new knowledge they learned.
Knowing this can be especially crucial when also conducting a Level 3 evaluation. For example, suppose the Level 3 evaluation results indicate that participants aren’t applying what they learned in a training program. In that case, you’ll have Level 2 test question data showing whether participants learned the information taught in the training program and whether they knew how to apply it.
Consequently, if participants aren’t applying the new information learned, you will know it isn’t because they didn’t learn it and don’t know how to use it. You’ll know that something got in the way AFTER the training that inhibited or prevented the participants from applying what they learned, most likely related to their work environment.
So, Which Is Better – Recall or Job Application?
In short, the most significant improvement you can make when creating Level 2 multiple-choice test questions is to write job application-focused test questions instead of recall questions. The following information describes the main elements of a multiple-choice test question and guidelines for writing valid, scientifically sound multiple-choice job application-focused test questions.
The Anatomy of a Question
Guidelines for writing job application-focused test questions:
Select a learning objective from your target training program.
Think about what learners need to know and do to meet the objective.
Identify real-life situations that reflect the actual application of the program content covered by the objective.
Choose one of the situations and write the stem (the question or problem) in 1 to 3 sentences. Possible stems include:
“What is the most effective method for…?”
“What would happen if…?”
“What is the most critical step in the process?”
“What is the most appropriate solution to this problem?”
“What is the next step to solving this problem?”
“What is needed to solve this problem?”
Identify plausible distracters. Think of things someone might say or do who thought they knew the content associated with this learning objective but didn’t understand.
While writing job application-focused test questions is more complicated than writing recall questions, the data you collect will provide much more valuable insight and make the additional effort worthwhile.
Level 2 Job Application Questions
In conclusion, the power of well-crafted Level 2 evaluation multiple-choice test questions cannot be overstated. These questions serve as a litmus test for not only assessing whether participants have grasped the material but also whether they possess the essential skill of applying their newfound knowledge in practical settings. By following the guidelines outlined in this blog, educators and trainers can shift their focus from mere recall questions to job application-focused inquiries, thus obtaining richer and more actionable data.
Ken Phillips is the founder and CEO of Phillips Associates and the creator of the Predictive Learning Analytics™️ evaluation methodology. He is also a measurement and evaluation master, having spoken and gotten rave reviews – at the ATD International Conference on measuring and evaluating learning issues every year since 2008. He also has presented at the Annual Training Conference and Expo every year since 2013 on similar topics.
Ken has pooled his measurement and evaluation knowledge and experience into a series of presentations explicitly designed for L&D professionals. The presentations are highly engaging, practical, and filled with relevant content most L&D professionals haven’t previously heard. In short, they are not a rehash of traditional measurement and evaluation theory – but fresh ideas and solutions.
Creating Level 2 Quizzes and Tests That Actually Measure Something
Uncover the impact of your training programs with Level 2 evaluation. Learn how well-designed test questions can reveal participants’ mastery of learning.
