Effective level 2 questions: Know-ing versus Know-how

Recall vs. Job application

Writing Level 2 evaluation multiple-choice test questions might seem easy enough. However, there is a big difference between writing test questions that test for what participants KNOW (the mere recall of facts) versus those that test for KNOW-HOW (job application).

“It’s all in how you write the questions.” -Ken Phillips

Four examples of test questions

Look over the following four examples and note the difference.

Example #1

What does the letter “I” stand for in the ADDIE model?
A. Initiate
B. Implement
C. Improve
D. Investigate

Example #2

You sent a detailed design document to a group of SMEs to solicit their feedback regarding a program design. This is an example of what element in the ADDIE model?
A. Analysis
B. Design
C. Development
D. Implementation
E. Evaluation

Example #3

What are the four elements of a High Performing Team?
A. Powerful Goals, Frequent Team Parties, Aligned Goals, and Engaged Team Members
B. Powerful Purpose, Aligned Goals, Inclusive Leadership, and Engaged Team Members
C. Powerful Purpose, Aligned Goals, Including Leadership and Engaging Meetings
D. Powerful Purpose, Directive Leadership, Aligned Goals, and Engaged Team Members

Example #4

A supervisor discovered a compliance problem with their team that could substantially negatively impact patients if not reported and corrected immediately. The best leadership style for this situation is:
A. Visionary/Authoritative
B. Directive/Coercive
C. Affiliative
D. Democratic
E. Pacesetting
F. Coaching

The benefits of application questions

As you can see, examples #1 and #3 are recall test questions, and examples #2 and #4 are job application questions. The benefit of job application test questions over recall questions is that they not only provide data regarding whether participants learned the material taught in a training program, but they also provide information regarding whether participants KNOW HOW TO APPLY the new knowledge they learned.

Knowing this can be especially crucial when also conducting a Level 3 evaluation. For example, suppose the Level 3 evaluation results indicate that participants aren’t applying what they learned in a training program. In that case, you’ll have Level 2 test question data showing whether participants learned the information taught in the training program and whether they knew how to apply it.

Consequently, if participants aren’t applying the new information learned, you will know it isn’t because they didn’t learn it and don’t know how to use it. You’ll know that something got in the way AFTER the training that inhibited or prevented the participants from applying what they learned, most likely related to their work environment.

So, which is better? Recall or Job application

In short, the most significant improvement you can make when creating Level 2 multiple-choice test questions is to write job application-focused test questions instead of recall questions. The following information describes the main elements of a multiple-choice test question and guidelines for writing valid, scientifically sound multiple-choice job application-focused test questions.

The anatomy of a question

1. Select a learning objective from your target training program.

2. Think about what learners need to know and do to meet the objective.

3. Identify real-life situations that reflect the actual application of the program content covered by the objective.

4. Choose one of the situations and write the stem (the question or problem) in 1 to 3 sentences.

Some possible item shells for writing the stem include:
• “What is the most effective method for…?”
• “What would happen if…?”
• “What is the most critical step in the process?”
• “What is the most appropriate solution to this problem?”
• “What is the next step to solving this problem?”
• “What is needed to solve this problem?”

5. Identify plausible distracters. Think of things someone might say or do who thought they knew the content associated with this learning objective but didn’t understand.

While writing job application-focused test questions is more complicated than writing recall questions, the data you collect will provide much more valuable insight and make the additional effort worthwhile.

Level 2 job application questions

In conclusion, the power of well-crafted Level 2 evaluation multiple-choice test questions cannot be overstated. These questions serve as a litmus test for not only assessing whether participants have grasped the material but also whether they possess the essential skill of applying their newfound knowledge in practical settings. By following the guidelines outlined in this blog, educators and trainers can shift their focus from mere recall questions to job application-focused inquiries, thus obtaining richer and more actionable data.