00:01

Welcome to this week’s episode of the HRDQ-U In Review podcast, where we bring you the latest insights and practical tools for enhancing soft skills training in your organization. This podcast is brought to you by HRDQU.com, and I’m your host, Sarah, Learning Events Manager at HRDQ-U. Today, I have Dave Vance joining me to discuss the webinar, Measurement Simplified: The Key L&D Program and Department Measures. Dave, thanks so much for joining me today. Oh, you’re welcome. Pleasure to be here.

00:30

So Dave, we had a really exciting webinar just a few weeks ago. It was actually your first time joining us over here doing an HRDQ-U webinar. So welcome. It was really great to work with you on that event. And I’d love to talk a little bit more about the topic here on the podcast. But for our new members here who maybe didn’t get a chance to listen to the webinar yet and are tuning into our podcast episode, can you share a little bit about who you are and what you do?

00:59

All right, well, real quickly, my background’s in economics. I got a PhD in economics. I was the chief economist at Caterpillar and head of the business intelligence group when they asked if I would be interested in starting our corporate university, which I knew nothing about. So that was my entry into the learning field. And we stood up the Caterpillar University, and that was a great experience. And after that, retiring from Caterpillar, I had the chance to write the book, The Business of Learning.

01:25

And then we started the Center for Talent Reporting, where we created the Talent Development Reporting Principles, a framework to help us all with measurement. Out of that came the Books Measurement to Mystify, and eventually the ISO Standard for L&D Metrics that incorporates TDRP. So that kind of brings us up to where we are today. I’ve since retired from CTR. Now I teach at several universities and conduct webinars and workshops.

01:52

And Dave, what changes do you see happening in the L&D space right now? Well, you know, there are some that are really interesting. I think we continue to make progress on learning. But having been in this field now for 25 years, although we’re making great progress in learning and certainly in the new areas of AI, when it comes to measurement and business strategy.

02:16

And sometimes it feels like we’re going over the same ground again. I think it’s because a lot of people move on. So we teach them, we bring people up to speed, but they end up moving on. Now we’ve got a new group of people coming in that really can benefit from understanding a little bit more about the basics in both measurement and strategy. So things change, but they also remain the same. Yes. It’s like clothing, right? The trends always come back.

02:44

It is. So what exciting things are you up to next? Well, Peggy Parsky, my co-author on Measurement Demystified series and the assistant director for the Serentel reporting, and I think it’s probably time for a new book entitled Measurement Simplified. So you’re the first ones I’ve talked to about this, but we’ve got the Measurement Demystified series out there and we’re happy with that.

03:09

But it’s a long book, it’s like 400 pages. And I wouldn’t say it simplifies it, but it lays everything out. People come back to us and said, can’t you please boil it down to something simpler? It’s a great reference book when I need to dive in and get an answer to a technical question, but it’s a lot to read for a first timer. So I think there’s a need for yet one more book, much shorter, much more direct.

03:35

that basically just gives people what they need to get started. It won’t make you an expert. It doesn’t answer every question, but we got a book that’ll do that. Let’s have a book that will really help you get started. So that’s what I think our project is going to be this year. That’s really exciting. You’ll have to let us know once the book is published so that we can share that out with the audience over here. I’m sure that they would want to be able to get their hands on that.

03:59

And so we recently did this webinar together, Measurement Simplified, the Key L&D Program and Department Measures. Can you share what the key takeaways were for our registrants at this event? I think one of the key takeaways is it’s helpful to have a framework. Other professions have them. Accounting has them. Astronomy, biology. And we need our own frameworks. And that just makes everything easier because we’re on the same page. We have a common language, common terminology.

04:26

And so in our case, if we adopt this talent about reporting principles framework that has reasons to measure and different types of users, but even more importantly, three types of measures. So if we can all focus on having three types of measures, efficiency, effectiveness, and outcome, and you always want to balance among those measures, and then we identify what the most basic measures are in each category. For someone that asks then, what should I do? Where should I start? We’d say, look at the framework as a starting point.

04:56

and talk about efficiency effectiveness and outcome measures. And then we’ll talk about the basic efficiency effectiveness measures and how to approach outcome measures. And if we use the framework and we start with the basics, then I think it really can simplify the task, which otherwise can seem overwhelming given that we have more than 100 measures in L&D. And people have written, just Jack and Patty Phillips, have written like more than 50 books.

05:23

So it’s not surprising people feel overwhelmed because we’ve got a lot of books out there, a lot of webinars, but I think there’s a way to make it simpler. And Dave, what are the biggest misconceptions about measuring the effectiveness of L&D programs? That it’s too hard to do and it’s too expensive and we’ll never be able to do it.

05:47

That is a misconception. And that’s one of the things that we and others are trying to make people feel more comfortable in doing. So just start with the basics. Start with a few efficiency measures, which we all have, like number of participants. And when was the course promised? Do we deliver it on time? What did it cost? What was the completion rate? That isn’t hard. On the effectiveness side,

06:13

We’re all measuring level one participant reaction today, so that’s not hard. We would say be sure, though, to carry it through to level three application. So let’s also measure the application. And a lot of people do level two whenever it makes sense to give a test. So if we do that, at least through the basic efficiency, the very basic effectiveness measures, then you’ve got your start. It isn’t that hard. The misconception is it’s going to be too hard to do.

06:41

But it’s not if we can talk about it in two minutes. And then for some programs, the really important ones, the ones you spend a lot of money on, have a lot of visibility, then we can talk about carrying it to an outcome in an ROI. But you don’t need to start there. Let’s start with the basics. Get your basic efficiency and effectiveness measures. And then for the really important programs, we can look at an outcome measure. And that does get a little more complicated. So don’t start there. Start with the easy stuff.

07:09

work your way up to the harder stuff. And so then how can L&D leaders ensure that they’re tracking the right metrics without getting overwhelmed by data? I think given the framework, you’ve got a list of usual suspects, we’ll call them. And you should always consult your list of the usual suspects to see if those measures don’t make sense for a program. For almost any kind of a program, you’re going to want

07:37

to know the number of participants, the completion date, the completion rate, and the cost for almost any type of course you can think of. On the effectiveness side, why wouldn’t you always do level one and get participant reaction? Where it makes sense, give a test for level two. Add a question about intent to apply to your post-event survey, which will give you application data. It’s as simple as that. And I can’t imagine a course

08:06

where you would not want level one and level three. For some courses, knowledge tests don’t make sense. Okay, so don’t use them for every course. But I think for every course, even compliance courses, even onboarding, you’d want to be asking about level one and level three. I just don’t know why you wouldn’t. And then as I said, for some courses, but not most, you’re going to push it a step further into the type of outcomes and whether there was an ROI.

08:35

So I would approach it with a checklist mentality. If leaders want to make sure they’re measuring the right things, you’ve got a checklist of basic measures. Work your way through that list. And you should always be doing the basics. And if you’re not, you’re not doing what you need to. And what are some common mistakes that organizations might make when they’re measuring LND success? And how can they avoid them?

09:01

I think some of the common mistakes are stopping too soon, thinking that all that’s important are number of participants and completion rate on the efficiency measure side. So you need to go a little bit more than that. And on the effectiveness side, we’re not all yet getting to level three application. Many people are stopping at level one and level two. And it’s so easy now by adding one question about intent to apply.

09:31

To the post-event survey, we are asking the level one questions, which you’re already doing. So just add one more question. So I think the biggest mistake today is for those organizations that are not getting level three application rate information. That’s a big miss because we know, unfortunately, in our field, the scrap rate, that percentage of learning, which is never applied and thus never used and thus a waste of time.

09:59

is about 50%. About half of all the hard work we do is for naught. It’s not used, it’s scrap. That’s an embarrassingly high number. We’ve got to get the application rate higher, which means the scrap rate goes lower. We should be at an application rate of 70 or 80%, not 50%. So people are stopping before they get to the level three.

10:27

And even for some organizations that are doing some level three, we should do it for every single course. Because if you don’t care where the course was ever applied, why are you bothering with it to begin with? so I think that is maybe the biggest opportunity we have in learning still is to get the level three. I’d like to get to level four and level five where it’s appropriate. But if you’re not doing level three, let’s get level three first before we go on to levels four and five.

10:58

Wow, 50%. That’s a huge number. 50%. Wow. If our CEO and CFO knew that, they would be aghast. They would be thinking, we’re going to cut their budget by 50%. Yeah. So it’s an embarrassing figure that we have to address. And then, so how would you recommend balancing qualitative feedback with quantitative data?

11:26

when evaluating training success? Yeah, it’s a great question because some people out there like anecdotes and some people like numbers. Some like both, I do, but there are others that really prefer one of the others. I think it’s always best to give both. So I would always give the data on the completion rate and when we got it done and the cost, and I would give data on the level one and level three, but I would also be sure

11:55

to extract from my surveys some great anecdotal information from people who say they love the course and here’s why they loved it and they’d recommend it to everyone they know because for some people that carries more weight than any set of numbers. And if you can’t come up with good comments like that, then I think there’s a problem. So I think you’d always want to balance it. I think you’d always want to share both.

12:24

the comments. And it’s easy now. AI does a great job in summarizing comments. You know, in the old days, someone had to go through and look at all the survey responses and copy down, which seems funny now, copy down the good comments you want to share. AI can do this now. It can categorize the type of comments you’re getting. It can pull out good ones. So it’s easier than ever to share both the hard data and the comments.

12:53

What would you say one simple change organizations can make today to improve how they measure the impact of their learning initiatives? Measure level three and share it. So if you’re not doing it today, add that one question about intent to apply. And it’s really simple. And you can automate it. Just ask what percent of the content that the participants think they will be able to apply back on the job.

13:19

and you can let them score it on a decile scale, 0, 10%, 20%, through 100%. So you can automate it. So it’s easy to get the data. It’s pretty easy to answer the question for the participant. That is the single biggest thing that we can do now easily to get the whole profession up to level three. Everyone today would say they do level one, like 95, 99%. It should be the same 95 or 99 % saying they’re doing level

13:49

at least intent to apply, even if they don’t follow up two or three months later to ask about actual application. So that’s the simplest thing we can do. The next thing I would suggest, but it’s not so simple, is that you really should have a strategy for measurement and reporting. So you should have thought through before the next fiscal year begins what that strategy is going to be. It might be a written document, a category, had a written plan. It might be a PowerPoint.

14:19

But you, meaning the people in the learning function, the CLO, the BPF training and the measurement analyst, you should have a plan. And it should be a plan that’s set before the year even begins. So you know what you’re going to try to do for the year.

14:36

And Dave, before we wrap up today, where can listeners go to learn more about your work? Well, that’s great. So once we close down CTR that was acquired by the ROI Institute. So if you go to the ROI Institute dot net, that’s their website, and you go to CTMR, the Center for Talent Measurement and Reporting. This is the follow on organization to CTR now at home at the ROI Institute.

15:04

So you can learn about the workshops, learn about the webinars, pick up a copy of the book, Measurement Demystified. And there’s a company field guide for that. So you can do exercises in your own organization. But I would really recommend starting with some of the free webinars that you can find on the ROI Institute website. Check out the workshops and get a copy of the book. And then keep an eye out for this new book, Measurement Simplified, maybe coming out next year.

15:34

Great. Well, thank you, for your time today. Oh, thank you. It’s great speaking with you and being able to share something that I’m excited about. And I think we have great potential as a profession to continue to make progress in the measurement and strategy areas. Absolutely. And if you have yet to watch the webinar, we will have that linked below. Make sure you check that out so that you can hear more from Dave on that topic. And we hope you enjoy listening to the HRDQ-U In Review podcast.