Proving the Value of Leadership Development: Case Studies of Top Leadership Development Programs

0:03
Hi, everyone, and welcome to today’s webinar, Approving the Value of Leadership Development Case Studies of Top Leadership Development Programs, hosted by HRDQ-U and presented by Drs. Patti and Jack Phillips.
0:16
My name is Sarah, and I will moderate today’s webinar. The webinar will last around one hour.
0:22
You have any questions or comments, please type them into the Questions box on your GoToWebinar control panel, and you can access today’s handout under the Handouts drop-down of your control panel.
0:34
Today’s webinar is sponsored by HRDQstore. Training professionals everywhere are feeling pressure to prove the value and impact in which their training initiatives can bring to the organization but that a proven return on investment or ROI of learning. Management buy in and training budgets might be in jeopardy. HRD. you store provides a range of tools for calculating the ROI for your learning initiatives.
0:57
Don’t let your initiatives get asked before you prove their bottom line value. You can learn more at HRDQstore.com/ROI.
1:06
And now I’d like to welcome our presenters today, Drs. Patti and Jack Phillips.
1:11
Patti is a renowned leader in measurement and evaluation. She helps organizations implement the ROI methodology in more than 70 countries around the world and has authored or edited more than 75 books on the subject of measurement evaluation and analytics.
1:26
Since 1997, Patti has been a driving force in the global adoption of the Roi methodology and the use of measurement and evaluation to drive organization change.
1:37
Her work as an educator, researcher, consultant, and coach, supports practitioners as they develop their own expertise in an effort to help organizations and communities thrive.
1:47
And Jack is a world renowned expert on accountability measurement and evaluation, He provides consulting services for Fortune 500 companies, non-profit entities, and government and non-governmental organizations globally.
2:00
He’s the author or editor of more than 100 books, conducts workshops and present at conferences worldwide.
2:05
Jack has been featured in The Wall Street Journal, Business Week, and Fortune He has been interviewed by several television programs, including CNN. Thank you both for joining us today.
2:16
Thank you, Sarah. It’s a pleasure to be here with you again today. Um, I’m Patti Phillips, as Sarah said, from the ROI Institute, and I’m happy to be here with Jack.
2:25
Jack likes to say hello, a glad to be here. Great session today, thanks for the good turnout.
2:31
So yes.
2:33
We appreciate it. So, Jackie could go to the next slide.
2:36
Yes. So, today’s event.
2:38
We’re going to talk about the value chain, or the need for the value chain for leadership development, looking at that chain of impact. So, as we invest in leadership development, what do people think about it or are they buying into it?
2:49
Are they gaining the knowledge, skills, information, and size they need to, in order to go out and apply what they’ve learned or change their behaviors? Then we follow up to see if, in fact, they changed the behavior, and the consequence of that behavior change and ultimately, demonstrate that return on investment. So, we’ll talk about that value chain. We’re also going to talk about the need to show value in terms of impact, in ROI for leadership development.
3:12
We spend millions and millions of dollars on Leadership development program, but the question still comes up, you know, what’s it worth? Is it really worth the effort, the time, the resources to get us where we want to be, given the investment?
3:26
And then we’ll also talk about how programs are designed to deliver results. In the end, we’re going to share a case study. Jack’s, kinda walk you through a case study and then we’ll give you some next steps from there. So, Jack, you want to share some of the background.
3:40
We’ve been pushing this issue for a long time and it’s working helping.
3:45
People who are involved in leadership development show the value and see the value of what they do.
3:51
And it started back in 2004 with a book called the Leadership Scorecard that’s showed, How do you evaluate leadership all the way to impact and Roi then we Adam Graw Hill Book Measuring Leadership Development.
4:05
I think that’s 2012 and we showed more detail of how you do it and put some case studies.
4:13
The next two books are with our partnership with ATD, um, this one is measuring the success of leadership development, has more case studies. We offer more examples of what’s going on, and, again, updated process.
4:32
And moving over to the soft skills book, Leadership development as part of that.
4:37
Part of the soft skills category.
4:40
And we provided more case studies there.
4:45
And our most recent book we’re going to highlight today, it will be published next month. It’s proving the value of leadership development.
4:55
And here we got 12 case studies. But we actually showing you one of those case studies in some detail. And we’ll talk about that.
5:04
So there’s a lot of documentation here pushing it to those levels that the executives like to see. And so you have some takeaways here.
5:14
There’s a you don’t have to buy the book.
5:17
This is your guide. We’re going to show you. this is like a CliffsNotes.
5:20
If you remember in college where we, we’ve got the quick version, short version, it’s 32 pages. This tells you how to do it.
5:30
We’re offering that to you And we’re going to go through a case study in a, in this new book proving the value of leadership development.
5:37
It’s a program that launched during the pandemic.
5:40
And it’s empathetic, inclusive leadership which is a very common topic these days.
5:48
So we’ll talk about that and show you all the steps in the process. And then you have your slides here.
5:56
All the slides we have in. There’s a lot of content here for you. Might. you might want to come back and review again at some point.
6:02
Lots of resources here To do this.
6:06
Now, this new book, we’re so proud of it, it’s our most recent edition, but we’ve written a lot, as you can see, a lot of great endorsers.
6:14
But here’s a couple that you might know.
6:17
They’re well-known professors and also gurus, and the leadership development area. And they obviously had some praise about this book and its contribution.
6:29
Even Peter suggest the slip it into your CEO’s desk and the case studies here. Interesting here, we’ve got 12 case studies, and it shows a combination of businesses, and non-profits and governments. And that’s what we see is going on these days.
6:49
So, it’s not just a business issue. And something interested in here.
6:54
Is it all but three are identifying their names?
7:00
In the past, we’ve had probably about half the case studies, the organization, to identify who they are.
7:07
We’d like to obviously identify them, but the three of them here would not.
7:12
It’s not that.
7:14
it’s a, it’s a case study that’s not so important. And impressive. They are in fact, they do nothing but make the organization look good, but there’s a reluctance sometimes that name the names privately. We can what the one we’re going through today is this one, it’s a well-known technology company. It’s almost a household name for it. Is a household name for all of us. Will be going through that one today. Talk about it a little bit.
7:41
Now, let’s think for a minute about leadership.
7:44
Sometimes, it doesn’t deliver the value that businesses appreciate, leaders appreciate.
7:51
The Oh, we’re going to show you a value chain in a few minutes and that value chain, it shows the different levels of outcomes.
8:00
The number 1 and two Measures that executives want to see is impact and ROI in that order.
8:08
Impact impact comes first and the value chain Roi is the last step.
8:15
But why we get to see a lot of programs, it’s not working, we, fortunately, we see fewer of them, which because people are now designing for the results they want. They’re taking care of these kinds of things here.
8:31
And that number one, will be talking about today, is failure to align the program to the business.
8:38
No. I know we focus so much on the behavior. and the behaviors are important, but behaviors without impact, without consequences as just being busy.
8:46
And the executives understand that.
8:48
And if you say, Hey, we’ve got all this leader behavior in place, the behaviors are there a reaction of a lot of executives. So that’s great. I appreciate that. Now, how is that affecting the business highs of helping the business?
9:03
Sometimes they’re asked to to make that leap themselves.
9:08
But they prefer us, as it providers to make that leap shows some data.
9:14
And another important issue is not expecting success, pushing it.
9:21
through objectives. Yes, we have learning objectives, but let’s add to that: Application Objectives and impact objectives, and even Roy, Objective, if you’re going there.
9:32
So that is a success that we expect, And we tell everyone in the program, success is not incurred until we get to the impact without the impact.
9:42
We’re not successful, and that’s a critical thing.
9:46
We’ll keep coming back to those Issues, And here’s the point, very, very well connected to, that, is not designing for application.
9:54
And, in fact, we designed for application a lot, with 360 feedback, for example, but not so much with the impact. And that’s what we want to keep working on.
10:06
And sometimes we, we wait till the program is complete, and then we get asked that question.
10:14
So we spent a lot of money on the program. What was the ROI for that?
10:21
It’s a little late, after it’s all been done to, to make this work for us.
10:26
So we need to think early about this. And finally, this is our proof.
10:32
So, we’ve got the title of this session proving the value of leadership, which means we’ve got to sort out the effects of our leadership development program from other factors. It’s an absolute must.
10:43
We’ve got to do it.
10:45
That makes us credible, makes us believable to prove that we made a difference.
10:50
So we have the value change here, and it’s, it shows you how every leadership development program that’s ever been implemented, unfold in terms of its value.
11:05
It comes from a logic flow of data, but most of you probably know that I’m going to measure probably 95%, at least, or more.
11:12
It’s a classic logic model, dates back to 1800, so it’s not a new concept.
11:18
But here’s how it works for us. We have inputs that’s who’s involved in our program.
11:22
How long they’re involved in the program, and the cost of it, And maybe we’re throwing convenience there, but that’s all input doesn’t connect to the outcomes.
11:34
And the outcomes are in these categories: Reaction, learning, application impact Ottawa, Reaction C, The critical thing here is have reactions that are powerful reactions, that will predict application.
11:49
Hey, we got some of those here. Like relevant, is it relevant to my work? Is it important to my success? Is it something I will use? Is it something I would recommend to others, see, all for those, will connect to impact. That’s the kind of reaction we need from our leadership development program.
12:07
And of course, we’re learning some marvelous concepts and competencies and just beautiful skill sets that we’re offering here.
12:17
And we acquire them, we know how to use them, how to work with them, how to make them come alive, we learn that, that’s important.
12:25
But then, it’s not what we, No, but it’s what we do, is some people say, Now, we gotta go do it, and we do it, And we get them to use it, use it, often be, successful with it.
12:37
And we want to attract that and see it, but we know it breaks down a lot here, They just don’t follow through and use it, so we look for the barriers and enablers, those are the things that get helped us be successful, and the things that got in the way, that kept us from being successful.
12:54
So you’ve got process improvement, going on here.
13:01
You know, executives will quickly say, What does a bee behavior do for us?
13:06
Impact?
13:08
We want to look at things like productivity, quality of work, incidents, pretention, accidents, sales, new new clients, out of compliance, discrepancies.
13:21
All of those things, including collaborations and teamwork and satisfaction. So you gotta have two sets of data here. Impact. You’re going to have tangible.
13:30
It’s converted to money intangible that we can’t convert to money credibly.
13:35
A tangible is gonna go down into the ROI.
13:39
We’re going to convert it to money and compare it to the costs, and that’s going to get us to the ROI.
13:45
We calculate with a benefit cost ratio and an roi both. And the reason is that the BCR the benefit cost ratio comes from governments, it’s been there since the second century AD.
13:59
And embedded in the government’s cost benefit analysis, that’s what we do.
14:05
Now, we’re going to also use ROI. That’s more of a business term. But it’s been around 400 years.
14:12
So, it’s alone time, over 100 years.
14:16
It’s been a dominant method in businesses, show value for capital expenditures. Now, it’s being used in the last two decades to show the value of non capital and that’s what we have here, a non capital investment. So, we’re going to have both. So, there is a value chain.
14:32
You got to respect that, know it, think it, live it, and work it and report by the value chain.
14:41
So, petty, take us through some poll here and see how we stand as a group. All right, thank you. So, here you see some statement, some assumptions, hypotheses, and what we’d like for you to do. Sarah, are you going to launch the poll for us?
14:54
What we’d like to do is read each one, and we just want to know, are these items mostly true?
15:04
Sorry. I’m asking the wrong question. Click the ones that are mostly true.
15:09
Thought we were doing turned faults, I mean, which is true, which is False, but wanted to pick which ones mostly true. Sorry, apologies for that.
15:19
Yes. Yeah. Read each statement and identify those that are mostly true.
15:27
So, nice. I’m worried about the Oh, good. Thank you guys.
15:31
This Guide, the poll launched here. I see lots of votes streaming already.
15:38
So, we hope 15 more seconds here to submit your answer.
15:48
Anger, about 50% and just a few more.
15:53
Again, mostly true.
15:59
OK, great, let’s get those results up on the screen OK, thank you guys, so the same myths that are mostly true, so.
16:09
Then Most Of them are Mostly true, so one most learning and development is wasted not is telling 41%.
16:17
Said, That is mostly true, even though there’s a lot of data out there that suggests, most of it is wasted, or much of it is wasted.
16:26
But I think we’re getting better at that thing. We’re getting better at targeting the training, the learning. I think we’re better at aligning it to needs putting the right things in place at the right time.
16:39
In terms of the others, mostly, most of them are mostly true, the one that landed with the highest is most executives, you hard skills, more valuable than soft skills. So, we get the most agreement on that one.
16:57
In terms, Secretary said, Jack, you see anything, you have comments? Yeah. Did say that. Say that last one is where we come in with leadership development.
17:07
So they’d saying, It’s more, it’s more valuable. What’s the hard skills?
17:11
But we would argue that, that’s not true.
17:15
Even those executives, if you listened to them, they’d say, you know, you create them.
17:21
The most innovative company with great leadership, the most admired companies, with great leadership, hey, the best places to work comes with great soft skills and with our leaders.
17:34
And so soft skills, with leadership development, leading the way, it’s what gets great organizations.
17:43
So, they just, the problem is we haven’t showed them that soft skills can deliver value.
17:50
Now, we’ve got the world’s largest database of roi studies and soft skills.
17:56
No doubt, there’s no no one close, and we know, and we see.
18:02
Hey, soft skills, Roy, are a lot more, lot higher than the hard skills.
18:09
Hard skills, we mean teaching someone to do their job, the tests and procedures, or it may be technical training, or it could be stem, science, technology, engineering, math.
18:20
Yeah, so those are the hard things that we think about, but the saw things of communication, and teamwork, and empowerment, and leadership development.
18:31
So powerful. So here’s the problem, Every one of these should be mostly false.
18:38
So when you’ve got such a high percentage that true, it creates a problem, as you can see.
18:43
If you’re admitting that, a lot of learning development is wasted.
18:47
And we don’t measure what the executives want.
18:50
And we don’t show that make a difference, by the way, to make a difference.
18:54
It has to be application and the impact. That’s how you make a difference.
18:59
Or if if they see it as a cost and not an investment, hey, this explains why budgets often get cut here.
19:07
First to go as they often say. So we kind of reverse that. We’re working on it we see a lot of progress.
19:14
We need more make sure, that we connect what we do to the business show that And Often to their executives. So we want over overcome these.
19:26
We do see improvement Patti, some of that our polling we did in the during the pandemic had these numbers 80 to 90% on every one of those. That’s right.
19:37
And it starts getting better. So, this is good news. So, let’s go to another poll.
19:42
Just see, what, What Why are we extra interested in things like impact and roi now?
19:49
What what’s driving our curiosity about showing the value at this level and here are some options. You get you to check.
19:56
Which ones up, all that apply, you can check any of those.
20:02
So, Patti, go ahead and go with this.
20:05
Votes are coming in, so, thank you, Sarah for open-ended poll. Says, select all that apply.
20:10
Why you pursue an ROI now, and we’re seeing, based on early returns, a lot of proactive participants in the room.
20:20
So, I’m wanting to explore new techniques to measure.
20:23
Some will need to demonstrate more value for money.
20:28
So, we’ll let you guys go for just a little bit longer. We would like to get 60%, or more, And we’re at 50% voting, so just another couple of seconds.
20:43
I do want to comment, someone posted, and yeah, the Q and A They are how they reference soft skills.
20:51
We are seeing that language shift. Now, what is a soft skill? Anyway, right? They’re really critical skills. Skills.
21:02
And we’re starting, as Jack said, that that percentage in the past has been higher. And we think that, you know, executives are starting to see their critical skills. So it, by the way, we don’t like the word soft skills either.
21:18
No core skills. Sometimes. Power. Powers hills, Yeah, yeah, that sounds better. But a lot of people don’t know what we’re talking about.
21:25
It seems like the soft is ingrained in our vocabulary. It actually came from the military years ago where they began to classify their skill sets into hard and soft.
21:38
So people relate.
21:40
We understand that the word soft, I’ll, I’ll almost infer something, not as important or valuable as the hard skills.
21:50
So I think it gives us a bad rap just in the name there.
21:54
So we do need a name, that if we can get that name, to be pop, or so we understand that we’re with you. Yeah, yeah. Yeah, but someone else even popped, and we refer to people skills.
22:04
So a few people referred to as people skills. Nevertheless, they’re critical, right?
22:10
So let’s take a look at this poll, so why aren’t we doing it? And Jack, as I mentioned to you, when the early polling was coming in, a lot of proactive individuals on the call today.
22:20
Because if we look at that last item, 76% want to explore new techniques to measure program success. That is where you want to be. You want to be on the proactive side of the continuum, not the reactive side.
22:34
Yes.
22:36
That’s moving from reactive to proactive and so, it’s good to see proactive here.
22:41
If you’re in the first two categories, you gotta move quickly. You got a lot of pressure. Safe.
22:46
You wait for this requests for roi.
22:51
You have three things that happen to you.
22:53
one is, you are now defending your program, because usually that request comes with almost a challenge. Could you show me the ROI of this program?
23:07
And so you’re defensive, and it’s a short timeframe when they really want to see the results. You can’t say, Well, let me go out and build a capability here and learn how to do this, and then maybe we’ll do it on the next one.
23:21
They want to see it on this one.
23:22
So you’ve got a short term timeframe to deliver, that’s almost impossible.
23:26
And third, you’ve got ROI on someone else’s agenda. We want to keep it on your agenda.
23:32
You need to drive this bus, and so it’s good to see.
23:37
We’ve got so many down in the, oh, proactive area.
23:42
So, let’s look at what do you do about this? Betty …, which has now become the most used evaluation system in the world?
23:51
Absolutely. So, you want to follow a process. You need to process model, process models. Help us explain what exactly we’re going to do. They also help us explain what we did when it’s done. And that’s what gives you some reliability and the results that you’re reporting.
24:04
So, the process here really starts with that critical question, is why aren’t we doing what we’re doing? That’s part of that evaluation planning. And planning your evaluation begins. before you ever even design the program. It starts with, why aren’t we doing? What is the opportunity for the organization to make money, save? Money avoid costs, do greater good while making money, Saving money, avoiding cost? And then, what are the specific business measures that need to improve.
24:33
That will, if they improve, will move us toward that opportunity, or help us solve that problem. So, you all know design, thinking and type of design, works out problem resolution, or leveraging an opportunity. That’s what we’re doing with leadership development.
24:48
So, we define what is the problem, what is the opportunity, and then what are the specific business measures or indicators that tell us, we’re moving that need to improve. That will tell us that we’re moving toward that opportunity. And then, we use those amazing performance consulting skills we all have to sort it out, what is happening or not happening that if we changed, it, could help us improve the measure.
25:12
And that’s where solutions start to bubble up, and then we say, well, what does that people need to know, and how best can we roll it out, so that they buy into it?
25:21
So we decided on a solution, and then we set it up for success with objectives as clear, specific objectives. They get nebulous. Objectives will give you a vegan nebulous program and Vegas fabulous results. So, specificity is going to drive the results, and then all we do from there is design the solution or the program around the objectives, roll it out, and collect data.
25:42
We collect those reaction and learning data during the program, We wanna make sure people are buying in. They’re acquiring the requisite knowledge, skill, information, and insight. And we follow up to see if, in fact, they’re doing what we want them to do, that they’re applying the knowledge skill. And inside, that, their behaviors are changing.
26:00
And also, we follow up to see if the impact measures are actually improving. Through the analysis, we’re going to take that critical step, to isolate the effects of the program. This is a step that so many people overlook and intentionally over lock, saying, We just can’t do it.
26:16
Of course, you can, you can use control group, you streamline analysis, you, could use your mathematical modeling worst-case, Go ask is an estimation process. That has a little bit of credibility to it, but we’ve gotta carve it out. We’ve got to be able to say, this is how much of the improvement is due to our program? And here’s how we know that, that step to isolate the effects of the program, then, to get the ROI is just a matter of converting your impact measures to money.
26:43
Bringing in the cost, calculating the ROI, and for some programs, or some measures, or not going to convert them to money, they’re just as important as those we do convert to money. That’s your intangibles.
26:56
And then we just roll it out, communicate the results, use the data, and that’s the process. We need a process. I think it was Deputy. Edwards Deming, Jack, who said, if you can’t explain what you do as a process, you don’t know what you’re doing. And that’s where you don’t want to be. When you’re explaining to the senior leadership, how you got to that ROI. You want to be crystal clear. People respond to processes. Now, W Edwards Deming also said, a process without standards is not a process, so Jackie, to go to the next slide.
27:28
We do ground our methodology and a set of standards that ensures credibility and reliability. We want to make sure we’re telling the complete story with confidence that we’re given the most credible story. We make conservative assumptions. We collect data, we use, data that we have, we don’t make our data we do. We ensure we include the fully loaded cost. Our denominator said the Standards to ensure that we’re conservative and that were credible. So whatever process you use, you want to make sure you have standards that help ensure people are following the process. The same way, every time they may change how they’re collecting data, that may change some of their techniques, given the context or the program, but they are following the same process, and making the same assumptions. Now, the good news is that you’re not going to do this for every program.
28:17
There are some leadership development programs that we just nice to have, nice to do for our people. Hey, some are not too expensive, some are not too long. Some are not involving too many people, and they’re not high profile.
28:31
Those we may not evaluate to impact, in our Y, So what you see on the screen now are our recommendations, in terms of them, minimum number of programs, to evaluate, to impact in ROI?
28:44
So, we look at all the programs that you offer. For example, we’re suggesting 10% men, A mom are candidates for ROI. So we work a little bit upright, up principle, 80 20 rule 20 percent of what we invest and gives us 80% of the impact. But a minimum 10% of your programming is probably delivery impact.
29:05
We suggest that’s a minimum to start with.
29:08
And then, why? About 5? 5 to 10%?
29:12
The programs go to impact. And not why?
29:14
Because, again, that everything is driving those measures directly, said the middle column is what the minimum is, but we did a benchmarking study back in 20 20 of a very small sample of our user groups. And you can see that there’s more evaluation at impact, an ROI, then we even set as our minimum standard, you can, you can overdo it. I mean, it’s good to see because progress is being made. There are probably more strategic with their investments but you don’t want to overdo it. Right? It’s all about investment management, re-investment allocation.
29:50
Allocating your efforts to the right things, but we’re really pleased to say that this much progress is being made jacket, I want to make any comments about benchmarking data before we look at. Yeah.
30:02
This is, if this is easy to see with your entire L and D function, for example, where you have sometimes, we’ve got several clients with our thousand programs, And you can see, you need something like this to decide which ones push to that level. When you look at the leadership development.
30:20
Sometimes it’s just a few programs, and, and sometimes, up.
30:25
Most of them, are very important.
30:27
And so, they drop, they just kind of accountability.
30:33
So you get a higher percentage, maybe, of your, just, your leadership development push to that level.
30:39
And I think that’s one reason we have this number, so high here, because leadership development as a topic.
30:48
It’s probably it’s our number one area where …
30:51
views when we have our certification program, where people come to our to learn how to do this, and they select a project that they want to evaluate all the way to Roy.
31:02
The projects they’re working with and a group of 15 people, for example, you have 2 or 3 working on leadership development because it’s so important, so needed, so expensive, Yet, we just haven’t connected it to these levels of accountability so much. So that explains, I think, why our benchmarking is a little higher for those levels.
31:24
Now, here’s the criteria that we suggest when you’re selecting a program, too, push to this level.
31:31
The question is, So which program should go to that level? Probably number one, is this one, the cost.
31:38
The more expensive the program, the more it needs just accountability.
31:43
The highs are the largest costs we’ve ever seen for our program, was a leadership development, and it’s $200,000 per person.
31:53
It was a government owned oil company from Norway.
31:58
We’re sending it executives to a program at the University of California, Berkeley, and because it cost so much, and it was owned by the government, owned by the people of Norway, the CEO says, yeah, I think we need to show the value of that.
32:14
So, the point we’re making costs will get us there.
32:18
If it’s very important, it’s solving a huge problem.
32:20
Maybe we got it staff retention issue, and we’re trying to get it improved.
32:26
Maybe it’s part of our strategy for workforce experience, maybe, are just what is it the executives like to know in a Tuesday?
32:36
The soft ones get us there.
32:38
So we’re just thinking through these kinds of elements to decide which program, if we got a combination, are of possibilities of of the of them are evaluating several programs. Now, we’re going to take you through. The rest of the program essentially is a case study.
32:57
And only take you through what Patti’s just presented, Those 12 steps.
33:02
I’m going to show you those steps come alive in this program, but also show you that how we give the briefing to the executives. Now, this is, again, a well, well known technology company.
33:16
They, they pride themselves and having a lot of great diversity and inclusion going on.
33:22
But they noted some things.
33:24
It wasn’t not as, as working as well as they would like, And so this is what led to this program.
33:31
So they wanted to teach this content, based on some analysis, they thought it needed to be their leadership program. you can see the skill sets here.
33:43
It’s this kind of flavor of empathetic and inclusive leaderships.
33:47
And it’s it’s reminds us of what our leadership development has been tweaked a good bit during the pandemic. So we focus more on these two issues of empathy and inclusiveness. So those are skill sets. It is designed this way to two days of classroom learning.
34:06
It was, uh, live virtual during the pandemic and with some e-learning modules upfront and some coaching.
34:18
Technology, support, and networking at the end.
34:21
So, that’s our program design.
34:23
We were evaluating with them, 48 group of 48 individuals involved in this program.
34:32
The first point that Patti may start with why list the business measure?
34:37
They did some analysis, and they saw some things occurring.
34:42
That probably is related to these issues or leaders need to be a Morse.
34:47
And since it, a little more sensitive to their team, listening more, and maybe talking less.
34:56
But making sure, we’ve got people included and always of our organization. And they think they had some voluntary turnover connected to this.
35:05
And some transfers people transferring out because they don’t feel comfortable working with one group and they move to another.
35:13
And some absenteeism, maybe a little lack of engagement, but there could be other measures as well.
35:20
And so, they, they thought this is needed, and they put the competition together, and as it is, they invited people to come to the program.
35:31
Here’s how they took care of the second step that Patti mentioned.
35:35
You make it feasible. Select the right solution.
35:38
They basically says, Why don’t you select a measure you’d like to change?
35:43
And we’d like to offer a few here that might be possibility, but we want you to select two at least.
35:51
It could be volunteer turnover, it could be transfers out of your group.
35:55
It could be unplanned absences if that’s increasing more than it should be.
36:02
And then, other measures may be around Project Manage, but project time, or it could be productivity, quality of work.
36:10
Compliance, discrepancies, You know, it could be anything Select two Measures that you want to improve, but only if you can improve these with your team using these competencies.
36:26
And when they make that connection, say, Yes, I think I can improve that, that’s all you need, we we’ve got an affirmation that. It looks like this is going to work for us.
36:36
So, it’s participants who’ve made the connection there, and that is critical, because now, it’s a debt that dramatically changes the, the focus of this program.
36:50
If we didn’t have that, we’d be teaching them some new competencies or some site lead, different competencies than they’ve had before.
36:59
It’s all about competencies, and they, they’re not thinking, why am I doing this other than you asked me to do it. Are you requiring me to do it? Are you suggesting I’m doing it?
37:12
But now, by having them select two KPIs that they want to improve, two KPIs.
37:22
Them saying, I think I can improve it using these competencies, now. That’s the reason they’re there.
37:28
To improve those two measures.
37:31
It’s not there to learn leaders, new behaviors.
37:35
Yes, we do, and we use them, but that’s a means to the end, the end is now the impacts, There’s more ownership for this. There’s more interests and making it work now, because they want to improve those. You remember, select two KPIs that you want to improve.
37:51
So that’s important.
37:53
And the way you can visually look at this, this is Patti’s favorite, a tool that we have here, and it’s a V model.
38:02
Alignment model that you can imagine is, if you look at the value chain that we see here, is connected by objectives that Pei was talking about.
38:12
That is, it’s the objectives that drive the data that we need here.
38:17
If you’re wanting to measure learning, you look at the learning objectives. It’s as simple as that.
38:23
So we have, we need objectives, but the objectives come from the needs.
38:27
So as Patti pointed out, the needs drive our program.
38:33
But the needs a little different here. Now it starts with payoff needs and business leads together.
38:38
This is why we’re basically for pay off need We’re asking the funder, the …
38:44
requester, the supporter, Hey, would you like to have more out of this program than you’re putting into it?
38:51
How do you think they would respond? How would you respond?
38:54
Most of most would say, Well, yes. I’d like to have more than it’s costing as.
39:00
Well, OK, then, here’s how How we’re going to get the value, It’s the business need.
39:07
That’s what value we’re delivering, So the business need, And it payoff need go together. Together. It makes why, because we now want to identify the business measures.
39:18
We need to prove to get to the positive RMI, and that’s why we let them now select, those two measures, but now we’ve gotta be doing something different.
39:29
In this case, they’ve agreed a need, these competencies I can use these competencies, and we’re going to learn these competencies in this program.
39:38
And I got to see these competencies as relevant to my work important from ISSS, Something I would commit to make work and I’d recommend it to others. You can see, This is now the solution.
39:51
Step 1 and 2 goes there. Step three is going to be here with the objectives, and we’re going to go there, Next.
39:59
And here it is.
40:01
So three things we do here.
40:04
one, is we remind everyone involved in this project.
40:08
No success doesn’t occur until we have the impacts.
40:13
It doesn’t occur when we’ve had the behavior that’s in the interim, achievement, to get to where we need to be.
40:21
And with that in mind, we develop the objectives, it shows how we get their objectives for reaction, and learning, and application and impact.
40:30
An ROI, if you go into Roi, and they were in, they did, then they’re all given to the stakeholders.
40:39
For everyone, the designers, developers, these supporters of the participants, the program owners, of the managers, or the people in the program.
40:50
Oh, everyone gets these objectives.
40:53
And we’re asked to do your part to make sure we have the impact.
40:57
Because it’s all leading to the impact.
41:00
So there you have it.
41:02
The second about those objectives for a minute.
41:06
Let’s just, let’s try something here.
41:09
So, in your questions.
41:14
Patti, they have a chance to put something in there about it.
41:20
And so, let’s target, But these are saying, I’m sorry. Yes, so they have a chance to put it in here. In fact, I think what we want you to do, because we did not create a poll.
41:31
If you, Jack will read each of the objectives, let them write it down. So all of you, if you write down what level of evaluation, you think these objectives represent, and then we’ll share with you the answer, because we didn’t quit credit for them.
41:47
I think I think I can write it, right.
41:49
Only the slide here. So, what?
41:53
So, imagine the first sentence goes with each of these, and these are all connected to this particular program.
42:01
After completing this program, participants should decrease employee transfers by 20% in one year, So, think about if we collect data, to see if we’ve met that objective, at what level are re-evaluate: is it reaction, is that learning is application is an impact?
42:20
Is it Roy, put that number in place?
42:23
If you would, let’s go to the next one after completing this program. Participants should listen intently to team members routinely. No, that’s not as specific as we’d like to see it, but it’s still an objective.
42:36
At what level, if I collect data to see if they’re there doing that, what level am I evaluating?
42:44
Put one for Reaction two for Learning. three, for Application or for Impact for Roy.
42:52
Let’s go to the next month.
42:54
After completing this program, participants should be able to demonstrate the five steps to defuse a conflict given to individuals in a heated argument.
43:04
OK, so, if I collect data to see if I met that objective, what level?
43:10
1, 2, 3, 4, or five. Right to content, four out of five on relevance.
43:19
And then deliver at least 20% return of investment numbers.
43:27
So, it’s a These are important.
43:32
Here’s what we have here.
43:34
You’ve got a reaction objective done, number four.
43:39
Break the content four out of five.
43:42
Learning is you, This is this. There are more object to share, but we just give you a sampling here.
43:48
Be able to demonstrate that’s learning, and then, doing something on the job with your team members.
43:55
Number three is Level three issue, today’s hashtag two here.
44:00
And the one of the impacts is four, oh, the reduced the transfers and then the ROIs five, Just a quick exercise there to get you thinking about the multiple levels of investment.
44:16
Excuse me.
44:16
Multiple levels of objectives, Very powerful.
44:21
And we’re sometimes stuck on Level two.
44:29
We wrote a book for ATD not too long ago, and it’s this one.
44:33
It’s called Beyond Learning Objectives, and the subtitle is Develop Measurable Objectives, it linked to the bottom line, the message that ATD is getting to there, membership with that book. That’s a title that they selected. Not Us.
44:48
They said, Look. We’re quite good at developing learning objectives, but we gotta go beyond that these days.
44:54
Push it up to Level 3 and 4, maybe five if you’re going to setting objectives as powerful. By the way, you need that book.
45:02
Just send either one of us a note, jack at our institute dot net or Patti are always strict on that, we’ll send you a copy of that book just for your resources. Now, let’s continue on.
45:17
Can we go back to that slide? Real quick, because there’s a great question in the Q&A.
45:21
So, why is listening intently at level three, and demonstrating five steps to defuse a conflict level, OK, that’s good. If you looked at the complete objective, after completing this program, participants should listen intently to team members routinely.
45:40
The only way I know that, as I’m going am on the job now, and see if they’re actually doing it.
45:47
That’s not a measure of what we’ve learned, that’s doing, that’s, that’s why it’s application.
45:53
Now, the second one is, be able to demonstrate, hey, now you’re showing me that you can do this.
45:59
And that’s usually a role play or skill practice that you can do this. You’re demonstrating to me that you.
46:08
That you’re able to do, to use these skills. That’s a measure of learning. It’s not on the job.
46:15
I’d have to go on the job, and what you’d do that, and that you’re demonstrating for me on the job.
46:21
That’s not, I’m, as an observer. I’m in solution you anyway, we’re still getting level two, and not level three, because we have the effect of the observer there.
46:32
And if you know I’m watching, you’re going to do what you know, maybe not, what you’d actually be doing, if I wasn’t there. So you can see, that’s a to be able to demonstrate.
46:43
what you can do.
46:46
Again, we can talk offline, so more or less. if you’d like, are we happy to send you this book? This is important.
46:53
I don’t want to push this off, but it’s, we’ve got a lot more steps to go here. But this is powerful.
46:58
It’s still not done so well, And spite of the toolkit, we wrote a TD, and it was a best seller for them.
47:06
Let’s keep going.
47:09
Let’s go here.
47:10
Now, we’re ready to report results. When we, we tell a story to the executives here.
47:16
And this is how they react.
47:17
And we bring out the things that are very important to us, particularly those that often correlate with actual application. This is something I will use.
47:28
Look at the third bullet point, that’s so powerful.
47:31
That’s the best correlation there. If, if people will commit to doing something, they’re going to probably do this.
47:38
Remember, this is collected anonymously and they’re saying, we’ll use it.
47:43
You can almost count on that being use. This is powerful reaction here.
47:48
Now, learning spaces the skill sets, how well they learn those with role plays as skill practices in the classroom, and you can see some not quite where we want it to be.
48:01
but still very impressive results here for Level two. Now, executives don’t care so much about this or move quickly if it’s a particular point you want to bring up.
48:11
You’ll want to emphasize it here.
48:13
If you’ve got disappointing results coming. It’s gonna probably show up here and they just didn’t learn it.
48:19
Remember, it breaks down sometimes. Now, let’s keep going.
48:23
Making it stick. That’s level three.
48:25
To measure the extent of use and success with use. Am I using this? Is successful with me? You won’t both, here?
48:34
And, again, you want to bring out those key things that are making a difference here, things that are, that shine in your respect.
48:42
That are powerful to the executives, that get some interests here.
48:47
Now, we go, because we got people doing things, then let’s just step back for a minute.
48:52
How do you collect the data?
48:54
Hey, here’s the different methods are being used. This is from my benchmarking.
48:59
Again, the benchmark that Patti mentioned, this is a benchmarking, whether we call it best practice, that’s organizations in the range of one year to five year and the implementation.
49:09
This is what they’re telling. Hey, this is dominant.
49:13
We, we’ve been dominant there for quest surveys and questionnaires. The good news is that the numbers going down it for years, that was 100%.
49:23
We are trying to reduce reliance on surveys or questionnaire. So, for so much data collection, and this is good news.
49:32
And this is just looking at databases. A, This is good, because we don’t have to ask anyone. We just go monitoring.
49:38
We do that for level four, almost always, but for level three, we’re getting more and more that we can just go look at the system and see what they’re doing. But here’s our action plans. We’re very proud of this. This is continuing to grow. We push this an action plan, that was built into this process here: Action plan.
50:00
Where you have a measure you have in mind, is a goal, and you’ve got the action steps to actually achieve that.
50:07
And then, of course, interviews and focus groups are part of that.
50:12
So, let’s look at make it stick. Excuse me, make it credible. This is our proof.
50:19
We’ve got impact now data and we’re going to sort out the effects of our program.
50:26
This is explored, tried, and used, and this is typically what you say.
50:30
They, the best method is, Patti points out, is experimental versus control group. And they looked at, can match up to 48 with 40 other people that look a lot like them.
50:42
The problem is, that could be in the same kind of role, that with the same kind of teams, but they have to really have the same measures in mind, the same measures they need to focus on, and that makes it almost impossible to get a matching group.
51:00
Trend line analysis would work for us because we just trend the data. But that’s of too much work for them, And it turns out, that’s too much for us, we don’t want to do 48.
51:10
That’s 48 times 2. That’s 96 trend lines.
51:13
If all of them submit data, we’d have to do that analysis ourselves.
51:19
I don’t think any of us want to do not 96 trend line analysis.
51:25
Whoo!
51:27
That’s the the tools that we’re going to leave with you. Show you what we mean by that and how it actually works.
51:33
So they have to come to estimates, but you collect the estimates that’s for the most credible people. That’s the person who’s using this, The sorting out the effects of this program. Remember, it’s not their program.
51:45
It’s your program.
51:47
And they’re making it work.
51:49
And so there’s like likely to be no conflict of interests here.
51:55
And then you’re asking them percent of this improvement that you see for impact goes to this program? Now it is an estimate. We don’t lack estimates.
52:05
So we one of our standards at Paddy put in front of you number seven was two.
52:16
Just for the error of the estimate. But that’s where the confidence estimate.
52:20
So, if I give 30% this improvement, two, this program, and I’m 80% confident, that’s means 20% error, 100% confident, is no error. 28% means 20%. I want to take the 20% out and do that by multiplying, by 80. So 30 times 8 is 24. We’re going to claim 24%, not 30 seaweed making it more credible. You gotta do that to get the buy in from the Executives.
52:53
There are other isolation methods, but we just talk briefly about control groups and trend lines, And this estimate, there are others.
53:02
The good news is that about a third of the studies we see are using experimental risk control group.
53:08
The really impressive news is it 40% are using a trend line analysis of it may mean that you don’t have to use estimates. We try not to. But if we do, we’re going to make them credible.
53:21
Converting data to money. Hey, the good news is if it’s data that’s important to us, it’s probably already been converted.
53:29
The three measures that they offer people to consider selecting, there’s turnover.
53:37
Transfers and absenteeism turnover, HR gave them a number hundred percent, 150% of annual pay and transfers, they provided an estimate there.
53:50
They didn’t calculated routinely, but they provided an estimate from from them the most credible source. They’re the experts, absenteeism.
54:00
Heather acts of monetary value for absence, which is a little over the daily wage rate.
54:07
So, the point is, for this measure, they’re often already there.
54:13
And then they got some others.
54:15
And they sometimes, they picked other measures and they had the choice of if you’ve already know what it is, put it then, we call that as standard value.
54:24
If not, as an expert, if not you estimate, I’m going to show you that here.
54:29
So, converting data to money.
54:33
Just a side note here. Here’s the technique. So good news, at the very top there, something we call standard value.
54:39
It’s a measure that’s been calculated approved by management.
54:44
That is a monetary value. The monetary value has been calculated approved by management, and it’s communicated to the people who need to see it.
54:54
Hey, stiffed, 52% of them is over half the measures in our studies.
54:59
Oh, they already know what it is.
55:02
That’s good news, and they can find it from an expert 45% of the time.
55:08
And, hey, here’s also, you can estimate it, if that’s not available. Sometimes you can go to the database. It’s usually more us doing it than them doing it.
55:18
So, here’s just a summary of the monetary benefits Have many selected these different measures of turnover, transfers, absenteeism, and others you’re falling through. You’re adjusting for the error of the estimate here.
55:36
That’s a confidence. That’s how much goes to this program, the confidence. Those are all multiplied there.
55:42
So, you get monetary value with a table like this. You put that together, that’s selling monetary value, delivered from this program, 762,393.
55:54
Monetary benefits.
55:55
We’re going to compare that now to the cost.
56:00
But before, as mentioned, this, There are some important measures that are not converted to money.
56:08
We knew those in advance, and we got them to connect it to them on a five point scale.
56:14
Or one is no connection to his son, three is moderate, four is significant, five is very significant.
56:22
So 1 to 5 scale here, uh, And that’s, these are all connected.
56:29
So this is powerful. This is, this may be more important than the ROI. Remember, these are impacts not converted to money.
56:38
The ones that went to money, they’re going into the arawak calculation When we put the cost in here, here’s the cost of the program. I’ll let you look at the detail when you see the case study, if you’d like. you got any questions, we can certainly help you with that.
56:52
So bring them together with this calculation.
56:55
Calculate the BCR and the ROI, and you might want to do this with your calculator.
57:04
That is, Well, let’s say I think you’re putting it, put the numbers up.
57:08
Those of you who are working ahead here, you might want to get into this and calculate the ROI.
57:15
So, this is just the benefit cost ratio and this is now the Roi.
57:21
So we do those together, it looks like this.
57:24
That means for benefits, wherever benefit we have, we’ve got.
57:32
Forever, just ever costs. We invest for every dollar we invest. We get 1.97 benefits, almost $2 per …, 97%, that says for every dollar we invest, we get our dollars back plus 97%, to ways of saying the same thing.
57:47
We make that point because BCR and Roy have different origins, but it’s really the same dataset. They would tell the story.
57:54
Here’s what they’ve done, debriefings, the executives, getting the data out to the immediate managers, and the participants and the talent development team, that’s critical. And they make things better.
58:06
Last step of the model, make things better and use it to fund.
58:13
They had some barriers and neighbors that helped them, make a few adjustments to make it a little bit better. Productivity was such a high in tangible.
58:23
There’s, there’s, there’s a plan to actually connect this to productivity on the next go around, getting them, to define that. So we could get it to money.
58:33
And you want it down at the bottom, if you will.
58:38
You’ll get funding for this program because you’ve just contributed.
58:42
And a huge way to the bottom line, You’re doing something, it needs to be done, But you’ve now added value, more value than you’re getting at, doubled your money, essentially.
58:57
We’re going to wind this up, very usual quote. You wanted to give us your quote.
59:02
OK, folks, forgot, Cindy, at the bank, here. So just remember when it comes to delivering results, hope is not a strategy. Luck is not a factor. And doing nothing is not an option. Changes.
59:15
Inevitable progress is optional. It is up to all of us to get the job done. So we’re so happy you could join us. Thanks for all the questions, Jack, I’ve been having a little chat sessions over here with the Q&A. He said, it’s been really, really good.
59:29
Thank you all.
59:30
Here’s the QR Code code for the tools that we have offered for you. And again, if you want the book, Andy at ROI institute dot net, just let him know. You can also e-mail jacker me, that, that’s that, that’s the book for, the objectives.
59:51
What we’ll do is, we’ll let you know when this book.
59:56
It’s going to be released its next month, and we’ve got a super deal for you.
1:00:01
We get a price that’s lower than the Amazon cost, and free shipping.
1:00:06
If you want it, we will be happy to get that book in your hand, so we’ll be coming back to you on that, But the free book that we can give you today course, is the setting the objectives.
1:00:18
Hey, Jack again. Yes, go ahead.
1:00:21
And I’ve mentioned this.
1:00:24
People learn this process through certification.
1:00:29
This is where people come to learned this process with a project in mind and they completed project to become a certified Roi professional …
1:00:41
members on this webinar get 10% discount there, you see the discount code and you’ve got the QR code there.
1:00:52
And we’ve got a two day leadership webinar, excuse me, a workshop in Salt Lake City. Interesting thing here.
1:01:02
This is hosted by leadership development provider called … Institute. one of the studies in this book is Ann Arbor … study. So here’s what you have.
1:01:15
To me, this is the most significant development we’ve seen in some time.
1:01:19
A provider of leadership development is offering to their clients a workshop at their headquarters, to teach them how to measure the ROI of their program.
1:01:32
Now that’s, that’s amazing development.
1:01:35
So what we see is that, changing with, with leadership development providers, they know they need to push it to this level, and they’re saving are quite good example of doing that.
1:01:46
Questions.
1:01:48
We did have one person who wanted to see slides 26 and 38. 26 and 38.
1:02:00
Yes, I know she’s still on or not and quickly here, yes, 38 is who we’re communicating this to executives in the face-to-face briefing, almost always when you have roi for the first time.
1:02:17
It’s 3 to 4 page, Executive Summary, goes to the immediate managers, of the participants who are in the program. They’re also managers, but it’s the next level.
1:02:27
The participants, an infographic, We’ve got examples of those infographic with all the data on one page, and the Talent Development Team, so we all learn from this. They get a full report of everything.
1:02:42
So that’s what’s coming there. That’s for 38. And what’s the other one?
1:02:49
I think that’s the one she wanted, was 38, 26, 26 as the other one. So if you can shed 26, as well.
1:03:00
And those are on a five point scale.
1:03:06
With an objective of four out of five, this is what we have, or extent of views and success with use.
1:03:17
That’s just a listing of the competencies, and it’s how they are they using them, and are you having success with them?
1:03:24
Perfect. Yeah.
1:03:27
Going back to Q and A Any other questions.
1:03:30
Sarah, maybe we have time for a quick question.
1:03:34
Let’s see. So, Patti has done a great job at answering questions. And the questions back here throughout our time today.
1:03:45
So, I think, Patti, that you were able to get through them all, if I if I have that, right.
1:03:52
I think good, yeah. And with that, that does bring us here to the top of our session. So, thank you so much, Jack and Patti for another valuable webinar today.
1:04:04
Thank you, Sarah. So, it’s great to see.
1:04:07
Yes, and thank you all for participating in this week’s webinar. I hope to see you all next week for my next session, Employee Engagement in the Virtual Workplace: Key Strategies for Success. You can join me at the same day, and same time, so Wednesday at 2:00PM Eastern Time. And with that, we will let you all go today.
1:04:28
Thank you so much again for joining us for another webinar OK, thank you all, Have a great rest of the week, and great summer.