Bridging the Technology Gap in Skills Assessment

All Resources

Subscribe: Apple Podcasts | Google Podcasts | iHeartRadio | RSS | More

“If you think about how technology has aided education, what do we have? We have the LMS, which is very good at knowledge transfer and knowledge learning. It’s very good at knowledge assessment as well; you can build exams and if the exams are well built, the LMS can deliver them and dissect the data and all those kinds of things. And on the skill side we have simulations that are very good at teaching skills. We have emerging AR and VR that are very good for skills as well, in terms of teaching them. But what do we have in terms of technology to assist us with skill assessment? Well, nothing. I mean, if you think of it as a four-quadrant kind of matrix, there’s this big gap under skills and assessment. This, to me, is the last piece in terms of kind of the broad application of technology to learning, to training and assessment.” Murray Goldberg, CEO of Marine Learning Systems, is back with us to discuss his solution for the skills assessment gap.

Murray Goldberg was a tenured faculty member conducting research on the effectiveness of web-based learning in the department of Computer Science at the University of British Columbia. In 1997, Murray founded WebCT, which grew to be the world’s leading LMS serving 14 million students in 80 countries at 4,000 universities and colleges. WebCT sold to Blackboard in 2006 for $200M. Since then, Murray has created a new company, Marine Learning Systems, to address learning in the maritime and other skill-oriented industries. Marine Learning Systems is experiencing rapid growth and now counts among its customers more than half of the world cruise segment, as well as ferries, workboats, coast guards and others.

Goldberg has won numerous prizes including the UBC Killam University Teaching Prize, the 2000 National IWAY Award for outstanding contributions to information technology, and was named as the recipient of the New Media Hyperion award new media in IT. In 2004, Goldberg was granted an Honorary Ph.D. from Southern Cross University for pioneering work in advancing Ed Tech. globally. Also in 2004, Goldberg won the national Manning Awards foundation Principal Award, a $100,000 prize honoring the country’s most outstanding innovator. Most recently, Goldberg was named one of the top 15 Canadians in digital media by Backbone Magazine.

Host – Steven Maggi: How do you assess the skill level of all of your employees? It’s a big question. Nobody knows quite the right way to do it. There’s been some efforts at it; however, we have a product now that can really do this about as well as you can, with more of an objective look as opposed to subjective. Anyway, to tell us all about this product today is Murray Goldberg. He’s the founder and CEO of Marine Learning Systems. He was with us last week, and we want to talk to him specifically about this new product.

And one thing you came up with, which I want to talk about, is a thing called SkillGrader. Some of your people gave me a test run through of this, and I have been involved with safety programs in various industries all my life. I’ve never seen anything like this. It was a way of making everybody really accountable and really quantifying the type of skills that you acquire over time, which was fascinating. Are people afraid of it sometimes or do they love the idea because they know they’re taking away some of the subjectivity that’s involved in any kind of workplace?

Murray Goldberg: So, it’s early to say for sure, because the SkillGrader is a fairly new initiative – and I’d love to tell you just a little bit about where it came from, because to me it’s very interesting. To answer your question specifically, it’s a fairly new initiative and so its use is limited so far. We really just built the commercial version of it, so we have some beta testing and some pilot testing. But the first anecdotal evidence is that it doesn’t make people afraid, in fact on the contrary. What we’ve been told is that in the previous scenario, where skills are typically graded by a sophisticated assessor; he or she has a form that they fill out on paper and they go through – a lot of it’s somewhat subjective, you can’t get away from that when it’s a paper-based kind of assessment. Whereas previously, at the end of the assessment, the trainees would have a discussion with the assessor and it was very easy for the trainees to disagree about how they were assessed, because there was a fairly high degree of subjectivity. And so, that was a problem – not just for the trainees, but for the assessors as well – because they were always in a position of having to defend how they assess these things. With the SkillGrader, so much of that subjectivity is taken out. It’s highly objective, at least compared to how it was done or how it’s still done in many cases, but how it will be done in the past, I’ll say optimistically. So, what happens now is at the end of an assessment, if a trainee is looking at the outcome of that assessment, there is very little room for argument – something either happened or it didn’t happen. And so, you might be able to argue, “Well, does what I did constitute a 3/5 versus a 4/5 on a paper-based form?” The SkillGrader is based on a more binary model, where you get very deep insights, but a more objective kind of assessment criteria. So those arguments go away. And reports, by the way, come out immediately and automatically, which is another advantage when you’re talking to trainees.

Steven: Right, and it seems to me, for the employee there’s that sense of fairness. It really comes across there, where again, it isn’t “Well, this guy doesn’t like me, or she has a thing for me where she doesn’t want me to succeed”. This is pretty objective, and I thought that was really positive. Tell us how you came up with this in the first place? Where did the SkillGrader come from?

Murray: Yes well, I wish I did come up with it. Just like so many things in life, it kind of came up, in some sense. If you think back just a little bit, it’s actually shocking to me that we didn’t come up with it or that somebody else hasn’t come up with it before, because if you think about how technology has aided education, what do we have? We have the LMS, which is very good at knowledge transfer and knowledge learning. It’s very good at knowledge assessment as well; you can build exams and if the exams are well built, the LMS can deliver them and dissect the data and all those kinds of things. And on the skill side – that’s the knowledge side – on the skill side, we have simulations that are very good at teaching skills. We have emerging AR and VR that are very good for skills as well, in terms of teaching them. But what do we have in terms of technology to assist us with skill assessment? Well, nothing. I mean, if you think of it as a four-quadrant kind of matrix, there’s this big gap under skills and assessment. This, to me, is the last piece in terms of kind of the broad application of technology to learning, to training and assessment.

And so, how did we come up with this? So, what happened was, we have a huge cruise line customer. I’d be happy to say their name, but I probably would have needed permission to do so. They are fantastic and I certainly won’t say anything bad about them. They’re remarkable. They’re remarkable in terms of their dedication to safety. They have a very large simulation training center, maritime simulation training center in Europe, who is also a customer of ours because it’s owned by this large cruise line. Those people are expert assessors. First of all, they are expert skill teachers because that’s what they do, and they are expert assessors. They are really remarkable to watch in the simulation scenarios – how they guide them through and how they assess the outcomes of the teams and the individuals, in terms of how they contribute to the teams. They realized that there was a gap here, and they said “We want: 1. To be able to do more assessments” and by the way, assessments are limited by the availability of expert assessors, and so we want the level of expertise required by the assessor or the observer to be a little bit lower of the required level, a little bit lower, so that we can do more of these, that’s number one. We want them to be much more objective rather than subjective, and we want to be able to mine data out of them much more easily. Because they had the problem where everything was done on paper, where it was very hard – because of the subjectivity – to compare my assessment that I’ve done today versus one that I did a year ago; or, an assessment that you did today, Steve, versus one that I did today. This is all a problem. And because the data is all on paper, it’s very hard to mine. So, the result of course is that they needed some technical help in doing this.

So, they – “we” as their vendor – and a German human factors performance group kind of got together and said, “What’s the best way to do this?”. We came up with what they actually refer to as “SEAS”, which was a specific assessment tool for simulators. And we built one for them, just as a prototype, just to kind of vet these ideas. Interestingly, the prototype was released to them, just for a little bit of testing, and before very long, it ended up being used 100% for every assessment – for every simulation assessment – they did throughout the organization. It was that much of a hit that it became used ubiquitously throughout the organization. The technology wasn’t really ready for that because it was built as kind of a prototype proof of concept; but nonetheless, they did, and it’s very positive.

So now, what we’ve done is we’ve made a tremendous number of changes to make it much more general, in terms of the algorithm and the backend that’s used to assess the teams and the individual contributions of the teams. We’ve turned it into an app, so that you can take this app on a tablet and run it anywhere, and it applies to a much broader set of skills. Really, any observable skill that’s performed by either an individual or a team can be assessed by this. So, this app – you know, I started by saying the app is really just coming out right now, and so we don’t have a tremendous amount of data with it, aside from its use at this large simulation center – but we are very excited about it. Because number one, I think it’s going to be a big hit in maritime, but I think it will be a big hit in any safety critical kind of setting, especially where you are multi-sited and you want to really keep control of how assessment is done. You want to set those standards at the top of the organization and make sure they are followed down lower in day-to-day assessments. But, I also think it’ll be very interesting to see what the uptake is like in large distributed brand-focused organizations – even hotels or food and beverage – that really care that everything is done in exactly the same way to their high standard, regardless of where they are. I think it’s tools like this that will actually enable that kind of thing.

Steven: So, we’ve got this exciting program, SkillGrader. You are fascinated by learning management systems, it’s your passion and so forth. So, it’s the perfect time to ask you the magic question, which is – take your crystal ball out – “In 5 to 10 years, where do you see LMS going at that point? Is there anything that you are particularly excited about?”

Murray: You know, there is a lot that I am really excited about actually, and maybe that’s not a huge surprise, because you’re right, I really do care about this. Let me defend my excitement about learning technologies just briefly, and then I will actually answer your question. What do we have that has a greater opportunity to make a societally significant change than educating or training people better? Because that’s what we do as a culture to advance ourselves. Not only will better tools help us better educate our doctors, our lawyers, our mechanics, whatever; but, they will help us better educate and better train our teachers and our trainers as well, who will then go on to become better teachers and better trainers, and better educate the next generation and so on and so on. That’s why I’m excited about these technologies because I think fundamentally what they do is they have the opportunity to improve our society, to improve our culture, to improve our prospects as humans, effectively. So, that’s why I care about it.

What do I think is really exciting right now? You know, on the LMS front, it’s really about data. Being able to slice and dice data, being able to use machine learning and AI approaches to examine in a fine-grained way how people learn, the path they take, the successes or the failures that they have, and who they are; and combine that all together to be able to present to somebody automatically – okay, based on who you are, this is how you are most likely to achieve success – and we’re going to walk you through that basically. So, on the LMS front, I think that’s very exciting. It’s a form of adaptive learning, and with machine learning, we now really have the tools at our disposal to take some of this and make it very exciting. We have a partnership – you know, because of my relationship with the University of British Columbia – there is a data science program there, and we are working with them on that to do some of the very earliest parts of this, so I’m very excited about that.

I also think that within skills, there is a lot of very exciting stuff happening in AR and VR and simulation. I mean, we all know about that – that’s in some sense not new news, it’s slightly old news now, if you can call it old news – but, I think that’s going to be really fascinating, especially when you take it and you apply it to teams. I mean, we all have the experience now, or many of us will have the experience now, of working on a shared Google Doc; where I am typing here in Vancouver, and Steve, you – wherever you are – you can see me typing and you correct as I’m typing, so we are collaboratively doing something, but at a distance. I think as we extend AR and VR and extend it to teams who are desperate, who are located around the globe, and present to them scenarios that they can work together and learn as a team on the same shared simulation effectively, and by the way, have the accessor be a part of that and be able to replay it and be able to view it from any angle – I think we can take AR and VR and extend it, using some of the technologies that we have to really address this team-based skills aspect of training that’s very hard to otherwise address. So, those are the things that excite me the most right now.

Steven: We are all excited about it, and I know Murray Goldberg, that when those are out there, you are going to be right in middle of it. We are excited. If we want to find out more about your company, Marine Learning Systems, where do we go online?

Murray: Yes, so it’s not hard – www.MarineLS.com .

Steven: Murray, if we want more information on SkillGrader, where can we go?

Murray: You can find us online, Steve, at www.SkillGrader.com .

Steven: We will check it out. Thanks so much for being with us today, Murray, we really appreciate it.

Murray: Thank you, Steve. It’s been wonderful and I really appreciate you taking the time.