Big question and OcTEL

[imogenbertin]

Sorry folks – usual lack of time means one last long final post…
My big question was based around my previous supply chain studies which emphasise providing value to the customer, and my own experience of being a learner using TEL. It was informed by a concept from marketing which has been tweaked by digital marketing – the funnel. Online purchases have a very low completion rate – it’s often around 3% or lower and much of the work that’s involved in digital marketing is about tweaking the user experience to achieve small increases to increase the few who make it to the bottom of the funnel. I think there are parallels with TEL and specifically MOOCs.

So this was my question: Why don’t TEL people respect the time of their learners? Why is UX given so little priority? There must be several reasons why most online courses are such bad experiences and the drop out rates are so high.

My intention in asking about these questions derives from another Japanese supply chain concept – Mura, or unevenness. Mura can loosely be translated as friction, or rocks on the stream bed. When you have lots of resources/water in the stream the rocks are not a problem. When the water level drops, if you don’t remove the rocks, the flow stops, the boat grounds – and the learner drops out of your course or is dissatisfied with their learning experience.

It’s my belief that a great deal of TEL open course dropout is due to really basic user experience problems like secure passwords, or issues with Java, and that by applying techniques from good technical support as well as removing these rocks, a much more satisfying experience can be provided. I would really like to compliment OcTEL’s learning technology guru Martin Hawksey on his fantastic work with the course reader and the “rogue’s gallery” and countless other features. On the other hand, problems using the OcTEL forum interface nearly did for me…

David Jennings mentioned in the webinar the difficulty of getting feedback from disaffected learners, and I think we’ve all seen this in conventional courses – it can be very hard to get any information at all about why someone leaves. I would like to find out more about ways to get feedback from disaffected learners although my experience to date has usually been that they picked the wrong course or underestimated the time required.

Reading back over my notes from the past 11 weeks, emergent learning was probably the most interesting concept I learned. I feel a lof my practice is actually about engaging people to approach problems critically, with technology as an ally not a barrier. It’s about seizing the moment and responding appropriately within the limitations (time, money, ancient PCs, broadband). and It’s fascinated by the ability to finally measure better, and so respond to cause and effect.

Turning back to our week 10 topic of evaluation now and thinking about TEL courses I have taught, what do I need to know?

– Did the course do what it said on the tin?
– Was the information useful? Which bits?
– Were the activities interesting?
– What did you not like or find boring and why?
– Did you hit technology problems/were they resolved?
– How do they think they might use the information in future?
– Did I miss out topics of interest?
– Would you recommend the course to others/was it value for money?
– Your suggestions?
– Basic demographics (particularly important for courses involving digital literacy)

What evaluation methods have I used previously?

– Pre- and post-course surveys (phone and online and “quick questions” at the end of face to face classes)
– Making it clear from the start I am open to feedback from students during the course
– Keeping in contact with students after the course (I run an open Facebook group for students for example – UCCNewMedia)
– Recording participation in parallel and future courses
– LMS stats on numbers of visits/timing/frequency to course resources
– Giving video feedback to students individually part-way through the course and encouraging them to respond however they wish to the video feedback
– Employment (for funders, information to prospective students, and to trace progress)

As a learner myself, each week I have to fill out feedback surveys for compulsory self guided online training. I have no evidence anyone reads those surveys or acts on them, so I put in neutral answers every time saying that I didn’t know much about the topic beforehand, now I do, the length was appropriate and I don’t know what they could do to improve it. So as a result of this experience, I always try to keep my own surveys to learners short, anonymous and to allow free comments.  I try to summarise the anonymised feedback BACK to the students so that they know their comments are taken into account and acted upon wherever possible.

What I have lacked to date, and probably was my main motivation for doing OcTEL, has been peer feedback and evaluation from colleagues for me. I still feel this is lacking despite David’s efforts and I hope OcTEL 2.0 may look at some more structured feedback with activities and assessment, perhaps in conjunction with a CMALT paid-for registered strand.

I would also like to involve learners in designing the evaluation. I am nervous about losing peer interaction and experiences by moving to online.

What I would like to do in designing my next course, which will likely only have a face to face introductory session, with the rest online, is to use Google analytics/learning analytics to spot issues earlier on so I can change tack during the course if needs be. I want to be able to make the kind of alterations that digital marketers make to their funnel, but not for their purpose of “increasing yield”, rather to be able to be sensitive to the rocks in the stream and to try to remove them by providing better teaching, and a better learning experience. I found this Educause article very interesting: http://www.educause.edu/ero/article/retention-and-intention-massive-open-online-courses-depth-0#.Ua3p8JDd5RQ.diigo and I’m also fascinated by the work of David Pritchard at MIT, who is using A/B testing of content within MOOCs to optimise what works based on evidence (page 5) http://campustechnology.com/Articles/2013/05/02/Massive-What-Good-is-the-M-in-MOOC.aspx?m=1&Page=4

My hope is that the frustrating issues which restricted interaction and group work previously will be less when people are using their own technology than in the very restricted computer labs I was teaching in previously (just getting the projectors to work was fun… let alone logins/Internet speed issues) but I need to measure this and have plans in place if there is no improvement.

What I’ll be taking away from OcTEL is good contacts and discussions, a feeling of community and fantastic resources that I haven’t even begun to skim but that I know will be invaluable in my attempt to complete CMALT. I hope to take part in OcTEL 2.0 if I can. Thanks A MILLION everyone…!

What I think OcTEL is missing: an eCourse book and a Librarian.
Things I need to do as a result of OcTEL… Google Apps for education, storyboardthat.com, webquest, Wally Woods 22 panels, Humbase, more on eLearning group work activities, thinking about how to combine mass customisation with formative evaluation…  If only life were full of sabbaticals!

[Author]

Posts