You are browsing the archive for evaluation.

Feedback from ocTEL session at ALT-C

18/09/2013 in About this course, Course information

Our evaluator, Rachel Harris, has posted a short report on last week’s session at ALT-C. James Little also did a great job of providing a Twitter commentary with photos, which you can read back via the #octel tag until more tweets knock it down the timeline. Here are the slides that Rachel, Martin Hawksey and I presented in the second session. (We linked previously to Martin’s account of his own separate session on the ocTEL technology.)

The final slide was intended to provoke comment from the participants in the session in the event that they had little to say. As it happened, there was plenty of feedback and we didn’t need it!

There was quite a bit of feedback from ocTEL participants who classified themselves as “dropped-out”, saying that the beginning of the course was too overwhelming. (This was even after factoring out the effect of a fiasco we had in the first 36 hours or so with an email list that got out of hand with volume of messages.) We found that ocTEL participants retained the sense that not completing amounted to failure, despite our repeated counsel that that was not the case. In response, Roy Williams asked “Why do people talk about being overwhelmed?”. If you go to a conference, you accept the fact that I can’t go to all the sessions and have meaningful conversations with everyone. It is not particularly overwhelming. If people looked at MOOCs in the same way as conferences, then they might feel less overawed. In jest, we floated the idea changing our name to ofTEL, the open festival of Technology Enhanced Learning, the world’s first MOOF.

Some people did the course just out of interest to experience a MOOC, but that is not likely to be ocTEL’s audience in long-term.

If we’re serious about extending the reach of learning technology in HE and of ALT, then do we have to accept that a full-blooded “MOOC baptism”, as I called it, might not be the best way of doing that?  One person was glad that they had decided not to persuade the mainstream academic staff to do ocTEL, because they could have been very disorientated by it.

The advice we gave at the start, (for example, “This is a professional development course, and its designers trust you, as a professional, to make your own judgements about what learning activities are useful to you and which you can skip. The reason there are so many options and alternative ways of spending your time is precisely to give you choice and control over selecting a path that feels right for you.”) was not sufficiently well communicated to learners.

What can we do to address this, short of repositioning the whole course and making it completely un-MOOCy? One option that occurred to me is to get people to go through a pre-course classification (n.b. not qualification — the course would still be open) whereby we describe different behaviour profiles (they might be called “gentle explorer” “occasional sampler” and “in at the deep end”, for example) and ask participants to say which profile they think they will most closely follow. Of course this would be non-binding, but it would force them to reflect on what would be a realistic ‘commitment’ for them to make, as well as underlying the legitimacy of the “occasional sampler’ route. We might even have different badges for them?

I should stress that these thoughts are all personal reflections, intended to encourage further comment, rather than any official ALT policy. As it stands, we remain hopeful, but not yet certain, of an opportunity to run ocTEL again early in 2014. To keep up with developments, please watch this space.

Expectations taster

29/04/2013 in About this course, Course information, Evaluation

Last week we asked you to tell us what your expectations are for taking part in ocTEL. If you haven’t already, please do add your views via the expectations questionnaire!

We are really hoping to hear from more of you, but from the responses so far, here’s a taster of what the ocTEL community looks like, and what you hope to achieve.
Pie chart of location of respondents
As you can see from this pie chart, most respondents are based in the UK.

The majority also indicated that they work in Higher Eduction, but some cover other sectors including the NHS, private and third sectors. Over half are learning technologists or lecturers, but other roles such as manager and teacher are also represented.

We’re also interested in what experiences participants had prior to taking part in ocTEL. Responses so far suggest most have some experience of using technology for teaching, and of using social media (SM) for their own learning and for networking. Perhaps reassuringly for the course team (!), there are also participants who have joined ocTEL with a view to broadening their understanding in these areas. In the words of one respondent “I know I have big gaps in my knowledge regarding pedagogical theory and instructional design. I’d like to start ‘shading in’ these areas and find out what else I need to know for the future.”

Read the rest of this entry →

What are your expectations of ocTEL?

21/04/2013 in Course information, Evaluation

As noted in Friday’s overview, the team is keen to hear your feedback about ocTEL, and to act on it! We also have fairly detailed plans for a broader evaluation of ocTEL, which includes investigating impact on course participants. I have written in more detail on my blog about the thinking behind the evaluation, as well as the key evaluation questions we’re looking to address. These are organised in four themes:

  • Impact on staff competency (value in practice)
  • ocTEL Content and Design
  • ocTEL Discourse and Knowledge
  • ocTEL Community and Sustainability

There is also a sub-theme looking at the collaborative approach to open authoring and the impact of ocTEL on the course team.
Read the rest of this entry →

There’s nothing new in MOOCs

10/01/2013 in About this course, Evaluation

Towards the end of last year I started working with the ocTEL team on how to evaluate the Massive Open Online Course or MOOC they’ll be running in 2013. There is a *huge* amount of discussion on MOOCs at the moment from tips for those taking a MOOC to the future of MOOCs and HE. This article from the Chronicle provides a rather US centric timeline of developments and responses to MOOCs. There is also the recent launch of the slightly mysterious Futurelearn in the UK. There has been some very interesting discussion on this on the ALT mailing list. The list is members only, but quoting from a recent list posting by Diana Laurillard “Everyone in the field knows there’s nothing new in MOOCs” but we do need to meet the “massive demand for education, across all sectors from primary to lifelong, all over the world… It can’t be done without technology… It’s time to start looking seriously at what those models could be.

For me at the moment, the focus is on evaluating just one MOOC, which is still under construction. Nonetheless, when I started asking the ocTEL team what they hope the MOOC will provide, one of the themes that emerged quite strongly is the desire for a sustainable community to come out of the MOOC. This has led me back to work I undertook over ten years ago on the theory and practice of online learning communities. At that time, I was influenced by Etienne Wenger’s insights into Communities of Practice and Jenny Preece’s perspective of designing usability and supporting sociability in online communities. Figure 1 summarises the dimensions of practice Wenger attributes to ‘community’, and Figure 2 shows the relative similarities in Preece’s online community key features.

Dimensions of practice as identified by Etienne Wenger

Figure 1 – Dimensions of practice as the property of a community
(Wenger, 1998, p73).

Online Community Features as identified by Jenny Preece
Figure 2 – Key features of an online community, with associated characteristics
(Adapted from Preece, 2000)

These models will be reviewed and elements drawn into the ocTEL evaluation framework. I’ll also be looking at the concept of sustainability in relation to community. This has been addressed before by Bell et al in their 2007 book chapter entitled Evaluation: a link in the chain of sustainability. They highlight how the lack of persistence could come down to coordination failure or when the “costs of participation exceed the perceived benefits“.

So perhaps there is nothing new in MOOCs, or maybe it’s just a reminder to put old lessons into practice.

Bell, F et al (2007) Evaluation: a link in the chain of sustainability. In Lambropoulos, N & Zaphiris, P (Eds.) User-Centered Design of Online Learning Communities. Idea Group Inc.
Harris, RA & Niven, J (2002) Retrofitting theory to practice – a reflection on the development of an e-learning community. In Banks et al (Eds). Proceedings of Third International Conference on Networked Learning. Sheffield University.
Preece, J (2000) Online Communities: Designing Usability, Supporting Sociability. Chichester, John Wiley and Sons.
Wenger, E (1998) Communities of Practice. Learning, meaning and identity.Cambridge University Press, Cambridge.

** Adapted from a post on the Inspire Research blog. **

Skip to toolbar