This is an archive of the 2013 version of ocTEL.

10: Evaluating TEL and reflection

This is the last week of the course and we have saved a key topic for this week: evaluation. During this week you will identify commonly used methods of evaluation and review existing evaluation tools. You will finish the course by discussing your own experiences of ocTEL.

This is not only about you and the course, but also about how we can make things better next time round. As you know, the course was designed, written and largely supported by Members of ALT (see who’s who on our ocTEL team page) and your feedback can make a big difference to how we develop and run the course in future.

As we’ve already noted, the evaluation of ocTEL focuses around four themes. These aim to address individual perceptions of success, and we explored learners’ expectations near the start of ocTEL. But, the evaluation is also very much informed by the original intent of ocTEL, ie to help participants better understand how to use technology to enhance their teaching in Higher Education. The evaluation themes are:

  • Impact on staff competency (value in practice)
  • ocTEL Content and Design
  • ocTEL Discourse and Knowledge
  • ocTEL Community and Sustainability

Thus the evaluation also has a clear aim to learn from the experience of designing and delivering a MOOC, and to make use of this to inform future ocTEL cohorts.
ocTEL_badge

Accreditation and recognition

While we could not implement a formal accreditation mechanism for the course this time, we are working towards mapping the course against the criteria of CMALT (ALT’s accreditation scheme) and the UKPSF as well as evaluating Open Badges as a way for future participants to gain recognition for undertaking ocTEL.

This time, there are two options for all participants who would like to show that they have participated:

  1. Complete an electronic certificate of participation;
  2. Use the ocTEL badge image online.

The certificate and the badge will be available via the ocTEL site at the end of the course.

This week’s aims

Here are some of the challenges that we hope this module will help you tackle:

  • How can I evaluate my TEL project/course?
  • What are the most effective tools for evaluation?

And here are the learning outcomes we hope you will get out of it

  • Appreciate the importance of project and course evaluation and how to carry it out
  • Understand methodologies for carrying out evaluation
  • Critically review evaluation instruments or tools
  • Reflect on how your experience of the ocTEL course has compared with expectations, and how you can best build on this experience

If you only do one thing…

Evaluation is a key part of the project life-cycle, but often gets forgotten or given only a light touch. Considering a TEL initiative you’ve been involved with, how did you know it had been successful or provided benefit? Was the initiative evaluated? If not, why not? If yes, what evaluation took place, what did you and your stakeholders need to know and how were the findings used?

Using the headings below (either in a list or as a table)  reflect on the different ways you have come across for evaluating the effectiveness and impact of TEL. What are the advantages and disadvantages of each approach?

  • What do you need to know? e.g. how learners rated the support and how this differed according to their circumstances or demographics
  • What evaluation methods could help you find out? e.g. online survey
  • What advantages do these methods have? e.g. easy to collect information from large of people
  • What disadvantages do these methods have? e.g. often require incentives to increase response rates, disaffected learners may be less likely to respond

Read Evaluating Technology Supported Learning from Edinburgh Napier University. Compare the methods you have identified with what the authors have suggested.

Share your thoughts on your blog or via Twitter (using #ocTEL tag), on this forum topic, or via the JiscMail list.

Come and join the live webinar and end of term get together

To mark the final session of ocTEL, this week’s webinar comes in two parts, reflecting the double roles of our guest, Dr Rachel Harris. In the first part, Rachel will give an account of her experience of evaluating a large number of Technology Enhanced Learning projects in higher education. Rachel is also the official evaluator of ocTEL, and in the second part she will lead a live evaluation and feedback session about ocTEL.

The webinar runs from 12:30 to 13:30 British Summer Time on Thursday 20 June. You can access it from 12 noon onwards via this link.

We hope the webinar will bring together many of the authors and tutors who have developed and supported ocTEL. We will be holding a virtual end of term get together for all involved to discuss our experiences of the course, get some feedback and share memories. The session will end at a look at the future development of ocTEL and provide inspiration of how participants can continue their involvement in the course and the Association for Learning Technology.

Rachel Harris directs Inspire Research Ltd, an independent consultancy specialising in evaluation and research into learning. Rachel has over 20 years experience in evaluation, research and development, much of which has focused on technology enhanced learning. This has included leading a team to establish an international online collaborative environment; supporting small practitioner-led e-learning initiatives and a university-wide virtual campus; acting as an evaluation critical-friend on Jisc-funded programmes; and independently evaluating various learning related projects. Rachel is a long-standing member of ALT, the BPS, the HE Academy, and serves on ALT’s Research Committee.

Activities for this week

Activity 10.1: Reflecting on evaluation instruments/tools

(1 hour)

On your blog or via Twitter (using #ocTEL tag), on this forum topic, or via the JiscMail list, share an instrument or tool that you have used to evaluate and measure impact and provide a short critique on its effectiveness. Read and comment on entries from two or three other participants.

Activity 10.2: Revisit your case study from Week 9

(30 minutes)

If you developed a case study in Week 9, add in a section on how you evaluated the effectiveness and impact of the project, and re-publish the case study. If you have not developed a case study, you can use a different example instead.

Activity 10.3: What do you think about ocTEL?

(90 minutes)

For you

We started the course 11 weeks ago by thinking about one big question and we hope that you have kept in mind what it was for you. As part of the final activity for the course, we’d like to encourage you to revisit your ‘big TEL question’ and see how your view of it has been changed and reframed by the course (and indeed whether it it still the same!). Share your thoughts on your blog or via Twitter (using #ocTEL tag), on the original forum post if you made one or this forum topic, or via the JiscMail list.

We also recommend that you take a bit of time to look through the weeks and bookmark or otherwise keep a record of the best bits for you. What is most relevant for your practice? Which resources could you use or re-use?

The materials for the course including the recordings of live sessions and the forums will remain on the ocTEL website so that you can refer back to them later, if you are missing something.

For ocTEL:

As we mentioned in the introduction to this week, we are seeking your feedback on any and all aspects of the course, including its platform, delivery, content and format. Completing the evaluation should take approximately ten minutes.

Thank you for your time and feedback!

Checklist

  • Assess different approaches to evaluating TEL projects or teaching and learning implementations
  • Come and join our webinar and end of term get together on Thursday
  • Download the certificate and online badge to tell the world you’ve participated
  • Complete the ocTEL evaluation form
  • Have a look at what’s next and keep in touch!

Finished ocTEL? What’s next?

If you have been finding ocTEL useful, then there is much coming up that you might get involved in. Here’s a starter for 10:

  1. Keep in touch with ocTEL: Follow @ALTocTEL to get involved in or notified about ocTEL 2.0 as a tutor, author or participant;
  2. Become a member: Join ALT and get active in our community starting from £10/year;
  3. Come to the conference: Register to attend this year’s ALT conference in September;
  4. Gain accreditation: Find out more about how you can gain recognition for the skills you have using ALT’s peer-based accreditation scheme CMALT;
  5. Focus on your interests: If you have found you are interested in a particular topic, consider joining one of our Special Interest Groups or Regional Special Interest Groups;
  6. Share your thoughts: If you have a blog post, case study or other materials you would like to share more widely, submit them to the ALT newsletter;
  7. Get published: submit your research to be published in Research in Learning Technology;
  8. Follow @A_L_T: keep informed about upcoming events, publications and news by following @A_L_T on Twitter;
  9. Watch the most popular videos: visit ALT’s not-for-profit YouTube channel to watch many of the videos featured on the course, see URL;
  10. ocTEL alumni: As an ocTEL alumnus we want to keep in touch with you and spread the word, so do keep in touch!

Resources and more to watch, read and research

Notes and commentary

Evaluating what you have implemented is important in understanding how effective your course or project has been. Oliver (2000a) identified five approaches for carrying out an evaluation of online learning:

  • Formative – carried out by members of the course team for the benefit of the course team as a way of identifying revisions and improvements.
  • Summative – carried out by an external auditor intending to provide judgement about whether a project has met its aims or a course has met its learning outcomes.
  • Illuminative – an approach which seeks to understand the perceptions of the participants in terms of what is important to them through phases of observation, inquiry and explanation.
  • Integrative – a combination of the summative and integrative approaches.
  • Evaluation for Quality Assurance

Once you’ve decided the type of evaluation it’s important to consider the following aspects of the evaluation cycle, from defining aims, through data collection and analysis, to drawing conclusions, and reflecting back on the aims. For more information about the evaluation cycle see Embedding Learning Technologies – Evaluation Cycle (pdf).