Week 6 – Where do we go from here: enhancement, evaluation, and reflection

With a mixture of sadness and excitement we enter the final official week of ocTEL. Since the end of April it has been a joy to see participants of ocTEL contribute over 300 blog post, 800 forum contributions plus reaction and interaction in other spaces. The final week of ocTEL is on enhancement, evaluation and reflection. This is something the ocTEL team will be doing over the next couple of weeks as we assess what went well and what could be improved if ocTEL is run again. To that end we’ll be contacting you again towards the end of the week with a course survey and outlining our next steps. This final week is slightly different in that our webinar is scheduled for 12.30pm today (details below)! Hopefully this will give you a chance to engage with the content for this week earlier giving you more time to look back at previous weeks and see if there are any activities/resources you’d like to revisit in the final official week.

We’ve covered technology in ocTEL; we’ve covered learning. This week we’re focusing on enhancement – what shapes it takes, and how to achieve it – and evaluation – how we can decide that our interventions have been successful.

As you will all be at different stages in terms of introducing/working with TEL, we offer different routes through the week depending on which core activity you undertake.

Check-in Badge

10 Points

Click on this badge link to 'check-in' and see who else is also active in this topic.

This week’s aims

This week will help you achieve the aims you prioritise from the following:

  • identify and analyse the essential elements of successful educational enhancement initiatives based on TEL
  • analyse the cost effectiveness of a particular learning experience and use TEL to improve this effectiveness
  • appreciate the importance of project and course evaluation
  • understand methodologies for carrying out evaluation
  • critically review evaluation instruments or tools
  • reflect on how your experience of the ocTEL course has compared with expectations, and how you can best build on this experience.

If you only do one thing…

Choose between focusing on enhancement or evaluation.

TEL One Badge

150 Points

Each week we award a TEL One badge for completing 'If you only do one thing…'. Click on this badge title for full instructions

How to enhance the effectiveness and efficiency of particular TEL approaches

Watch either

Write down in short bullet point form a list of

  • elements of the Saylor or xMOOC approach that you think could be applicable to your context (what you’re involved in teaching, to whom, with what goals and constraints)
  • problems you might anticipate with the approach

Think of different situations where efficiencies might be important in

  1. reaching more people or providing a richer experience for the same cost
  2. reducing tutor costs by encouraging more elements of the learning experience to be peer-based or self-organised by learners
  3. reducing production and infrastructure costs by using free resources and technology
  4. taking the learners’ perspective in getting a sound, rounded, education with minimum financial outlay.

Describe those situations and identify any potential tensions or conflict in these situations between the goals of effectiveness and efficiency. Share your thoughts on your blog or via Twitter (using #ocTEL tag), on this forum topic, or via the JiscMail list.

OR

The Whys, Hows and Whats of Evaluation

Think about a TEL initiative you’ve been involved with.

  • How did you know it had been successful or provided benefit?
  • Was the initiative evaluated? If not, why not?
  • If yes, what evaluation took place, what did you and your stakeholders need to know and how were the findings used?

Reflect on the different ways you have come across for evaluating the effectiveness and impact of TEL, using the headings below. What are the advantages and disadvantages of each approach?

  • What do you need to know? e.g. how learners rated the support and how this differed according to their circumstances or demographics
  • What evaluation methods could help you find out? e.g. online survey
  • What advantages do these methods have? e.g. easy to collect information from large of people
  • What disadvantages do these methods have? e.g. often require incentives to increase response rates, disaffected learners may be less likely to respond

Compare the methods you have identified with the methods and approaches used by the contributors to this week’s webinar

Share your thoughts on your blog or via Twitter (using #ocTEL tag), on this forum topic, or via the JiscMail list.

Come and join the webinar

Enhancement and evaluation of TEL

Monday June 16 at 12.30 – 1.30 (BST).

This webinar will include three guest speakers who will talk about the enhancement and evaluation of a major example of TEL which they have been involved in. Our three examples will include one at module level, one innovation based on introducing new devices, and one based on an overall institutional strategy.

Our three speakers:

Mark Kerrigan is the Director of Teaching, Learning and Assessment in the Faculty of Health Social Care and Education for Anglia Ruskin University. He will be talking about the enhancement and evaluation of the iPads in Science initiative, a project by the University of Greenwich (where he used to work). More information and a case study on the project can be found here:

http://www.ucisa.ac.uk/bestpractice/Copy_of_publications/effective_use.aspx

James McDowell is University Teaching Fellow in the School of Computing and Engineering at the University of Huddersfield. He will be talking about the development of work which won him the 2011 ALT-Epigeum Award for the Most Effective Use of Video. This award recognized the important contribution he has made to the enhancement of student learning through his use of video feedback – http://eprints.hud.ac.uk/19915/ .

Keith Smyth is a Senior Teaching Fellow and Senior Lecturer in Higher Education at Edinburgh Napier University, where he leads the MSc Blended and Online Education and works on TEL-related strategic initiatives. Keith will be talking about the development, dissemination and evaluation of the 3E Framework for technology-enhanced learning, teaching and assessment. Since being introduced as the basis for Edinburgh Napier’s Benchmark for the Use of Technology in Modules, the 3E Framework has been adapted and implemented by over 25 institutions within and beyond the UK. http://3eeducation.org/3e-framework/

Watch the webinar using Blackboard Collaborate: http://go.alt.ac.uk/octel2014-week6-recording
Watch the recording of Blackboard Collaborate live stream on YouTube: https://www.youtube.com/watch?v=7c2QzMVq3b8

Webinar Badge

100 Points

Tune in or watch the weekly recording of each week's webinar to achieve the this badge. Click this badge link to view Instructions for getting this badge.

Activities for this week

Activity 6.1: Analysing the cost-effectiveness of new learning technologies

(1 hour)

  • Read why productivity and sustainability in HE is an issue, as suggested by The Economist’s take on “Baumol’s cost disease”.
  • Check out a few of the resources listed below (e.g. on evaluation and costing).
  • Do some searching to see if you can find further resources that link this general area to your specific concerns and context (use Google, also ask people you think might know of useful materials, collaborate with other ocTEL participants). Try and answer the question, “What can I take away from this area and apply in my teaching-related work?”
  • On your blog or via Twitter (using #ocTEL tag), on this forum topic, or via the JiscMail list, note your initial impressions about which strategies look promising or unpromising to you.

Activity 6.2: What do you think about ocTEL?

(90 minutes

We invited you at the start of this course to think about big questions and we hope that you have kept in mind what your question(s) were. As we approach the end of the course, we’d like to encourage you to revisit your ‘big TEL question’ and see how your view of it has been changed and reframed by the course (and indeed whether it is still the same!). Share your thoughts on your blog or via Twitter (using #ocTEL tag), on the original forum post if you made one or this forum topic, or via the JiscMail list.

We also recommend that you take a bit of time to look through the weeks and bookmark or otherwise keep a record of the best bits for you. What is most relevant for your practice? Which resources could you use or re-use?

The materials for the course including the recordings of live sessions and the forums will remain on the ocTEL website so that you can refer back to them later, if you have missed something or not had time to complete any of the activities.

Checklist

  • Post your reflections and contributions using the #ocTEL tag via your blog, Twitter, the online forums or other channels
  • Join the webinar/watch the recording
  • Check out the TEL Explorer activities
  • Tweet about your experience of #ocTEL and find other participants on Twitter

Resources and more to watch, read and research

Evaluation

Enhancement through increasing reach and/or quality for same cost

Enhancement through reducing tutor cost

Enhancement through reducing production and infrastructure costs

Being a low-cost learner

Be a TEL Explorer

This is additional content and an activity to enable you to explore topics that are really relevant to your own practice. Explorer activities are part of the course, so we’d like to encourage you to share your ideas on your blog or via Twitter (using #ocTEL tag), on  this forum topic .

Activity 6.3: Exploring enhancement and evaluation in practice

(2/3 hours)

  • Find an example of innovation in TEL which interests you.
    You can use other ocTEL participants as a resource here as they may know of innovative practice in whatever curriculum area you wish to explore. Other sources include websites from JISC, HEA etc. and awards for innovative practice such as NTFS and those from ALT, the Guardian and THES.
  • Contact the innovator and arrange a short conversation to explore (at least) the following issues:
    – what was the impetus and rationale behind the innovation?
    – what has the innovation achieved so far in terms of student learning?
    – what has its impact been on the staff involved and the wider institution?
    – how do you know it has had this impact? What evaluation strategies and methods have been used?
  • Write a brief commentary on the outcomes of this conversation and publish it via the blog or other methods you have been using on this course so far. We recommend you  check your summary with the innovator before publishing to make sure that you have interpreted everything correctly.

TEL Explorer Badge

250 Points

Each week we award the TEL Explorer badge for completing activities outlined in the course material. Click on this badge title for full instructions

Notes and commentary

The challenge of evaluation

Our webinar will look at examples of enhancement in TEL in higher education and compare them in terms of their objectives, the relative success of the intervention, and how it was evaluated.  These examples and their discussion will mean different things to you depending on where you’re starting from and what ambitions you have for your educational work. Here are some of the challenges that you might like to tackle this week.

  • How can we most efficiently help more people access learning experiences?
  • How can we take advantage of TEL and create a richer learning experience within resource constraints (e.g. by redesigning the curriculum and/or cutting overheads)?

During this week you will identify commonly used methods of evaluation and review existing evaluation tools. We hope that you will be able to find time to finish the course by discussing your own experiences of ocTEL. This is not only about you and the course, but also about how we can make things better next time round. As you know, the course was designed, written and largely supported by Members of ALT (see who’s who on our ocTEL team page) and your feedback can make a big difference to how we develop and run the course in future.

Evaluating what you have implemented is important in understanding how effective your course or project has been. Oliver (2000a) identified five approaches for carrying out an evaluation of online learning:

  • Formative – carried out by members of the course team for the benefit of the course team as a way of identifying revisions and improvements.
  • Summative – carried out by an external auditor intending to provide judgement about whether a project has met its aims or a course has met its learning outcomes.
  • Illuminative – an approach which seeks to understand the perceptions of the participants in terms of what is important to them through phases of observation, inquiry and explanation.
  • Integrative – a combination of the summative and illuminative approaches.
  • Evaluation for Quality Assurance

Once you’ve decided the type of evaluation, it’s important to consider the following aspects of the evaluation cycle, from defining aims, through data collection and analysis, to drawing conclusions, and reflecting back on the aims. For more information about the evaluation cycle see Embedding Learning Technologies – Evaluation Cycle (pdf)

The efficiency debate

The idea of getting maximum learning for minimum cost is met with wary scepticism in many quarters. It smacks of putting managerial values ahead of teaching, or of cutting costs to the point where the learning process becomes brittle and no longer has the agility and resilience to deal with the unpredictable demands of teaching (see David Noble’s paper, linked above, for an example of this critique).

Issues of deskilling teaching work or filleting the soul of education institutions are not trivial, and you can focus on them this week if you wish. But this is also an opportunity to ask about the different ways that learning and teaching are enhanced in Technology-Enhanced Learning. Do they result in learning outcomes that are richer in themselves, accessible to a wider range of people in a wider range of contexts, easier to achieve, or cheaper to accomplish?

Some enhancements are incremental, some are transformational. Sugata Mitra’s work on using technology as part of “Self Organised Learning Environments” in contexts where there is a lack of good quality teachers  hints at radically new models for supporting learning.

Realising the enhancements that may be possible depends on an understanding of learning context. We appreciate straight away that it’s cheaper and safer for pilots, dentists and surgeons to learn their skills using simulations, but other contexts of learning present different challenges. This week you have the opportunity to select a context of your choice and explore what kinds of enhancements might be possible to make the learning more productive for more people. You can make your own value judgements.

4 responses to Week 6 – Where do we go from here: enhancement, evaluation, and reflection

  1. I was last last week how long it would take me to design a blended course for low literacy and ESOL students with little experience of using computers let alone the internet.

    As an FE college as a whole we are committed to putting part of all the courses on-line. Partly no doubt because of cost but partly to prepare students for an on-line world.

    However, for a low literacy adult student to try to attend a blended learning course in a non-native language seems to have little benefit (if I’m wrong please contact me). That said, this year they have enjoyed TEL in the classroom where I (and more ICT experienced students) can support them. They have enjoyed using websites such as vocaroo and google docs for peer assessment.

    • Maybe there is potential in blended learning for these students if the technology is used in ways which would not be practicable in the classroom. What sort of access do these students have to technology outside the classroom? As ocTEL is going into archive soon, please drop me a note if you want to continue this conversation – profpeter@btinternet.com

  2. With just a couple of tiding up articles to make for OCTEL 2014 until I finish, the sun sets on what I have found to be an interesting course and online learning experience.

    I do hope that others can benefit from the course if and when it runs again.

    With the gamification of OCTEL grabbing my interest, I thought it only fair that I repay the OCTEL team with a badge of their own from me.

    Enjoy you have earned it!

    Cheers
    Glenn

    Thanks OCTEL Badge
    Thanks OCTEL Badge