Reflections on the #oerrub agile approach to evaluation

image of reports

Dull but worthy . . .


(image: http://pixabay.com/en/notes-office-pages-papers-print-150587/)

I’ve already posted some reflections on the agile approach that the OER Research Hub has been developing. In this post I’m going to try and share some of my reflections on my role as an evaluation consultant to to project and the agile or flexible approach we have developed to my input and (open) outputs.

Evaluation should be a key part of any research project, built in from the start and not something that is just left until the end of a project. However,sometimes it can slip off “the list”. As well as evaluating actually research outputs, it is also important to evaluate the processes that a project has used. In the case of the OER Research Hub, evaluation has been build in from the start, and their own, open, evaluation framework details their evaluation approach along with a pretty comprehensive overview of project evaluation.

Having this framework has made my role as an evaluator much easier. I had a very clear starting point with specific questions developed by the team which were driven by the overarching aims and objectives of the project. The framework guided me in my exploration of the project and focused my discussions with the team.

However, the framework is just that, a framework. It doesn’t “do” the evaluation. One of the things I have really enjoyed about my role with the project has been the flexibility, agility and open-ness of the team in terms of my input and in turn outputs.

Last year I worked with the project as they approached the end of their first year of funding. At that stage the project was still in the early days of its collaboration developments and data collection and so my main focus a review of the work so far, and to work with the team in terms of dissemination planning for the remainder of the project. I was also actively encouraged (in fact it was in my contract) produce blog posts as outputs. This is, I think still fairly unusual for evaluation activity, but it fits well both for research project about OER and open education, and my own open practice.

Other outputs from me included what I called my “brain dump” of my initial reactions and thoughts on the project outputs so far, some SWOT analyses, and a “dull but worthy” summary report. These were shared only with the team.

Even in open research not everything can or in many cases should be open, particularly if, as last year, the evaluation is focusing more on the mechanics of the project rather than the outputs themselves. I am a firm believer in making things open, but that what “stuff” you decide to make open is useful. Some of my outputs were only of use to the team at that particular time. However, the sharing the overall approach in a open way via this post is probably a more appropriate, open and (hopefully) useful resource for others.

This year my role has evolved again to more of what I would call a more of a critical friend. The project funders, the Hewlett Foundation are conducting their own evaluation of the project, so I have been working with the team in reviewing their outputs in relation to the focus of that evaluation. As with last year there has been some flexibility in terms of my input and outputs, but again blog posts have been part of the contract. This year I have spent most of my time meeting and talking with the team. I have seen my role more about encouraging reflection and talking through the teams next steps in relation to their data, findings, dissemination and sustainability.

It’s the latter where I think the real challenges lie. I don’t want to steal the thunder from the project, but they have got some pretty good evidence on the impact of OER (emerging findings are already being shared via their infographics page and blog posts). Their OER impact map is already providing an innovative and meaningful way to search and explore their data. But what next? How will the work and findings be built on both in the OU and the wider (open) education community? Will this project provide a secure foundation for an emerging research community?

These questions are key not only for the project, but also for their funders. The Hewlett Foundation have spent a lot (over $100 million) on OER over the past decade, so what is next for them? In terms of mainstreaming OER has the battle really been won? Martin and I have slightly different opinions on this. The project research is showing some really strong evidence in a number of areas in terms of winning/impact. But we are still only scratching at the surface and most of the research is pretty much North American focussed. Some of the models and evidence, particularly around text books, doesn’t have as much relevance in other parts of the world. More global research is clearly needed and is very positive to see the collaborations the project has developed with organisations such as ROER4D.

Building a new research community and discipline take time. However having a research element built into projects could provide additional stimulus, security and as well as short and long term sustainability. Is the future of the OER Research Hub as a set of static tools and guidance, or something more organic that provides a focus not only in supporting to grow a research community, but also in aggregating up evidence and sharing wider trends back to the community? In parallel with the continuum of reuse of OER highlighted, surely there needs to be a continuum of research.

Again I will be producing another “dull but worthy” report along side my blog posts, but if you want to join a wider conversation about open reflection and evaluation have a look at the current Open Researcher Course. There is a week of activities dedicated to the area, including a couple of good overview videos from Leigh Anne Perryman who also wrote the OER Research Hub Evaluation framework.

Where Sheila’s been this week – revisiting #OERRHub and the researcher as an API analogy

I spent the early part of this week in Milton Keynes with the OER Research Hub team as part of the second phase of the project evaluation.

When I worked with the team last year one of the things that intrigued me about the project was the fact that they were planning to apply and adapt an agile programming  approach to the project.

As I pointed out then, I felt there could be challenges with this as typically the outputs from research projects aren’t as concrete as most software development products, but I could see the attraction of this approach.

Bringing researchers who form part of a globally distributed team together for set periods to focus on certain aspects of research project does make sense. As does having some kind of structure, particularly for focusing “group minds” on potential outputs (products), adaptation of peer programming could be useful for peer review etc. However implementing “proper” agile programming methodology to research is problematic.

But if we stick with the programming analogy and stop thinking in terms of products, and start thinking of research as a service (akin to software as a service) then maybe there is more milage. A key part of SaS approaches are APIs, allowing hooks into all sorts of sites/ services so that they can in effect talk to each other.

The key thing therefore is for the researcher to think of themselves more as the interface between their work, the data, the findings, the “what actually happened in the classroom” bits and focus on ways to allow as wide a range of stakeholders to easily “hook” into them so they can use the outputs meaningfully in their own context.

In many ways this is actually the basis of effective digital scholarship in any discipline and of course what many researchers already do.

A year on, and after experiencing one of the early project sprints how has it worked out?

Well everyone knew that the project wouldn’t be following a strict agile methodology, however key aspects, such as the research sprints have proved to be very effective. Particularly in focusing the team on outputs.

The sprints have allowed the overall project management to be more agile and flexible. They have brought focus and helped the team as a whole stay on track but also refocus activity in light of the challenges (staff changes, delays to getting surveys started etc) that any research project has to deal with. As this is very much a global research project, the team have spent large chunks of time on research visits, going to conferences etc so when they are “back at the ranch” it has been crucial that they have a mechanism not only to report back and update their own activities but also to ensure that everyone is on track in terms of the project as a whole.

The sprints themselves haven’t been easy, and have required a lot of planning and management. The researchers themselves admit to often feeling resentment at having to take a week out of “doing work” to participate in sprints. However, there is now an acknowledgement that they have been central to ensure that the project as a whole stays on track and that deliverables are delivered.

I was struck this week by how naturally the team talked about the focus of their next sprint and how comfortable and perhaps more importantly confident they were about what was achievable. It’s not been easy but I think the development, and the sustaining of the research sprint approach over the project lifespan has paid dividends.

Returning to the wider API issue, last year I wrote

I wonder if the research as API analogy could help focus development of sharing research outputs and developing really effective interactions with research data and findings?

Again, one year can I answer my own question? Well, I think I can. From discussions with the team it is clear that human relationships have been key in developing both the planned and unexpected collaborations that the project has been undertaking. At the outset of the project a number of key communities/agencies were identified as potential collaborations. Some to these collaborators had a clear idea of the research they needed, others not so much. In every case as the research team have indeed been acting as “hooks” into the project and overall data collection strategy.

These human relationships have been crucial in focusing data collection and forging very positive and trusted relationships between the Hub and its collaborators. Having these strong relationships is vital for any future research and indeed, a number of the collaborations have extended their own research focus and are looking to work with the individual team members on new projects. As findings are coming through, the Hub are helping to stimulate more research into the impact of OER and support an emerging research community.

One of the initial premises for the project was the lack of high quality research into the impact of OER, they are not only filling that gap, but now also working with the community to extend the research. Their current Open Research course is another example of the project providing more hooks into their research, tools and data for the wider community.

The project is now entering a new phase, where it is in many ways transitioning from a focus on collecting the data, to now sharing the data and their findings. They are now actually becoming a research hub, as opposed to being a project talking about how they are going to be a hub. In this phase the open API analogy (imho) can only get stronger. If it doesn’t then everyone loses, not just the project, but the wider open education community.

The project does have some compelling evidence of the impact of using OER on both educators and learners (data spoiler alert: some of the differences between these groups may surprise you), potential viable business models for OER, and some of the challenges, particularly around encouraging people to create and share back their own OERs. For me this is particularly exciting as the project has some “proper” evidence , as opposed to anecdotes, showing the cultural impact OER is having on educational practice.

In terms of data, the OER Impact Map, is key hook into the visualizing and exploring the data the project has been collecting and curating. Another phase of development is about to get under way to provide even more ways to explore the data. The team are also now planning the how/where/when of releasing their data set.

The team are the human face of the data, and their explanations of the data will be key to the overall success of the project over the coming months.

More thoughts to come from me on the project as a whole, my role and agile evaluation in my next post.

Where Sheila’s been this week – student summer of innovation and known-ing things

On Wednesday this week I had a fantastic day in London with the successful Jisc Student Summer of Innovation projects. Building on from last years project, I think this is one of the best things Jisc has ever/is doing. It truly is putting students at the heart of the student experience. If you aren’t familiar with the programme you can find out more here. Basically students pitch their ideas for improving the student experience online, then via a process of voting and review a number of projects are selected to receive funding to develop their ideas along with mentoring and advice during the process.

I managed to spend a day in August with the projects as they refined their original ideas, and this week it was inspiring to see just have far they have all developed in such a short time. What fascinates me too are the clear themes and issues that the students themselves are identifying as areas that need new student facing services.  Feedback, study support, mentoring support both in terms of students at uni/college and those about to start all featured last year as well as this.  Hearing statements from students during their pitches stating “there is no culture of feedback in HE’ is a wake up for all of us. I don’t think there is a Uni in the country just now that doesn’t have some kind of assessment and feedback project/guidelines/support, but clearly some students aren’t seeing the impact of those yet.  Data was a big thing with the projects too – analytics, dashboards formed a large part of the pitches from a number of the projects.  This may be partly due to smart thinking from the project teams.  Data  and analtyics is not only an area that Jisc  is very keen to develop new services in, but it is also a reflection of the “data is the new oil” mentality in software development more generally. There are still huge assumptions that data from every service will actually be useful and that people (staff and students alike) will have time and capability to act on it in meaningful ways.

Alan Greenberg, former Education Executive at Apple also gave talk on “education technologies, insights and contexts”.  His insights into the business side of developing technology for the education sector was I’m sure invaluable to the projects.  I have to say, parts of it made me slightly uneasy as it did seem to be leading to a very content centric, and reductionist data driven view of education.  I know ultimately Jisc does want to develop some of these projects as market ready services, which is great. But imho, the strength of this whole programme is the experience that it is giving the students. Not all the ideas will be able to become services, or be successful.  Not all the project teams have time to fully commit to them as they are still studying. However the impact of the experience they are part of will stay with them, and having this safe space to experiment is really, really important. I’m sure it will stay with them for the rest of their lives and impact on whatever they do next.  Below is a my sketch note of Alan’s talk (note to self, don’t leave home without ipad or coloured pens again!)

Notes from Alan Greenberg talk, 17 September 2014

And finally, something I was going to write about last week but didn’t. Following David Kernohan’s “you’ll never hear surf music again” talk at ALT-C and the general “twitter isn’t what it used to be we need to move somewhere else” debate, in an attempt to keep up with the #edtechhipsters, I’ve been looking at Known, a new social publishing platform. Last week I wasn’t quite sure how I would actually use it and how/where it fitted into my existing online spaces. However last night after voting in the Scottish referendum, I want to share something more than a tweet or Facebook status update on how I was feeling. I remembered  Known and I’m now seeing it as a counterpoint to this blog, which will remain very much work related and focused.

 

 

Living with the VLE dictator

You know how it is, you listen and read to some “stuff” (and seriously great stuff that is worth listening to and reading). It sets all sorts of triggers in your head about how you work, what you do, and more importantly what you can do to in response. You get a great title for a blog post, then you see from your network that someone else has pretty much written what you had been thinking, but far more eloquently than you. At this point, dear reader, you really can just read this post The False Binary of LMS vs Open from D’Arcy Norman.

However, as the roots of this post were really seeded by listening to Audrey Watter’s recent Beyond the LMS presentation at the University of Newcastle, and as I’m still thinking about her ALT-C keynote and the importance of non North American narratives, I’m going to continue with my tuppence worth.

“Blackboard sucks” – that’s the consensus right? But as Audrey pointed out, even the new LMS/VLE kids on the block are selling their products by saying things like “it’s like just like Blackboard, except it’s blue like Facebook”. They are all about management, administration and not about the learner. They are built on a very traditional model of education. They are walled gardens. If you haven’t listened to Audrey’s presentation, you must.

As I was doing just that on Friday, a number of things were swirling through my head. At this point I probably should mention that here at GCU Blackboard is our VLE and for quite a while I have been mulling about writing a post titled “why I quite like Blackboard”.

Had it taken less than a year for me to be indoctrinated by the evil dictatorship that is Blackboard and by default all other VLEs? Am now I a willing conspirator in maintaining their status quo? Shouldn’t I be leading the insurgency or at least doing more to fight for open? At the sametime, scarily I was thinking terribly un-pc thoughts about benign dictators holding things together, and wondering if I could write a witty, yet well informed post comparing educational technology to the current situation in the Middle East or closer to home the Scottish independence referendum. I quickly realised that I probably couldn’t.

This morning via my networks came across D’Arcy’s post. And as I said, he had kind of written my post. Like D’Arcy, I work with and support the need for the boring, but oh so important administrative functions that our iteration of Blackboard support and that are needed for teaching and learning just now. If we got rid of Bb, I think it is fair to say there would be a fair amount of chaos for our students and staff alike. I have been in several meetings over the last year where a new shiny (and sometimes not so shiny) thing has been talked about with almost awe and wonder. This despite the fact that it just duplicates what are already doing within Bb but without the crucial integration “thangs” that automagically assign modules to students and staff. In these cases I have very much been advocating sticking to the ‘devil we know”, and trying to have a more holistic conversation about learning, where and how it (could) takes place in our context. I don’t want us to just move to something else that does the same thing only with a slightly nicer interface – if we are going to jump I want that jump to matter.

Too often our some of my colleagues really have no idea about what our students and staff are actually doing in terms of collaboration, networking. Because they don’t see it everyday, they think it doesn’t take place. Bb is one of our most stable systems too which again often goes unnoticed and unreported or there is an assumption that no-one uses it.

We are encouraging and seeing more sophisticated use of learning technology across our institution, we are committed to blended learning not only in the sense of blending f2f and online teaching, but also in terms of blending the systems we use. We can (and do) blend third party systems with Bb. Increased use of specs like LTI is opening up new possibilities. Bb themselves are going through some big changes and have been very supportive in listening and reacting to our need. Oh yes, I hear you sigh, that’s because they want to continue to get your business. Which of course is true, but from what I can gather that hasn’t always been the case.

Of course, changing culture is the key to making any technology have an impact in education (or anywhere else), and Audrey did highlight that in her talk. Much as I would love to experiment with more open, connected, student owned technologies such as the example she gave of the “domain of their own” the University of Maryland Washington, the culture in my institution isn’t quite ready for that yet. But it is a great example and one I will be sharing with colleagues and looking to see if we could do anything similar. I am seeing an increasing positive trend in terms of portfolio development which encourages and facilitates networking and open sharing by students.

Networks are also crucial for staff to share ideas, narratives, experiences, and often for me a sanity check. Realistically I don’t have enough influence in my institution to make sweeping changes, but I hope that I can bring ideas back from my network which can help move forward, or at least open up some different debates around our thinking and development of learning and teaching. I need to hear people like Audrey to make me reflect on my practice and share ideas with my internal nework.

So, although in one sense I may be living within a dictatorship, I do believe it is a changing one, one that is trying to catch up with evolving expectations. I may not be leading an insurgency but I hope that I am able to influence changes from within so that we have a truly flexible infrastructure and support mechanisms to allow the space and security for some radical thinking and changes to take place.

Summary of #GCUGamesOn Evalution Findings

As promised this post shares the summary findings from our recent online event, GCU Games On. As I’ve written about before we developed this very quickly (in a month from idea to online) so we were very aware of some of the pedagogic shortcomings of our overall design. However given the rapid development time during the start of summer holidays when most of our subject experts were on holiday we had to make some very pragmatic design decisions.

Overall the feedback was pretty positive and the whole experience is helping to shape our developing strategy to open, online courses. (Nb the text below has been adapted from an internal report).

Background

GCU Games On was an open online event designed to celebrate, explore and share experiences during the Glasgow 2014 Commonwealth Games. It ran between 16 July and 8 August 2014.

Instigated by the PVC Learning and Student Experience, it was developed in little over a month. Due to the time constraints (one month from idea to being available openly online) a simple design was developed which included: background and contextual information with relevant links, making a wish on our digital wishing trees, at least one twitter based activity and a medal quiz challenge each week. Sharing experiences of Glasgow 2014 via twitter was encouraged each week. Daily email updates were sent to all registered participants.

The event was delivered via the new Blackboard Open Education platform.

Participation

  • Registrations: 211
  • Countries: 12 excluding the UK
  • Digital Badges issued: 174
  • Tweets: 424
  • Digital wishes: 107

Evaluation

Of the 211 registrations, 22 completed the survey giving a 10.4% response rate. In addition, due to the use of social media (and in particular, twitter) a number of informal responses to the event were shared.

Summary Findings

The majority of respondents to the survey were female, aged between 25 to 65, based in the UK with no connection to GCU. The majority of participants were based in the UK, with 36% based in both Glasgow and Scotland respectively. 18% of respondents were from the rest of the UK, and there were equal numbers (4.5%) of respondents from other Commonwealth countries and non Commonwealth countries. From registration information we know we had registrations from Australia, India, Trinidad & Tobago, Ireland, Israel, Denmark, Canada Italy, Israel, New Zealand, Spain and South Korea.

59% of respondents had no connection with GCU and 45% of respondents cited wanting to experience online learning at GCU as their main reason for participating. The vast majority of respondents had some form of formal educational qualification, 45% up to Masters level.  This correlates to general trends in open online courses, but may also reflect a network effect from the Blended Learning Team’s network and promotion of the event. 95% of respondents found the site easy or partially easy to use.  54% of respondents completed all of the activities.

Open feedback was generally positive about the experience.

“I really enjoyed this as a bit of fun.  What I got out of it most was seeing new blackboard system in operation and it looks and feels very impressive.”

“I think looking at the Twitter feed this was spot on for what it was trying to achieve. Much fun was had by all it seems and the course gave a great scaffold to talk about their experiences at the games.”

“I do know it is hard to pull together a learning experience around an event like this and I guess that was weakness of this approach.  At times I think really perhaps due to lack of substance or clear learning outcomes – the learning design was a bit hit or miss – but I think you did achieve outcome of getting folks to engage with learning platform which was I think what it was about rather than the content”

 

GCU Games On Gold Medal

GCU Games On Gold Medal

Slides and stuff from #altc

Can’t believe it was only a week ago, but . . .  last Monday was quite busy for me at ALT-C all three of my presentations were scheduled for then.  They kind cover a lot of the “stuff” I’ve been involved in over the past year. Below are the slides.

Also “click here” to see all my tweets and pics from the conference.

Where Sheila’s been this week – navigating the marvellous and the monsters #altc 2014

This week I’ve managed to avoid that back to work and “omg look at that inbox” feeling for a few days by attending the annual #altc conference.  Last year’s conference was a bit of a personal highlight for me, but this year the team at ALT and all the co-chairs pulled off another great event,including two of the best keynotes I have been to in a long time from Catherine Cronin and Audrey Watters.

Both Audrey and Catherine highlighted the need for engagement to ensure the authentic voices, stories and experiences (from all of the education sector, not just ed tech) voices are heard and ensure that new noisy narratives (particularly from certain commercial sectors) don’t become the defacto history and more worryingly, future of education.

Audrey wove together an inspiring narrative (including references to Ada Lovelace, Roald Dahl, Luddites, Mary Shelly, Byron, F B Skinner to name but a few) about the creation of monsters, lost histories and control.  She reminded us that we can control the development of technological monsters through our combined efforts to inspire love technology and education. Catherine reminded us of the inherent ethical and political nature of education and how openness, online spaces and new forms of identity can empower us to make our voices heard. However, within this marvellous new open world there are power struggles too.

We may laugh at the idea of teaching machines, but in our drive for ever increasing personalised, mobile access to education, content vendors and governments are often (knowingly and/or unwittingly) all too willing to ignore the narrative from educators and buy into a behaviourist, watch video -> click through -> mcq-> (pay for) certificate = learning solution.

Helen Beetham made a very good point after Audrey’s talk that in the UK we do have a different narrative – particularly around the student engagement agenda within HE which is different from our North American cousins. But we are not immune, as the FELTAG discussions (particularly around % of online content) and stories illustrated.

There were many, many great stories shared over the three days of the conference. John Traxler reminded us of the danger of assuming our  developed, western global ideologies, learning theories and learning designs don’t automagically meet the needs of many emerging cultures (in particular Africa).  David Kernohan provided an entertaining yet mindful tale of how the heady days of open and social collaboration may well be at an end, as big business, governments and employers (included HEIs) start to close down, commercialie and control access and impose censorship.

Conferences like ALT are a great way for our community to strengthen and share our own “folklore” and build our collective narrative around the positive impact of technology within learning and teaching. We need to keep sharing our stories openly, and ensure our narrative, folklore, collective knowledge and wisdom is developed and shared as widely as possible.  Roll on ALT-C 2015.

Here are my visual notes from all three keynotes (NB if you click on the images you’ll go to full size, CC licences copies on flickr)

Visual notes from Jeff Haywood keynote, altc 2014

Visual notes from Jeff Haywood keynote, altc 2014

Visual notes from Catherine Cronin keynote ALTC 2014

Visual notes from Catherine Cronin keynote ALTC 2014

Visual notes from Audrey Watters keynote ALTC 2014

Visual notes from Audrey Watters keynote ALTC 2014