University of Huddersfield Repository

University of Huddersfield Repository

University of Huddersfield Repository

Johnson, Mark, Lager, Peter, Pollard, Bill, Hall, Graham, Edwards, Miranda, Whitfield, Pete and

Ward, Rupert

Between analysis and transformation: technology, methodology and evaluation on the SPLICE project

Original Citation

Johnson, Mark, Lager, Peter, Pollard, Bill, Hall, Graham, Edwards, Miranda, Whitfield, Pete and

Ward, Rupert (2009) Between analysis and transformation: technology, methodology and evaluation on the SPLICE project. In: "In dreams begins responsibility" - choice, evidence, and change. Conference Proceedings, 16th International Conference, 8-10 September, Manchester, UK.

Association for Learning Technology, pp. 40-50. ISBN 978 0 95 458 709 3

This version is available at

The University Repository is a digital collection of the research output of the

University, available on Open Access. Copyright and Moral Rights for the items on this site are retained by the individual author and/or other copyright owners.

Users may access full items free of charge; copies of full text items generally can be reproduced, displayed or performed and given to third parties in any format or medium for personal research or study, educational or not-for-profit purposes without prior permission or charge, provided:

• The authors, title and full bibliographic details is credited in any copy;

A hyperlink and/or URL is included for the original metadata page; and

• The content is not changed in any way.

For more information, including our policy and submission procedure, please contact the Repository Team at: [email protected]

ALT-C 2009

“In dreams begins responsibility”

— choice, evidence, and change

Conference Proceedings


Production editing by Lucy Saxton, ALT.

Produced with the assistance of

Melanie Fox (ALT), Oxford Abstracts, and Bob Pomfret

Design by [email protected]

Cover Design by Bob Pomfret

Printed by Nufield Press

ALT gives permission for authors to reproduce and distribute their own work on the understanding that the origins of the publication are fully acknowledged.

Please use the following format for referencing:

Your name (2009). Your paper title. In

Damis, H. and Creanor, L. (Eds). “In dreams

begins responsibility” — choice, evidence and change. The 16th Association for

Learning Technology Conference (ALT-C

2009). Held 8–10 September 2009,

University of Manchester, England, UK

Copyright of individual papers remains vested with individual authors and / or their institution. Other material is copyright

2009 the Association for Learning

Technology, and is Creative Commons licensed (Attribution-Non-Commercial 2.0

UK: England and Wales).

ALT is a registered charity, no. 1063519

ISBN 978 0 95 458 709 3

This document is available online at:


Section 1: Research Papers

0114 Taking the lead: learners’ experiences across the disciplines . . . . . . . . . .1

0299 Is there a Net generation coming to university?. . . . . . . . . . . . . . . . . . . . .10

0178 All hands on deck: CREWED for technology-enabled learning . . . . . . .21

0055 ‘Unfettered expression of thought’? Experiences of anonymous online role play . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31

0084 Between analysis and transformation: technology, methodology and evaluation on the SPLICE project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40

0031 Technology enhanced learning in 21st century mass higher education. Aspects of design, practice and strategy for a necessary step change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51

0207 Audio and screen visual feedback to support student learning . . . . . . .61

0288 Students and mobile devices: choosing which dream. . . . . . . . . . . . . . . .70

Section 2: Other Papers

0291 Next Generation User Skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82

0164 The learning technologies of the future: technologies that learn? . . . . .97

0123 When the shoe doesn’t fit: supporting students who are challenged by online educational technologies . . . . . . . . . . . . . . . . . . . . . . . . 104

0081 PortisHEad: portfolios in successful Higher Education admissions. . 111

0213 In the eye of the storm: preliminary evidence on the use of online learning diaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

0315 To what extent could Business Process Management Suite (BPMS) contribute positively to e-learning? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

0161 A taxonomy of podcasts and its application to higher education . . . 132

0102 Can acquisition of expertise be supported by technology?. . . . . . . . . 141


Index of Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 i



In keeping with the broad nature of this year’s conference theme, the eight research papers in these proceedings range from a focus on pedagogical and curriculum re-design to the development of a research culture and the spread of evidence-informed innovation. The gradual ‘coming of age’ of the learning technology domain and the responsibility which that entails is apparent here in the shift from small scale implementation to a more considered adoption of innovation, and in the critical reflection on both past experience and future potential.

This is apparent in the paper by Dickinson where he describes a strategic, longitudinal approach to the implementation and evaluation of blended learning in a large business school core module through a focus on pedagogical alignment. It is also voiced most eloquently by Traxler in his carefully considered ‘thought piece’ on mobile learning in which he reflects on the ubiquitous nature of mobile devices and highlights the emerging paradox for institutions with regard to central control of learning technology and the increasing drive towards learner independence.

Issues of agency, control and empowerment are challenging for researchers, practitioners and senior managers alike. They are reflected here in different ways, for example by Cornelius, Gordon and Harris as they analyse the experiences of tutors and learners in an anonymous role play activity; by Hardy et al as they describe the findings of their investigation into the personal use of technology by first year students from different disciplines; and by Jones and Cross whose large scale study of students from five universities suggests that the rhetoric surrounding the so-called net generation should be viewed with a considerable degree of caution. All of these papers confirm not only the growing complexity of the learning environment, but also the central role of the learner in informing the design and future implementation of technology enhanced learning.

The move towards embedding innovation is confirmed by Russell in her account of the transformation of two undergraduate engineering courses, one as blended learning and the other fully online, as part of a curriculum renewal initiative. The findings here again point to the need for appropriate pedagogical models along with a review of resources to underpin such changes. Complementing this, Rodway-Dyer and Dunne focus on feedback as a key aspect of student achievement as they describe the outcomes of an innovative study on the impact of audio and video feedback. Their conclusions highlight a range of pedagogical and practical issues to be considered if this approach is to be widely adopted.

Finally, based on a Realistic Evaluation approach, Johnson et al illustrate an innovative use of technology to encourage collaboration and engagement amongst a range of project stakeholders. By using techniques such as mind-mapping, micro-blogging and animated modelling, the researchers encouraged collaborative analysis and joint ownership of the project outcomes, thus suggesting new options for a technology-supported research methodology.

Choice, evidence and change are reflected throughout these papers and also in the evolution of the research strand of ALT-C. Some years ago a number of senior members of the ALT community agreed that it would benefit the conference to showcase some of its best research outputs in order to raise the esteem of the conference and also to act as a demonstrator of best practice for those who were new to the domain. The criteria for acceptance of papers were set deliberately high, and papers have needed to satisfy

reviewers with backgrounds in multiple disciplines — always a difficult task.

Consequently the number of research papers submitted and accepted has never grown sufficiently to develop into a fully independent research track as had been envisaged. As a result, an issue which will be the subject of debate at the conference and for the next conference committee will be the future of the research track in the years to come.

Hugh Davis, University of Southampton

Linda Creanor, Glasgow Caledonian University

Other papers

Some other papers are also significant; however they are not full research papers for one or several of the following reasons

They are thought pieces: they suggest connections or provide a set of views without being fully evidenced.

They are reports of work in progress.

They are policy rather than research focussed.

They report on small projects where the scale of responses is insufficient as yet for full deductions.

They are directed at practitioners who wish to use a technology or the results of a project.

All of their work falls within the ALT purview and inclusion in the 2009 proceedings is by way of an experiment which, if successful, will become a feature of future proceedings of ALT-C.



Judy Hardy, Simon Bates

The School of Physics and Astronomy,

James Clerk Maxwell Building, Mayield

Road, The University of Edinburgh,

Edinburgh EH9 3JZ [email protected], [email protected]

Jeff Haywood, Denise Haywood,

Hamish Macleod

Higher and Community Education, Paterson’s Land, Holyrood Road, The University of Edinburgh, Edinburgh EH8 8AQ [email protected], [email protected], [email protected]

Jessie Paterson

The School of Divinity, New College,

Mound Place, The University of

Edinburgh, Edinburgh EH1 2LX [email protected]

Susan Rhind

The Royal (Dick) School of Veterinary

Studies, Easter Bush Veterinary Centre,

Roslin, The University of Edinburgh,

Edinburgh EH25 9RG [email protected]


The irst year at university is a time of signiicant lux for students, as they adjust to unfamiliar environments, encounter new approaches to teaching and develop fresh learning strategies on the road to becoming self-directed learners. This sense of uncertainty may be compounded by the need to interact with unfamiliar and frequently complex online systems and technologies, possibly even before arrival. Furthermore, although technology is embedded seamlessly into the personal lives of many of today’s students, recent reports have questioned the widespread assumption that young adults have the sophisticated information skills and digital literacy needed to become autonomous learners.

In this paper we present indings from a recently-completed study addressing these important issues. We investigated the utilisation of ICT and learning technologies by irst-year undergraduates from a variety of different entry routes and academic disciplines, including

Physics, Divinity and Veterinary Medicine, at the University of Edinburgh. The focus of the work was on the impact of technology on students’ transition to university and how this changed as they progressed through their irst year. The overall shape of the research was based on a student-centred approach, with students’ own views and opinions placed central to the study; and used a holistic approach in which students’ use of e-learning and technology was set within the context of their learning experiences

Section 1: Research Papers

0114 Taking the lead: learners’ experiences across the disciplines


Recent reports into the current generation of students’ expectations of, and experiences with, the use of information and communication technology

(ICT) to support their learning have highlighted a number of common themes. It is clear that university students expect to use ICT routinely as part of their academic studies (ECAR, 2008; JISC / Ipsos MORI, 2007,

2008); indeed, very many UK university students now arrive with their own laptop, both for personal and academic pursuits. The use of the internet as a communication tool is increasing, within both academic and social contexts.

There is widespread use of social networking sites for communication, especially between peers, and perhaps in preference to what is now seen as the more ‘traditional’ email (Salaway et. al., 2008, Oblinger, 2008). Yet despite their widespread social use, the potential of Web 2.0 technologies for learning appears to be less well-appreciated or understood by students.

Furthermore, despite having a high degree of proficiency with common software and web search engines, there is evidence that students may arrive at university lacking some of the vital information literacy skills needed to fully support their studies (CIBER, 2008, Oblinger, 2008). There is also evidence of a small, but significant, minority of students who do not actively engage with ICT (JISC / Ipsos MORI, 2008). The majority of published reports have centred on young (typically aged 17–22) adult learners; the

‘Net’ or ‘Google Generation’. However, it is important not to over-generalise these findings and stereotypes to make inappropriate assumptions which do not apply beyond this particular demographic group, especially with the increasing emphasis on lifelong learning and adult returnees in higher education.

It is widely accepted that good course and curriculum design should align learning and teaching activities with the intended learning outcomes, and that assessment tasks should measure the extent to which these have been met. This is the foundation of ‘constructive alignment’ (Biggs, 1996, 2003), which is based on the (constructivist) premise that students construct their own learning from the activities with which they engage. More recently, the ideas of constructive alignment have been extended to incorporate a wider range of course ‘settings’ that may influence student learning, including curriculum aims and design; learning and teaching activities; learning support; assessment and feedback; course management; and students’ background, knowledge and aspirations (McCune and Hounsell,

Hounsell, 2005; Hounsell and Hounsell, 2007). The term ‘congruence’ has been proposed to describe the interactions between these factors and the quality of student learning. This model is particularly relevant to early-years undergraduate courses, where large class sizes, a relatively low staff-student contact and diverse student cohort — and hence a wide range of student perspectives and experiences — are generally the norm. The interplay between these factors, set within the context of students’ use of both institutional and non-institutional ICT and learning technologies to support their learning, forms the backdrop for the present study.




In this paper we present outcomes from a study of a heterogeneous group of first-year undergraduate students from a variety of disciplines and different entry routes at the University of Edinburgh in 2007/08. The focus was on ‘critical moments’; specifically, the involvement and impact of ICT and learning technologies on students’ transition into university, and the changes in their use of these tools as they progressed through their first year.

The overall shape of our research was based on two underlying principles, advocated by Sharpe (2005) and Mayes (2006). Firstly, that it is important to take a learner-centred approach, whereby the students’ own views and opinions are central to the study. Secondly, that the research should adopt a holistic approach in which students’ use of technology is set within the context of their learning experiences as a whole. Within this framework, the key questions that we wished to address were:

What are students’ expectations regarding the availability and use of e-learning at university?

How do students adapt and change their approaches to e-learning during their first year?

What are the factors that influence students’ choices of e-learning strategies and their utilisation?

To what extent do students use non-institutional technologies to support their learning?

The paper is organised as follows. To set the scene, a short description of the selected disciplines is given, including general characteristics of their students, courses and methods of teaching. This is followed by an overview of the methodological approach taken in the study. Our findings are then presented, and we conclude with some reflections and implications for the future.

The context: the disciplines, their students, courses and use of technology

The three subject areas chosen for this study were Divinity, Physics and

Veterinary Medicine, representing a cross-section of the wide range of disciplines available at the University. Academic staff in all three disciplines have substantial experience in the innovative use of learning technologies to encourage and support greater self-responsibility for learning amongst students. First year courses in these disciplines have a well-established online presence, marrying online and real environments in a blended approach to learning and teaching. While to some extent, the courses in this study may be slightly atypical of the majority of degree programmes at the

University of Edinburgh, it is because of their strengths in e-learning that this was an ideal time to study the student perspective.


The first year cohort in Divinity includes a wide age profile from school leavers to mature returnees, providing a population with highly variable

ICT literacy levels and engagement with modern technology. E-learning uptake by the academic staff ranges from no engagement to those using a rich blended approach where e-learning, including the use of Web 2.0 technologies (weblogs and podcasts), is an integral part of the teaching programme.


Physics students tend to be young adults and arrive with high levels of ICT literacy and devices. Within Physics at Edinburgh, e-learning has been used to support face to face teaching on campus for almost a decade, with recent excursions into Web 2.0 territory, using podcasts and wikis.

as a whole. To capture the breadth and complexity of their experiences we used a mixed-mode approach, including a series of relective diaries recorded by learners

(in video, audio or text format) together with surveys and focus groups.

Students do not form a homogenous group, and indings in this area are inevitably complex. They have high expectations and are generally conident with technology; however, they may not always recognise technology’s potential to support and enhance learning. The term e-learning does not mean much to them; there is simply learning with strands of technology running through.

This is relected in a strong desire for face-to-face contact, with technology used to supplement and enhance this.

Students are social, with informal group learning often facilitated by technology.

They ind their comfort zones and ways of working that are personal to them, and use technology to suit their own way of learning.

Veterinary Medicine

Veterinary Medicine students are typically high achieving, highly motivated individuals. All students have access to the school Virtual Learning

Environment (VLE), embedded within which are a number of resources under the umbrella of the ‘Virtual Veterinary Practice’. These include RSS feeds, webcams and wikis, with podcasts under development. E-assessment and e-portfolios are being piloted in selected courses. Individual teachers and courses vary in the extent to which they take advantage of these resources.


The methodology used in this study has been reported previously (Hardy et al, 2008) therefore only a brief summary is given here. A mixed-methods approach was used to collect a range of quantitative and qualitative data, including:

An institution-wide survey, conducted at the start of the year;

A series of reflective diaries throughout the first year recorded by all

24 participating students, 7–9 students from each of the three target disciplines;

An end-of-year survey of all 24 participating students;

A number of focus groups with the participating students and others.

Surveys were conducted either online or using paper and analysed using

MS Excel and SPSS. Diaries were recorded in either video or audio (using webcams), or in text format. Information from the diaries and focus groups was organised and analysed using NVivo8.

Reflective diaries were recorded by the students themselves, without anyone from the research team present. Diaries were recorded at key points over the course of the academic year. They were relatively freeform but were based around a number of themes and questions, chosen to reflect significant study-related events, see Table 1.

Table 1: Diary Key Points and Themes

Semester 1

Early First impressions and transition

Mid First assessed assignment

Late Exam revision

Semester 2

Return after irst vacation

Nearing the end of irst year teaching

Return from vacation, exam revision

Findings: the student year

Students cannot be considered as a homogenous group, and our findings are inevitably complex and wide-ranging. However, the main findings and messages are summarised under a number of themes, which broadly follow the trajectory of our students’ journey over the course of their first year at university.

Learning new university technologies

On arrival at the university, students were asked to comment on their prior experiences of using technology for their studies and how they expected technology to be used at university. At this point, they were just coming to grips with university life and, as might be expected, some things came easily while others were more of a challenge. For many students, the ‘flow’ from their previous studying or work was fairly straightforward, the use of technology in their courses was mostly free of challenges and they could establish a study-life balance that suited them. In addition, universities




1 are very different organisations from school and work, and offered some pleasant surprises.

“I knew the university have a lot of resources...which were going to give me a lot of facilities. But I couldn’t believe when I saw how much computers and internet are used in each course.”

“The resources I have used so far to do my studying are what the lecturers have recommended us to buy and the WebCT...It’s a bit different to what we did at school as we never got recommended books. We just used to Google everything and use search engines to ind out as much information as we could.”

The extensive use of technology from the outset for both studying and administrative tasks was seen by some as a challenge. This was a common view, even among students with good ICT skills, suggesting that the recent movement across all universities to high dependence on ICT has not yet been perceived and internalised by intending students.

“There was an over-reliance on computers in the irst two weeks!”

The non-optional nature of the use of technology may be part of this challenge, as personal and social use is to a large extent self-regulated. In addition, the expectations of most new students may be set particularly by use of ICT in schools, which is still generally much less pervasive than in universities.

Over the course of the year, technology was seen as providing a valuable contribution to effective study — one obvious benefit being making materials easier to access — but alongside the advantages were some significant disadvantages. These were often due to limitations in the way technology was used by teaching staff, the inability of students to navigate to important information (either due to poor guidance or failure to take note of it when offered) or access restrictions.

“Lecturers did eventually put everything on WebCT following encouragement (by the students). One was very good and put a good selection of different things — podcasts, internet sources, further reading, as well as her own power points and lecture notes.

But it was very lecturer dependent.”

“I’m disappointed that [speciic e-learning material] is only available in the university computers, it would be better to be available on [online] as I do not like to study in the computer labs.”

“I’m just starting to doubt the eficacy of... any online university portal. They are just a maze to navigate through and I am not always able to ind what I should.”

Using social / personal technologies

Many students used a combination of personal and institutional technologies for fast access to sources of information. Some also had preferences for particular technologies that assisted their studying.

“I can’t revise if I’m not in front of my computer. I use WebCT a lot, or msn for asking someone who knows about what I need to know...Also, there are many blogs where you can ind many interesting things that you can’t ind in course books...”

“I found in my own revision, that the use of my own personal computer and

‘Inspiration’, the Mind Mapping software, was helpful. I did a lot of mind mapping and also going over podcasts, audio iles, MP3 iles as a revision aid and also for constructing my revision.”

For some students, paper was the ‘real’ place to be working, while others clearly favoured electronic media. However, many felt the key was to obtain a balance between different activities.

“Overuse of computers during revision will shut down creativity — I prefer to scribble down things, even if you are going to type them up neatly later, take notes quickly

while you think about them and compare to other sections rather than searching through and copy pasting, combine bits and pieces, write bullet points by hand.”

“Although I feel equally reliant on both, if it came to a deinitive choice, I just could not cope without electronic sources and materials!”

“Electronic resources should aid and not replace paper but all the time the two should work in harmony and it’s brilliant that the WebCT service has been very useful.”

There is an instrumental component in this choice because some current academic activities preclude the use of technology, and so working (and practising) in the traditional medium of paper and pens may be most pragmatic.

“If you are going to be asked to write a [exam] response using a pen and a piece of paper then that colours how you will be using the online resources on the lead up (to the assessment) and you are more likely to ‘have a go’ at what you would be doing in the exam. If the nature of the exam was different and there was an online aspect to the exam or you could use a laptop, then all of the dynamic might be entirely different..”

Technology undoubtedly made it easier for students to communicate with others for group work, both formal and informal. It is difficult to be certain to what extent this is common practice across all students, or whether we observed a particularly committed, studious or ‘tech-savvy’ cohort.

The Vets appeared to make most use of online group activities, perhaps because the opportunity had been established for them before they arrived

(via a Facebook community set up by the previous year’s students), by the evident encouragement and support of staff, or simply because they regard themselves as a budding professional cohort. However, the use of online groups was not limited to the Vets, and these tools are likely to rise in popularity with time.

“I ind it really helpful to go over material with other students, whether I’m organising that through IT discussion boards or just emailing a person I know on the course and asking their opinion on something.”

“I use a lot of MSN and forums to communicate with fellow students and teacher, where I can ask questions and discuss with my colleague.”

Assessment and feedback

The diaries showed clear signs of strong personal feelings regarding both the actual and potential roles of technology for assessment and feedback; students were perhaps more polarised about this than on any other topic.

The majority of comments on the overall composition of assessment related to the balance (or lack of balance) between what was being learned and the assessment tasks. Written exams are the most common form of assessment at Edinburgh and quite strong views were expressed about the role that is, or might be, played by technology. For some students, ‘handwriting is still king’ and is expected be around into the indefinite future due to its flexibility, speed, ease of annotation etc. For others it was already a handicap to clarity of expression, with technology helping to avoid illegible handwriting and to align with the rest of their study and life. Some could clearly see both sides of the question.

“I don’t mind handwriting exams, I think that way it’s fair on everyone, whereas if an exam was to be typed it would be unfair on people who couldn’t type as fast as others.”

“I hate handwritten exams, I write slowly and it is very messy. This cause me to spend time correcting and writing, which means less time to think. I will deinitely do faster and better at electronic exams either by MCQ or by typing.”





“Obviously, certain subjects, such as Informatics, are examined via computers by deinition, but enforcing ICT on all subjects is progress for progress’s sake and will surely have the opposite of the intended effect. Handwritten work is far more natural and personal, and does not require any translation from the brain into which keys must be pressed, which often makes the student lose their train of thought.”

We had expected to elicit suggestions for innovative uses of technology

(either actual or potential) for assessment. In fact, this was rather sparsely addressed. The few students who responded did so only in terms of the sorts of technology-based assessments they had experienced, with quizzes the most commonly mentioned. Although students were unsure about using computers for long text-based answers, many felt that technology has value for short answer and MCQ tests, especially where it could maintain a high quality test environment.

“I like the online quizzes, it is not long and I can access it all the time at home, so that I can do it when I am ready and gave the best result, well, there is nothing to blame if I get a bad result.”

Despite the fact that almost all students have laptops, used extensively for preparing essays and other coursework, no-one suggested a future in which they would take exams on their own machines, and there was almost no prediction of greater ease of use of IT applications, e.g. for drawing or writing equations, or that physical exams involving objects or evidencing skills might move into virtual spaces. This tends to suggest that their limited prior experience of online assessment was restricting their view of what was possible.

There was also no consistent view about how technology might be used to create or deliver feedback, although in general durable over ephemeral formats were preferred, and perhaps textual over aural.

“I think that verbal feedback from a tutor is the most suitable form. It also gives students the chance to ask any questions or address issues regarding the exams that concern them. A recording/ pod cast of this feedback would also be extremely helpful for future reference.”

“I would rather have text comments than verbally-recorded ones since I’m a more visual person. I don’t retain information well if I just listen to it, so I’d probably have to play recorded comments over and over. I wouldn’t want verbal comments from a tutor because I have a bad memory and I’d probably forget what the tutor said as soon as I left the room.”


Two important transitions focused around students’ changing ICT skills and competences — new skills acquired through their studies, and pre-existing skills that were not used and hence may have degraded through lack of practice. Two-thirds of our students reported some new skills development, largely related to the specific technologies used within their courses.

Divinity and Vet Medicine students showed the most new skill acquisition, mainly associated with the academic use of blogs (Divinity) or discussion forums / wikis (Vet Medicine). Physics students did not offer much evidence of skill extension except for their use of personal response systems, which are used extensively in their first year courses.

All students used the VLE, which formed the core of e-learning in their courses. Students also used the internet daily for information gathering, especially Google and Google Scholar. There were a variety of strategies for managing information gathered online. Two-thirds of students read materials online without printing, and only a quarter printed out web-pages, e-journals etc. There was a similar pattern in terms of storing materials as files on a computer, with two-thirds of students saved URLs as bookmarks. No-one reported using anything other than the web browser for saving URLs — al-

ternatives mentioned to them included bookmarking applications and online social tagging services, but neither these nor other options were suggested.

One very important skill is the assessment of the quality of online materials.

Student’s rating of their confidence in this area is shown in Table 2. As a group, albeit small in number, graduate entry Vets were the most confident, with none reporting low confidence. There was no clear difference in confidence by subject studied among the other students.

Table 2: Student conidence in assessing the quality of self-discovered online


Very conident Conident Somewhat unconident

Not at all conident

4/23 13/23 5/23 1/23

Given the extent of the use of Google to find information, this ought to be a frequently practised skill, however students reported little evidence of direct advice from lecturers other than to take care in doing so, to avoid citing

Wikipedia etc.


As the end of the academic year approached, students were asked what advice about ICT they would offer both to the university and to students following on after them. Advice to students included:

Purchase a computer before you arrive and make sure you know how to use and maintain it.

Check the compatibility with university networks and course requirements.

Practice your keyboard skills, as this will save you time later.

Log onto systems and check your email daily, as much important communication takes place online.

University is about self-reliance, so make sure you have acquired skills to support yourself.

Advice to the university related to the quality of facilities and services, which most students rated quite highly at present, rather than about innovation, and overall the advice was pragmatic and perhaps not overly challenging, consisting of desirable actions that would not come as any surprise to teaching staff or support services:

IT provision is generally good, but do more, and do it better.

Not everyone has a computer, or uses it on-campus, so availability of access is important (computer labs, student residences and wireless network coverage).

Promote the university laptop loan service and laptop check-up workshops.

Consider offering preferential laptop purchase schemes.

Good training and support is available; make sure students know how to access it.

Be more consistent in the use of the VLE across courses.

It was clear from the outset that the students in this study anticipated extensive use of technology at university. They had acquired this view from the university itself; from school, work, family and friends; from the media; and probably from a view that ‘technology is everywhere’ based on general experience. Our students also arrived with, and maintained, a very positive view of the role of technology in their education — they did not need persuading of its value. If anything, over the year they acquired a more critical view of the lack of use of technology in their university courses, and saw opportunities missed for better, deeper and more consistent use that would make their studies easier.





However, to all of them this just appeared to be how they see learning, and to some degree teaching, in the first decade of the 21st century. We did not present this as ‘e-learning’ to them in our interactions with them, and they did not use the term spontaneously. Most adopted into their daily study lives institutional e-learning technologies on offer; VLEs, online quizzes and assessments, Web 2.0 tools such as the wiki, web-based submission of assignments, email etc without overtly querying their value, appropriateness or effectiveness (as tools). They progressively became regular users of the university’s extensive digital library; most of these resources were new to them, e-journals, bibliographic databases and e-books being little available to schools or the general public. Alongside, and interwoven with, these institutional technologies, they used their own technologies — mobile phones, laptops, Facebook, blogs etc — to communicate with each other, with staff and with family / friends, and to study and learn.

No classes required our students to own laptops, but all of their courses required them to carry out a substantial proportion of their studying using the internet and ICT. Students ‘moved’ their studying from machine to machine as was most convenient to them, making heavy use of both the university computer labs and their own laptops as suited them. What they did not appear to do, was use their handheld devices (almost entirely mobile phones) as tools for learning, e.g. to access the digital library or the VLE, with the exception of podcasts on MP3 players.

Several courses required students to work in groups, but this was often not scheduled or offered in specific time-allocated spaces. Some students adopted IM as a mechanism to support this activity, using this as an alternative to email and sms, either on their own or on university equipment.

In this sense, they adopted the institutional technologies that were necessary for them to use to gain access to learning materials and tasks (e.g. the

VLE), and adapted their own technologies (e.g. mobile phones, Google) to support their learning activities. They were generally fairly tolerant of system deficiencies (this may change in subsequent years when the pressure is greater), although older students from well-resourced workplaces perhaps viewed university systems as less professionally polished than the commercial systems they had experienced.

These reflections reinforce the commentary offered in US studies of technology in higher education published by ECAR in which they observed that students value convenience and control (Kvavik et al, 2004).

Few of our students offered reflections that pushed the boundaries of what we offer at present in terms of innovation in e-learning. This may be partly due to ‘self-selection’; the university does not emphasise substantial use of technology in most of its degree programmes, but presents a traditional university education in its public-facing information. It may also arise from an intrinsic cautiousness; high-achieving students may not generally see risktaking by the university as a ‘good thing’. Similar concerns have been voiced in other research studies, which have presented scenarios for the future use of technology in university education to student focus groups, and with student reactions to e-portfolio pilots (Tosh et al, 2005). This suggests that universities that wish to expand their use of technology into more innovative and non-elective activities need to do so with suitable advance preparation of student attitudes, despite the rhetoric of a few Net Gen proponents.


The students is this study thread technology through both their social and academic lives, learning new skills from the specific application of IT and bringing their own use of technology to bear to suit their own preferences.

They do not generally have high expectations from universities in terms

of novel or innovative uses of technology, but do expect reliability, predictability, and high quality use across their courses. There is a continuing need to understand the student perspective as we move into an even more technology-rich world. The diversity of the student population, coupled with the changing nature of teaching and learning, offers both challenges and opportunities. Keeping students’ voices central to research in this area will be key to the success in meeting students’ needs and aspirations.


We are grateful to the JISC for partial funding for this work under the e-Learning pedagogy programme. Thanks are due to the Support and

Synthesis team for their advice and encouragement. We would like to thank staff from EUSA (Edinburgh University Student Association), the SFC

(Scottish Funding Council) and the University of Edinburgh for valuable advice and support. Finally, we are hugely grateful to all of the students from the University of Edinburgh who participated in the project, and thank them for all of their contributions during the project and beyond.


Biggs, J. B. (1996) Enhancing teaching through constructive alignment. Higher Education,

32: 347–364

Biggs, J. B. (2003) Teaching for Quality Learning at University. 2nd edition. Buckingham:

SRHE & Open University Press

CIBER group. (2008) Information behaviour of the researcher of the future whatwedo/programmes/resourcediscovery/googlegen.aspx [accessed Nov 3rd 2008]

Hardy, J., Haywood, D., Bates S., Paterson J., Rhind S., Macleod H. and Haywood

J. (2008) Expectations and Reality: Exploring the use of learning technologies across the disciplines. In Learners’ Experience of e-Learning: Research from the UK. Sixth International

Conference Networked Learning

PDFs/Hardy_475-483.pdf [accessed Nov 4th 2008]

Hounsell, D. and Hounsell, J. (2007) Teaching-learning environments in contemporary mass higher education in Entwistle, N.J., et al. (eds) Student Learning and University

Teaching, Psychological Aspects of Education — Current Trends. British Journal of

Educational Psychology Monograph Series II, British Psychological Society, Leicester: 91–111

JISC/Ipsos MORI. (2007) Student Expectations Study, publications/studentexpectations [accessed Nov 4th 2008].

JISC/Ipsos MORI. (2008) Great expectations of ICT: How Higher Education institutions are

measuring up, [accessed Nov

4th 2008]

Kvavik, R.B., Caruso, J.B. & Morgan, G., (2004) ECAR Study of Students and Information

Technology 2004: Convenience, Connection and Control, (Research Study Volume 5) Boulder

Colorado: ECUCAUSE Center for Applied Research, ers0405/rs/ers0405w.pdf [accessed Nov 28th 2008]

McCune, V. and Hounsell, D. (2005) The development of students’ ways of thinking and practising in three final-year biology courses. Higher Education, 49: 255–289

Mayes, T. (2006) LEX The Learner Experience of e-Learning Methodology Report, [accessed Dec 4th 2007]

Oblinger, D. G. (2008) Growing up with Google: What it means to education, in Emerging technologies Volume 3,

GrowingupwithGooglewhatit/46571 [accessed Nov 12th 2008]

Salaway, G., Caruso, J. B., Nelson, M. R. (2008) The ECAR Study of undergraduate

students and information technology 2008

TheECARStudyofUndergradua/47485 [accessed Nov 3rd 2008]

Sharpe R., Benfield G., Lessner E. and DeCicco E. (2005) Scoping Study for the Pedagogy

Strand of the JISC e-Learning Programme,

[accessed Nov 27th 2007]

Tosh, D., Penny Light, T., Fleming, K. & Haywood, J. (2005) Engagement with Electronic

Portfolios: Challenges from the Student Perspective. Canadian Journal of Learning &

Technology 31(3): Special Issue on E-Portfolios





0299 Is there a Net generation coming to university?


The Net Generation is one of several terms used to identify a distinct generational group in ways that have relevance for teaching and learning.

This new generation is identified with young people born after 1983. At the time of writing this makes the oldest Net generation members 26. Some recent reports suggest that there is a further generational shift following the Net generation beginning in the year 1993 (JISC-Ciber 2008). For our purposes we accept this ten year period between 1983 and 1993 as the boundaries of our research. It is claimed that the Net generation prefer active to passive learning, have distinct information searching patterns and a low tolerance for delays. From these characteristics there are derived particular issues that might affect teaching and learning, for example the kinds of attention spans that are exhibited by students such as working in ‘bursts’. This new generation has been entering UK higher education since 2001 and on their arrival they encounter an increasingly extensive use of e-learning. Currently it is still not clear what the characteristics of this emerging student body are. Nor is it clear what the most important influences might be on student engagement with networked and digital technologies during their studies.

The term Net generation is most commonly associated with the work of Tapscott (1998 and 2008) and he argues that these young people are different to previous cohorts because of their experience of networked and digital technologies. He writes, for example, that:

Today’s youth are different from any generation before them. They are exposed to digital technology in virtually all facets of their day-to-day existence, and it is not dificult to see that this is having a profound impact on their personalities, including their attitudes and approach to learning. Tapscott (1998 a)

We are interested in how these changes might affect learning and

Tapscott suggests that the changes in technology have some ‘inevitable’ consequences for learning. Tapscott argues that that the ultimate interactive environment is the internet itself and that education will need to move from a teacher-centered approach to learning to learner centered approaches.

“But as we make this inevitable transition we may best turn to the generation raised on and immersed in new technologies.” (Tapscott 1999 p11).

Another key source for arguments about the Net generation comes from articles written by Prensky using the term Digital Natives (Prensky 2001 and

2001a). In a similar way to Tapscott he argues that digital natives are part of a step change in attitudes and styles:

.. not just changed incrementally from those of the past, nor simply changed their slang, clothes, body adornments, or styles, as has happened between generations previously. A really big discontinuity has taken place. One might even call it a

“singularity” — an event which changes things so fundamentally that there is absolutely no going back. (Prensky 2001 p 1)

Presnky’s comments were made about the entire generation in schools and colleges and they are not limited to students in universities. Prensky suggests the new generation thinks differently and he goes on to make the claim that the brains of the new generation are different to previous generations

(Prensky 2001a). A similar argument has recently been advanced, without


Chris Jones, Simon Cross

The Institute for Educational Technology,

The Open University, Milton Keynes

MK7 6AA [email protected], [email protected]


This paper reports the irst phase of an ESRC funded research project to investigate irst year students’ use of technology in relation to the idea of young people born after 1983 forming a distinct age cohort described variously as the Net generation or Digital

Natives. The research took place in ive

English universities in the spring of

2008. The research found a far more complex picture than that suggested by the rhetoric with student use of new technologies varying between different universities and courses. Some of the more discussed new technologies such as blogs, wikis and virtual worlds were shown to be less used by students than might have been expected. The Net generation appears if anything to be a collection of minorities with a small number of technophobic students and larger numbers of others making use of new technologies but in ways that did not fully correspond with many of the expectations built into the Net generation and Digital Natives theses.

substantiation, by Baroness Greenfield the Director of the Royal Institution in the United Kingdom. She told the House of Lords that children’s experiences on social networking sites:

“are devoid of cohesive narrative and long-term signiicance. As a consequence, the mid-21st century mind might almost be infantilised, characterised by short attention spans, sensationalism, inability to empathise and a shaky sense of identity”.

24th of February 2009 ( social-networking-site-changing-childrens-brains )

Lady Greenfield echoed Prensky in suggesting that exposure to new technologies and web services was likely to fundamentally change children’s and young adult’s brains. Prensky argued that in education there is a disconnect between the ‘digital native’ students and the ‘digital immigrant’ staff who retain the ‘accent’ of the pre-digital era even when they become socialized into a digital environment. This suggests that being a digital native or a digital immigrant is not a learned skill and in Prensky’s view it is a fixed product of early development. In a recent article Prensky has suggested that the distinction between digital natives and immigrants will become less important and developed a new set of distinctions around the term digital wisdom (Prensky 2009). However it is clear that both Prensky and Tapscott still suggest that the changes in technology lead to determinate outcomes.

Indeed Tapscott advances the technologically determinist argument that changes to pedagogy flow in an ‘inevitable’ way from changes in technology.

Diana Oblinger of EduCause uses the term Millenials for the generation born after 1982 and her work is supported by large scale annual surveys of students in the USA. However her argument continues to describe a whole generation and she claims to have identified a trend towards an internet age mindset. Oblinger also identifies what she describes as a disconnection between the new Millenial students and the institutions that they are enrolled in. Unlike Prensky, Oblinger and Oblinger (2005) suggest exposure to technology might be more important than age group, allowing older students to develop different approaches. “Although these trends are described in generational terms, age may be less important than exposure to technology.” (2.9).

Although some empirical research agrees that: “Students are ‘digital natives’ — having grown up with ICT and expect to use their own equipment at university.”(JISC 2008 p7) most recent empirical studies are less clear about the nature of new young learners. In the UK Margaryan and Littlejohn (2009) have reported that students use a limited range of established technologies for both learning and for recreational and social use. They also found that there were low levels of use and familiarity with virtual worlds and personal web publishing. In addition they reported that students’ attitudes to learning appeared to conform to fairly traditional pedagogies. Kennedy, Judd,

Churchward, Kay & Krause’s (2009) found that first year Australian students use of new technologies displayed considerable diversity in both patterns of access to technology and the ways students used these technologies. They argued that first year students possessed a core set of technology based skills but, that outside of these core technologies, students exhibited a range and diversity of skills (Kennedy et al. 2009 p117). Selwyn’s study of UK students (2008) agreed that the new generation of learners were no more homogenous than were previous generations and pointed to the continued existence of gender differences. However ECAR studies of US students report that in terms of skills with the core applications used for studying that there were few gender differences (Salaway, Caruso & Nelson 2008 p11). Work in

South Africa reported that, whilst almost all students were exposed to ICTs, there was a low use of these technologies for teaching and learning (Brown and Czerniewicz 2008). Overall there is growing empirical evidence that suggests caution in defining a new generation of young people in relation to their lifelong exposure to digital and networked technologies.




The survey

This research is the first phase of a two year study funded by the Economic and Social Science Research Council. The overall aim of the research is to provide an empirically based understanding of the Net generation as they first engage with e-learning in UK tertiary education. The research uses a mixed method approach including interviews and the Day Experience

Method, a form of cultural probe, to supplement the survey work (Riddle and

Arnold 2007). Five universities were selected to represent the main ‘types’ of university in England. Fourteen courses were surveyed representing a range of subject and disciplinary areas in both pure and applied branches of learning (see Table 1 for a more detailed description). A survey was chosen as the main intervention in the first phase of the research to provide a single snap-shot of student use of technology and to provide a background for further research including a range of methods in the second phase of research which has taken place during the 2008/9 academic year.

A questionnaire exploring first-year experiences of e-learning was developed by the research team and tested with a small number of students for timing and comprehension. Survey instruments developed by researchers in the

USA and Australia were considered and informed the design of some questions but they were not considered to be directly transferable to a

UK context. The instrument sought to collect baseline information about some of the key aspects of the students’ use of technology in their studies.

It was mainly composed of closed questions but included a number of open text responses. The instrument contained four sections: demographic characteristics of the respondents, access to technology, use of technology in university studies in general and finally course-specific uses of technology.

Table 1: University types (Jones and Ramanau 2009a)

University A University B


Founded 19th


Founded 1970s

(Polytechnic) university status in



Large urban metropolitan

Large urban metropolitan

Course units English



Information and


Veterinary science

University C University D University E

Founded 1970s Founded 1970s

Founded 21st

Century from university college

Large scale distance

Mid size campus outside small city

Mid size with multisite campuses in small towns

Modern Languages Journalism Science

Health and Social


Computing Psychology

The Arts

Accounting and


Social Work

A maximum of 1809 students nearing the end of their first year study at university were available to participate in the survey. A total of 596 first-year students completed the survey yielding a response rate of approximately

33%. A further 62 responses had to be excluded because students had either failed to finish the survey form or had not signed the consent sheet.

This purposive sample provides a robust basis for presenting a descriptive account of first year students use of technology.

The specific method of delivery used for each course was determined on a case by case basis. Students were invited to participate during a short presentation by a member of the project team or university teaching staff and, in the case of distance students, an email and letter were sent in place of the introductory presentation. Following this initial contact, follow-up emails were sent to all students on each course. Some verbal reminders were also given by teaching staff in subsequent lectures. Three versions of the survey were produced: an online version accessible via the internet; a


paper version for distribution and collection within a teaching session; and, for distance learners, a paper version that could be mailed to their home and returned in a prepaid envelope. Of the fifteen courses surveyed: nine courses used only online surveys; five offered a combination of online and paper; and one used paper only.


The demographic profile of our sample is shown in Table 2. Two features are worth noting in our sample. The gender distribution is skewed towards female students and the sample from University C (the distance university), is disproportionately over 25. Both these proportions are beyond what might have been expected as the course recruitments were not as skewed as our respondents. The Net generation age group in the distance university shows a skew that exaggerates the current recruitment patterns at this university as under 25s could be expected to be approximately 20% of the intake for first level courses.

Table 2: Key demographic characteristics (% of the total) (Jones and Ramanau 2009b)

University A University B University C University D University E

Male 22.3






UK Students

International Students

18–25 years of age





















Older than 25

Full-time student

Part-time student

Total number































Computer and network access

Just over three quarters (77.4%) of the respondents had access to a laptop and over a third (38.1% where n=554) owned a desktop computer. Only two

(0.4%) had no access to a desktop computer and eight (1.4%) no access to a laptop. Over half (55.4%) used a desktop computer in a public place but this suggests that a large minority of students only make use of private access to computing which could have implications for university provision. Around two thirds (70.1%) of those asked felt that their access to computers was sufficient to meet their computing needs whilst a further 26.4% said that it mostly met their needs. Only 3.3% of students said it ‘partially’ meet their needs and only one student said that their access did not meet their needs at all. A supplementary question (Q2.3) asked this minority to explain why their needs were only partially met or not met at all and 14 gave as their reason

‘cannot afford the necessary software / hardware’, 11 that ‘the computer is too old’ and 11 that ‘the place of access is inconvenient. Other open text comments included one student citing ‘excessive port blocking’ by university systems and another complaining about limited access in halls of residence.

Over half of the respondents had a broadband connection (55.6%) and

39.5% had access via a broadband wireless hub /router. Around an eighth

(13.4%) of students reported that they had a wireless mobile connection. We were surprised by the number claiming to have mobile broadband access but this was supported by open text answers. For example one student said: ‘[I have] no land line where I am: [so] using mobile phone as modem (GPRS).

Speed of 460.8 Kbps appropriate for text, but way too slow for media



1 content’ and another student said ‘mobile computing is becoming a priority’.

In another question we asked students where they accessed the internet and included the option of ‘anywhere, mobile internet’. Whilst this isn’t directly comparable with a question specifically identifying mobile broadband access, it gives further confirmation of the rough size of the minority of students because 11.1% of respondents reported using mobile internet. This latter figure is similar to the proportion of students claiming to have a wireless mobile connection (13.4%) but cross-tabulation reveals that 9% of students responded positively to both question and this would suggest that caution needs to be exercised in interpreting these results.

Student access to other devices was in some ways predictable. Almost all students owned a mobile phone (97.8%) and these phones generally came with a camera (91.9%); music player (77.25); and internet access (75.7%).

Less common were wifi (14.2%) and plain phones with none of these features (6.4%). This still meant that 35 of the students only had a basic phone and 6 of our respondents reported that they did not have any access at all to a mobile phone. Memory sticks were the second most commonly reported device (87.9%) but once again there were a small minority who did not own or have access to one (7.9%). An MP3 device or other digital music player was also a commonly owned device with 82.4% reporting ownership.

Other devices were less common such as a games console 38.4%, although this was one case that showed a significant amount of shared use (21.5%), and a large minority who reported no access at all (39.5%). A further question about the kinds of games players that were owned, included hand held as well as console players. Around half of students (50.2%) reported that they owned a games player of one kind or another and most who owned a handheld games player also owned a console. Personal Digital Assistants

(PDA) were used by very few with PDAs with wifi owned by 5.7% and PDAs without wifi by 4.6%. Fifteen individuals reported both owning a PDA with and without wifi and we cannot be sure if this implied two separate devices or was a double reporting of the same device. When we asked which of these devices they would miss the most if they did not have access to it

83.2% chose their mobile phone. In open text answers it appears that this was because the phones had several functions (i.e. the phone function was not the only, or indeed the primary, benefit).

Student use of technology

Students reported spending a considerable amount of time working on computers and using the internet.

The majority of students use their

Table 3: Hours spent using a computer and the internet computers for up to four hours a day and the internet for up to three

Number of hours


On a computer hours. However there are a small number of outliers that use the computer for over 10 hours a day and a handful of extreme users who access the internet for over 10 hours a day.

0 and over

1 and over

2 and over

3 and over





4 and over 15.8%

Respondents were asked how important internet access was for a variety of activities. The activities rated most important were accessing materials and communicating rather than downloading and uploading materials. This suggests that the idea that the Net generation are more likely to be inclined to

5 and over

6 and over

7 and over

8 and over

9 and over

10 and over







On the internet













Table 4: The importance of internet activities

Accessing course information

Accessing study material

Download / stream written material

Download / stream audio material

Download / stream TV and video

Uploading materials

(audio / images / video)

Keeping in touch with other students and friends








Not very important













17.6% participation might be somewhat exaggerated. However there appeared at first sight to be a minority who reported that uploading and downloading audio and video was important.

On inspecting the data it is not clear whether this apparent minority really does cohere and further analysis is required to establish whether downloading and uploading audio and video materials are an indicator of a coherent minority of students.

81.5% 13.0% 5.6% twelve types of technology. The horizontal bar charts below show the cumulative percentage of student responses:

We also asked respondents about the frequency of use of

Students were asked specifically about their use of particular technologies that have received significant attention in recent educational technology literature, blogs, wikis and virtual worlds. Perhaps surprisingly there is no evidence of a significant uptake of any of these technologies amongst the first year students and of virtual worlds in particular. These figures are consistent with those shown in Figure 1 when students were asked if they used a blog and a majority of students report never having used one. The percentages for wiki use are not directly comparable as those reported in

Figure 1 show the use of Wikis including Wikipedia and those in Table 5 ask if the students had contributed to a wiki. In this case the contrast is sharp with only a small minority of students having contributed to a wiki whilst a majority of students use wikis including Wikipedia at least weekly.

We asked students to report on how confident they felt (defined in relation to skills) using a set of common tools and technologies. Over 80% of students reported slight confidence and basic skills or better in using presentation software (87.5%), online library resources (86.5%) spreadsheets (84.9%), and in computer maintenance (82.3%). However, over a third reported no confidence or minimal skills (not known or not confident) using Virtual

Learning Environments (VLEs) (37.7%), writing and commenting on blogs and wikis (40.6%), and graphics software (36.4%); with almost two thirds

(60.3%) reporting no confidence or minimal confidence in video / audio editing software (Figure 2).

Table 5: The use of new technology

forms (Blogs, wikis and virtual worlds)









Contributed to a blog

Contributed to a wiki

Used a virtual world

Kennedy et al (2008) have made a distinction between what they called technologies for life and technologies for learning. We explored this distinction and asked two sets of Likert scale questions about the importance students placed on a variety of technologies firstly for study purposes and secondly for their social life and leisure (Figure 3). There are some interesting features to the responses. Firstly there is a small minority of students who never use email for study purposes and a similar small minority who never use email for social purposes. When cross-tabulated only 3 individuals never use email for either study purposes or social life and leisure but a further 42 cases then report low use of either email for study purposes (21 at 0–1 hour a day) or for social and leisure purposes (21 at 0–1 hour a day). It suggests that there are a minority of students for whom email is not heavily used and this might have important implications for routine methods of communication by universities with first year students.

In terms of the frequency of use of technologies in relation to their courses, around two thirds of students use computers, the internet and web and




Figure 1: Frequency of use of 12 technologies

60% 0% 20% 40%

a. Read and/or send email

80% 100%

b. Use mobile phone messaging (SMS/MMS) c. Chat using instant messaging (MSM, etc) d. Participate in online social networks (*) e. Use blogs f. Use wikis (including Wikipedia) g. Shop online h. Download/stream music i. Download/stream TV/video (iplayer, YouTube, 4OD, etc) j. Upload audio/images/video (*) k. Play games on a PC, console or handheld device l. Gamble online

the university intranet / portal at least once a day. Around two fifths use the

University VLE as least once a day, whilst over half never use a networked device as a course requirement. A fifth of students use the network to access library resources at least once a day and a majority of students access online library resources at least weekly. There are some interesting aspects of the use of technologies connected to course requirements. We asked, for example about course requirements to access online library resources and over 60% reported that there was such a requirement (61.7%). The results reveal that even, within specific courses, opinions do vary. On one course

Never (left most bar)




More than once a day

(right most bar)


Not known or not conident/no or minimal skills (left most bar)

Slightly conident/basic skills

Reasonably conident/ competent skills

Pretty conident/good skills

Very conident/ excellent skills (right most bar)

Figure 2: Conidence and skills in using tools and software

0% 20% 40% 60% 80% 100%

Spreadsheet software

Presentation software

Graphics software

Video/audio software

Online library resource

Computer maintenance

Computer security

Writing and commenting on blogs and wikis

Virtual Learning Environments (VLEs)


73.7% (14 students) reported that access to online library resources was not a requirement whilst 26.3% reported that it was; and on another course,

65.6% reported that it was (21) whilst another 34.4% reported that it was not.

Even on those courses with greatest agreement, not all student responses agree and these findings illustrate the variation within a single age cohort and the way course requirements and instructions only have an indirect effect on student understandings of them.

We also asked about thirteen technological tools in relation to what the students had used for study, and in relation to what the student thought they were required to use. Table 6 shows that in all cases the use of tools exceeded the perceived requirement to use them. However, for some technologies the difference is greater than others. For example, instant messaging, online quizzes, wikis and social networking sites are all used to a much greater extent than they are required to be. Email and the course website are almost universally used, although only three quarters of students believed that use was a course requirement. Wikis (including Wikipedia) were used almost as much as e-journals / e-books and the VLE. Only one in ten students used eportfolios and simulations while lower numbers used blogs and only a handful used virtual worlds.



Figure 3: Study purposes (left) social life and leisure (right)

0% 20% 40% 60% 80% 100% 0% a. Email a. Email

20% 40% 60% 80% 100% b. Text messaging c. Instant messaging b. Text messaging c. Instant messaging d. Social networking sites e. Chat rooms f. Virtual worlds (Second Life, etc.) d. Social networking sites e. Chat rooms f. Virtual worlds (Second Life, etc.) g. Internet telephony and conferencing (*) g. Internet telephony and conferencing (*)

* e.g. Skype, Vonage, Tesco, etc.

Never (left most bar)

0-1 hours

1-3 hours

3-5 hours

More than 5 hours (right most bar)

Table 6: Use and requirement to use on course


Course Web site


E-journal / e-books

Instant messaging

Online quizzes or tests


Simulations / computer models


Wikis (including Wikipedia)

Social networking sites

Virtual worlds

Discipline speciic technology / software

Other (please specify)
















Required to use















* e.g. Skype, Vonage, Tesco, etc.


Table 7: Printing activities

I always cut and paste irst 14.7%

I mostly cut and paste irst 26.9%

I mostly print the full document


I always print the full document

I have no preference



The figures in Table 6 support the data displayed in Figure 3 and the distinction suggested by Kennedy et al. between technologies for life (used for social life and leisure) and technologies for learning (technologies used for study purposes).

Finally we report a set of questions that explored the use of written materials online. Over two thirds (68.3%) reported that they save / download materials when accessing them online. The majority (56%) report mostly or always reading on screen, although around a third of students still report mostly or always printing out downloaded written materials. Asked about their practices when printing out materials students reported the results shown in table 7.

When asked about their writing activity, around a quarter of students report writing course notes on a computer but this proportion rose to around three quarters when drafting essays or coursework. Almost all students reported writing their final essays or coursework on a computer.

Discussion and conclusion

Our research confirms the complex picture found amongst students in other contexts (Kennedy et al 2008, Salaway et al. 2008, Margaryan and Littlejohn

2008 and Selwyn 2008). The findings also suggest that the kind of academic moral panic identified by Bennett et al (2008) and recently exhibited in the debate about Facebook and the brain sparked by comments by Baroness

Greenfield is over exaggerated. The first year students surveyed for this research are a diverse group and it does not seem that they are marked by their exposure to digital technologies from an early age in ways that make them a single and coherent group. This conclusion supports earlier work by

Selwyn (2008) who identified gender and subject and disciplinary differences amongst students in the Net generation. This should caution educational policy makers in universities and governments against adopting technological determinist arguments that suggest that universities simply have to adapt to a changing student population who are described as a single group with definite and known characteristics. This research, whilst exploratory, suggests that the picture is complex and our understanding of the characteristics of young students entering their first year is still very limited.

We would point to two results that support this argument. Firstly the limited use by students that is revealed in the survey of blogs, wikis (other than

Wikipedia) and virtual worlds. Secondly we would point to the existence of significant minorities, for example those who do not use either email or have access to mobile phones. It is often assumed that these two technologies are now universal and that all students have access to them and the desire to use them. Our survey suggests that this is not true for a small but significant group of students. It should not be assumed from these comments that our results suggest that there is little change taking place. We were genuinely surprised by the apparently rapid uptake of mobile broadband by students who are often in university residences with good broadband access and little apparent incentive to pay for such access. We are also intrigued by what we think is a growing seamless integration of new technologies into everyday life. It would seem to us that technology is not added to a life that exists without technology but rather student life seems to be infused with a variety of more or less universal technologies. We are also interested in the apparent changes to students’ reading and writing practices and the large minority that now make use of audio and video materials.

These areas of research will now be taken forward in a second phase of research that will explore the issues in more detail, making use of interviews, cultural probes and two further surveys of students at the start and end of their first year studies. The second phase of research includes two



1 further linked surveys of students entering university in the academic year

2008/09. These linked surveys at the start and end of the year will allow for examination of any longitudinal changes during the first year of exposure to university provision of digital and networked technologies. The surveys also provide the basis for recruitment of smaller sub-samples of students for interview and participation in a cultural probe intervention. This second phase should allow the research team to build a richer description of student uses of technology during their first year at university.


The research reported in this paper was funded by the UK Economic and

Social Science Research Council (Grant RES-062-23-0971). We would also like to acknowledge the assistance of Ruslan Ramanau and our collaborators at the five participating universities, in particular Susan Armitage, Martin

Jenkins, Sheila French, Ann Qualter and Tunde Varga-Atkins.


Bennett, S., Maton, K. and Kervin, L. (2008) The ‘digital natives’ debate: A critical review of

the evidence. British Journal of Educational Technology. Vol 39 No 5 2008 775–786

Brown, C. and Czerniewicz, L. (2008) in student use of ICTs in higher education in South

Africa. In: van Brakel, P.A. (Ed.), Proceedings of the 10th annual conference of world wide web

applications, Cape Peninsula University of Technology, Cape Town, South Africa.

JISC (Joint Information Systems Committee) (2008) Great Expectations of ICT: How Higher

Education Institutions are Measuring Up (JISC–Ipsos Mori Full Report), London, JISC.

JISC-Ciber (2008) Information Behaviour of the Researcher of the Future. Ciber briefing report. [accessed March 2nd 2009]

Jones, C. and Ramanau, R. (2009a) Collaboration and the Net generation: The changing

characteristics of first year university students. Proceedings of the 8th International Conference on Computer Supported Collaborative Learning: CSCL2009: CSCL Practices.

Jones, C. and Ramanau, R. (2009b) The Net Generation enters university: What are the

implications for Technology Enhanced Learning? M-2009: Proceedings of the 23rd ICDE World

Conference on Open Learning and Distance Education including the 2009 EADTU Annual

Conference, 7–10 June 2009, Maastricht NL.

Kennedy, G., Judd, T.S., Churchward, A., Gray, K. and Krause, K. (2008) First year

students’ experiences with technology: Are they really digital natives? ‘Questioning the net generation: A collaborative project in Australian higher education’. Australasian Journal of

Educational Technology, 24(1), 108–122.

[accessed August 8th 2008]

Margaryan, A. and Littlejohn, A. (2009) Are digital natives a myth or reality?: Students’

use of technologies for learning.


[accessed May 20th 2009]

Oblinger, D.G. and Oblinger, J.L. (2005) Educating the net generation, An Educause e-book publication. [accessed May 20th 2009]

Prensky, M. (2001) Digital natives, digital immigrants. On the Horizon. NCB University Press,

Vol 9(5).

Prensky, M. (2001a) Digital Natives, Digital Immigrants Part II: Do they really think differently?

On the Horizon. NCB University Press, Vol 9 (6).

Prensky, M. (2009) H. sapiens digital: From digital immigrants and digital natives to digital

wisdom. Innovate 5 (3). [accessed

February 1st 2009]

Riddle, M.D. and Arnold, M.V. (2007) The Day Experience Method: A Resource Kit. dtl. [accessed November 5th 2008]

Selwyn, N. (2008) An investigation of differences in undergraduates’ academic use of the internet.

Active Learning in Higher Education 2008; 9; 11 pp11–22.

Tapscott, D. (2008) Grown up digital: How the Net generation is changing your world. New

York: McGraw-Hill.

Tapscott, D. (1998) Growing up digital: the rise of the Net generation. New York: McGraw-Hill.

Tapscott, D. (1998 a) The Net Generation and the School. Milken Family Foundation. www. ?_function=detail&Content_uid1=109 [accessed Oct 7th 2008]

Tapscott, D. (1999) Educating the Net generation. Educational Leadership, 56, 5, 6–11



Carol Russell

Faculty of Engineering, UNSW

Kensington Campus, Sydney, NSW 2052,

Australia [email protected]


The University of New South Wales’

(UNSW’s) Faculty of Engineering is introducing a new process for designing and developing blended and fully online (distance) courses, as part of action research to support curriculum renewal. The process, referred to as

CREWED (Curriculum Renewal and

E-learning Workloads: Embedding in

Disciplines), is being used to develop key courses that add lexibility to student progression pathways. By integrating the design of learning activities with the planning and organization of teaching and support work, CREWED addresses some of the known barriers to embedding innovative use of learning technologies within disciplines. CREWED incorporates key features of two course development models from the UK, one emphasising team building and the other emphasising pedagogical planning. It has been piloted in priority curriculum development projects, to ensure that the disciplinary organizational context is supportive. One pilot is a fully online distance version of a postgraduate course. The other is a blended version of an undergraduate course. Both are core (required) courses in accredited professional engineering degree programs and were previously available only in face-to-face mode. The UNSW pilots have conirmed the importance of articulating clear pedagogical models, and of planning ahead for the resources required to put these models into practice, as part of departmental capacity building, especially where teaching has primarily been treated as an individual classroom-based activity that competes with disciplinary research for academic staff time and resources.

0178 All hands on deck: CREWED for technology-enabled learning


UNSW’s Faculty of Engineering is developing a new process for designing and developing blended and fully online (distance) courses, to support curriculum renewal in the discipline. The course development process,

CREWED (Curriculum Renewal and E-learning Workloads: Embedding in Disciplines), is being used to develop key courses that add flexibility to student progression pathways. The context is one where there are clear drivers for curriculum development, and where learning technologies can enable this development, by increasing flexibility and supporting new types of learning activity.

The faculty is aiming to expand distance and blended study options to provide more flexible pathways for students to achieve an accredited professional degree in engineering. Redesigning learning activities for new learning media and environments creates an opportunity for pedagogical review, as part of curriculum renewal. However, UNSW is a traditional campus-based university which, like other similar universities, still largely relies on traditional classroom teaching methods. CREWED aims to overcome some of the known contextual barriers to the adoption of new learning technologies in traditional campus universities,

This paper describes two pilot projects which form the initial stages of longer-term action research aiming to build knowledge of how learning technologies can enhance curriculum development within a discipline. Two courses have already been designed, developed, run and evaluated using a new team-based process. In both cases, there has been explicit attention to team process and managing the staff workload required for designing courses with technology-enabled learning activities.

The research is building on prior work in the UK, to develop a practical approach that will enable embedding of new learning technologies in mainstream campus teaching practices and systems. The pilots in UNSW identified some remaining barriers to implementing this team-based approach, and the outcomes suggest where further work is needed to address these.

The pilots aimed to:

■ establish, evaluate and embed an efficient team-based process for developing innovative learning activities, incorporating use of learning technology within curriculum development for Engineering degree programs in the University of New South Wales (UNSW)

■ build a knowledge base of the staff time, skills and resources needed for creating and running technology-enabled learning activities

■ demonstrate how use of digital learning technologies can be embedded in academic departmental organization and in discipline-specific educational designs.

Analysis of the two pilot projects, and their outcomes, also contributes more broadly to understanding of how to build capacity for innovative use of learning technologies as an integral part of curriculum development within a higher education discipline context.




Background and rationale

Contextual challenges and opportunities in engineering education

Several Australian universities are seeking to strengthen the building of engineering graduate attributes through design project work, and to incorporate international initiatives such as the CDIO (Conceive-Design-

Implement-Operate) framework within the discipline. There are also moves to introduce a 3+2 degree structure in university engineering programs. Both of these are drivers for curriculum renewal. Online learning resources and tools are already proving essential for managing large undergraduate classes in which team-based design projects are replacing some traditional lecture and lab activities.

UNSW’s Faculty of Engineering is Australia’s largest, with around 4200 equivalent full-time students. The faculty has identified some specific curriculum development goals, for which new learning technologies offer enabling features — in particular providing more flexibility of routes through to accredited 3 year, 4 year and 5 year degrees. However, there are barriers to the introduction of innovative teaching using educational technologies in UNSW.

In the faculty, teaching is a large-scale organised activity, run by academics who also have research responsibilities. About 90% of academic staff members are ‘research active’ according to the University’s definition.

Teaching work is measured in terms of student contact hours and course or program coordination responsibilities. There is no built-in allowance for developing new types of learning activity. Such work has to be treated as a special project, and there is little knowledge of how to plan and allocate staff time for it.

Removing systemic barriers to effective use of learning technology in campus universities

Laurillard (2002, p227) maintains that collaborative development is crucial for developing effective use of learning technologies, because of the range of skills needed. She also observes that staff time and resources need planning at institutional and departmental level, but that academic staff time is rarely costed in relation to specific areas of their work. Academics in traditional universities spend a significant proportion of their time presenting through lectures and marking and spend relatively little time designing.

For many academic staff, the introduction of new technology has been “a nightmare of overwork and lack of support” (Laurillard, 2002, p229).

Previous projects on embedding e-learning design in university teaching have developed a team design process (Gilly Salmon, Jones, & Armellini,

2008) or have focused on pedagogical planning tools for academics (Diego, et al., 2008; Laurillard, 2008), but have not integrated these with the planning of staff workloads within academic departments and disciplines.

Especially when under pressure, individuals adopt behavioural strategies that minimize enquiry, based on ‘theory in use’ learned through socialization rather than on explicit espoused theory based upon evidence (Argyris, 1999).

Theories are also embodied in organisational systems such as academic workload models. It is mainly an individual academic responsibility to develop new learning resources and there is usually limited or no support available for developing new digital media (Uys, Buchan, & Ward., 2006).

There is a lack of organised and articulated knowledge of how to plan and allocate university staff time to developing use of new technologies to best advantage within disciplinary departments.

The aim of this research is to generate new workload models for new educational designs and new technologies as part of a systemic approach


within academic departments, by including the planning of staff workloads as an integral part of a team-based design and development process. The underlying conceptual framework is that a discipline-based university department is a complex adaptive system (Russell, 2009). Figure 1 illustrates how, in disciplinary learning and teaching, systemic adaptation to contextual change involves interdependence among forms of teaching and learning activity, material resources used and organising processes. Attempts to change pedagogy without also addressing the other complementary changes will result in a homeostatic response that minimizes, or even cancels out completely, the impact of the change (Kezar & Eckel, 2002).

The development and piloting of a practical process that can take account of and document the interdependencies illustrated in Figure 1, in the context of UNSW Faculty of Engineering curriculum development priorities, also contributes to addressing a broader research question:

In the context of disciplinary curriculum development, how can the use of new learning technologies be integrated with development of new forms of learning activity and changes to departmental teaching processes, so that each of these helps rather than hinders the others?

Research methods

Action research

The research is a practical intervention in a complex university learning and teaching system, seeking to identify and adjust the key interdependencies illustrated in Figure 1. The aim is to help the disciplinary system to adapt, with the introduction of new learning technologies forming part of the adaptation. Several writers advocate action research approaches for such interventions in complex organizational systems, higher education curriculum development and online learning (Mitleton-Kelly, 2003; G.

Salmon, 2001; Trevitt, 2005; Zuber-Skerritt, 1992). This paper reports on the early cycles of an ongoing action research project (Figure 2).

The CREWED process

Figure 3 shows a flowchart of the course development process as piloted, called CREWED (Curriculum Renewal and E-learning workload: Embedding in Disciplines). The CREWED process is based on the Carpe Diem model

(Gilly Salmon, et al., 2008) for building team-based capability in e-learning design. The main benefit of this process is that it offers a clear result to busy academics for a short and contained investment of their time. Another UK project, the London Pedagogy Planner (Laurillard, 2008) provided ideas on how learning designs can be made explicit as part of a planning process.

To this was added explicit planning and evaluation of the workloads for developing and running the course, in relation to the pedagogical models being used. There was also evaluation of the effectiveness of the design, as implemented, for student learning.

The two courses chosen for piloting the team-based design and development process represent different aspects of the Faculty’s curriculum development priorities:

■ a distance version of a core introductory course in all postgraduate mining programs, piloted in Semester 2 of 2008, previously offered only as an intensive campus-based course in Semester 1 each year

■ a blended version of a core 1st year course for undergraduate chemical engineering programs, offered in blended mode in the 2008–9 summer term, previously offered only as a standard classroom Semester 2 each year.

The pilots therefore engaged with all parts of the disciplinary learning




Figure 1: Disciplinary learning and teaching framed as a complex adaptive system

e.g. flexibility for students, improving progression rates, programme accreditation


Discipline curriculum priorities e.g. staff time, departmental support e.g. e-learning development support


Pedagogical design, innovative learning activities using new tools . . rather than discipline custom and practice


Learning spaces, teaching resources, tools, facilities, time and workload

system illustrated in Figure 1, including and especially curriculum development priorities.

Implementing CREWED

Figure 3 shows the intended version of the process. In both pilots it proved impossible to arrange for the academic staff involved to be available for two consecutive days. So instead of a single intensive 2-day session we

(the course development teams) attended a 1-day workshop and follow-up half-day workshops, with development tasks scheduled between workshops, to maintain momentum. Like the original Carpe Diem model, everyone who would have a role in designing, delivering and supporting the student learning activities took part. Each course development team included:

■ a facilitator;

2–3 core academic course team members;

■ an educational technologist;

■ an educational developer;

■ the outreach librarian for the Faculty (to provide 3rd party resources, copyright clearance and information literacy support).

The support staff worked hands-on with the academics on design and development activities. The academics were asked to ensure that all the basic course learning resources were available in digital form to use in the first workshop. Both pilots also involved students as ‘reality checkers’ to work through some of the online activities developed, and give feedback. For the distance course this was done between workshops, and for the blended course the students came in at the end of a workshop and tried out the activities with the team present.

Another emergent change in the process from the Carpe Diem model was in how the course design was captured and visualised. In the first pilot


Teamwork and planning support . . . rather than individual academic responsibility


Figure 2: Action research cycles

Engineered curriculum renewal

Carpe Diem LPP

Student need for flexible study options

Constaints on staff time for redeveloping curriculum

Evaluation of inputs and outcomes

Pilot distance PG course in mining engineering

Pilot blended UG course in chemical engineering

CREWED process for online and blended courses in UNSW


Sustainable course development process for UNSW Engineering

Evaluation of inputs and outcomes

Further UNSW engineering courses

Courses in other disciplines

Engineering courses in other universities

Sustainable course development process to support curriculum change in different contexts



1 workshop, for the Mining course, we started to use a storyboarding process to capture the design, and found that this did not work well as a method for making the pedagogy specific, nor for planning student and staff workloads.

The LPP project developed open source software for representing and planning implementation of the learning activities in a course or subject module. Although the software defaults to Laurillard’s conversational learning model, it has the potential for use with other pedagogical models.

(Diego, et al., 2008; Laurillard, 2008). As the facilitator, I drew on this idea to suggest using a spreadsheet representation of the course timeline, including all aspects of student and staff activity, and resources and tools used, on one sheet with estimates of course totals of student and staff workloads for running the activities clearly visible.

The spreadsheet representation brings together and visualizes the whole course design and each team member’s role in supporting the learning activities. It also maps how each learning activity contributes to the course learning outcomes and their assessment, and to disciplinary graduate attributes. In the first pilot we were also able to map the course timeline onto the five-step model of levels of engagement in online learning (G. Salmon,


The planned staff activity could then be compared with the actual activity to plan for the same course and for other similar courses, in future.

Evaluation by students involved both the observation of student participation in the online activities while the pilots were running, and a short Survey Monkey questionnaire after the students had completed the final assessment.

Outcomes and what was learnt from them

Table 1 summarises what was learnt from both pilot projects. The two projects had contrasting outcomes, which highlighted the need to take into account the different academic contexts of the students and the staff.

However there were some common factors in both projects.

Figure 3: The CREWED process as piloted

Facilitator Library

3rd party resources, refs IP

Core academic course team

Discipline knowledge

Educational developers, e-learning support

Hands-on development support

Reality checkers

(e.g. students)

Development team

Continuing hands-on development work

Ongoing staff input

Initial brieing meetings Team

Resources e.g. lecture notes and slides, tutorial activities, podcasts, simulations, lab prep, activities, textbooks

2 day workshop

Course design

Complete development

Detailed plan

Run course how to improve efficiency and effectiveness


Learning and capacity building

Some common outcomes from both projects are:

■ identification of additional sources of support for developing and running technology-enabled learning activities, from within the disciplinary community;

■ building the experience and the expectation of teamwork with support staff (learning designer, educational technologist, library);

■ increased confidence in introducing new technology-enabled learning activities into courses.

In terms of meeting curriculum priorities and the learning needs of students, the design process worked better for the mining course than for the chemical engineering course. There are a number of possible reasons for this — differences in the team, in students and in the delivery mode. It is not surprising that 1st year students, many of whom have already failed the course, are less skilled as independent learners online than professional graduates. Yet other 1st year courses in the Faculty of Engineering have been able to introduce design project assignments where students work independently in groups, using online support blended with classroom sessions. The design of these courses, however, is much less didactic and content-driven than the engineering chemistry course, which in the main semester version also has a higher than usual failure rate. The course team therefore faced more challenges in the extent of redesign needed, and in developing a shared view of what could be done.

The summer course pilot identified that the timing was problematic, in that it is in a period when academics are busy preparing research grant applications. Showing how PhD students can help with design and assessment work will be very helpful for future blended summer courses.

Pedagogical models and learning design practice

In many engineering disciplines, planning of resources and workloads is part of the discourse. The academics involved in the UNSW pilots immediately appreciated (in theory at least) the concept of designing and planning student and staff time for new learning activities using spreadsheet models and even Gantt charts. On the other hand, some educational concepts and research methodologies are harder for engineering academics to understand

(Borrego, 2007). The CREWED process aims to overcome this through participation of skilled educational developers and other support staff working closely, hands-on, with Engineering academics to achieve a tangible result. The focus is on experiential team-based learning to achieve specific and immediate objectives within the discipline. One participant commented that the process is ‘staff development by stealth’.

The pedagogical models used may depend on context — the level of study and the institutional program structures. The Salmon 5-step model (G.

Salmon, 2004), provided a planning tool for a fully online postgraduate distance course. With a course team who initially were sceptical about the type of online facilitation needed, the measured pattern of student and staff online activity established the validity of this model.

Although the 5-step model was introduced to the team in the undergraduate chemical engineering course, the core academics resisted engagement with it, and preferred to structure the course around content topics. The learning models are still largely individual rather than social. This is problematic in a context where engineering graduates need strong teamwork skills. However, there was progress in that the learning outcomes for the course are now more clearly articulated, and the team are beginning to move towards more active learning models that are enabled by the technology.

Instead of wholesale adoption of particular learning models, the project aims



1 to adapt and combine pedagogical models to develop discipline and contextspecific models that can be owned by the academics involved, and verified by their own experiences of developing and running the courses — building what Argyris (1999) calls ‘actionable knowledge’.

Evaluation and conclusions

The pilots proved successful as the first part of an action research process that still has some way to go, particularly in developing a substantial knowledge base upon which to build workload models for designing, developing and running new types of learning activities and embedding these workload models into departmental systems. Further pilots are planned and the two reported in this paper have built the foundations for the next stage. In particular, the work has developed a new design, development and planning process for online and blended courses that can take into account discipline-specific curriculum development needs, and facultyspecific resource constraints. The pilots have also provided local examples of practical solutions to these constraints. While the specific solutions are context-specific, the process for reaching them could be used elsewhere.

Two more course development projects are underway using the CREWED process. One is a postgraduate blended course, piloting a new institutional online learning management system, run in 2009 semester 2. The other will be another 1st year undergraduate in blended format in the 2009–10 summer session. Both will build on what worked and what didn’t work in the pilots.

A particularly useful outcome has been the development of a course design spreadsheet template. The template is a simple tool to collect, capture and represent different aspects of a course design and its constituent learning activities along with the teaching work involved, and has already proved useful for other course development projects. There

Table 1: Learning from pilot projects

Postgraduate distance course

Initial consultations from March 2008.

Design & build workshops April. Build and test May-June. Course ran July-Nov

2008, with adjustments and additional resources added when required. Final review Dec 2009.

Learning outcomes already fairly well deined and most time spent on design of activities for online environment.

Moodle online tools and resources

(quizzes, video, notes, discussion forums, group assignment, inal assessment by individual work-related project report)

Used 5-step model to structure activities and plan facilitation.

21 students, started, 15 completed and passed (typical for PG Mining).

Feedback indicated students were engaged, but found assessment workload heavy. Appreciated response to requests for additional resources.

5-step model relected in student and facilitation activity (See Figure 4).

Small School with specialist academics.

Library rep ran one activity. Drafted in additional academic to help run online activities and mark assignments.

Undergraduate blended course Outcomes and comments

Initial consultations and lecture recording from Sept 2008. Design & build workshops October. Course ran

Nov 2008-Feb 2009. Final review Feb


Much work needed to clarify learning outcomes. Links between class and online activities problematic, as were relationships between topics and learning outcomes / assessment.

Final design built around 5 distinct content topics. Online topic-related quizzes and built-in feedback.

Conditional release of lectures recordings for subsequent topics on quiz scores.

Lab work, computer assessments and inal exam as in classroom mode, with

1 f2f tutorial.

The 1-day plus half day design and develop sessions were reasonably successful. Adequate preparation of resources beforehand is necessary, and academics need help with this.

The UG course team were more content-focused than the PG course, making the design process harder work and results poorer.

Salmon 5-step model useful for distance course, but less so for blended UG course. PG course assessment workload too high for staff and students.

UG blended pedagogy needs more work, but pilot provided useful online resources for use in main session course, which may help better use of f2f time.

9 students started, 2 dropped out immediately, 4 of remainder had failed course previously.

Direct Qs only in online forum, quizzes appreciated, but lecture recordings

(main resource) not used as intended.

Much better results with PG distance course than with UG blended course.

This could be because of student independent learning skills and motivation at PG level, combined with more coherent course design.

Students poorly prepared for f2f tutorial and only 3 passed exam.

Some useful learning on design of online group tasks.

Academics unavailable for online facilitation. Research student converted existing tutorial questions to online activities with feedback, and responded to students’ online Qs.

Both courses needed more facilitation input than the academics anticipated and involved an additional person.

Use of research student to augment academic staff worth repeating in other courses.


Figure 4: Student and staff online activity in the Mining Engineering course











All is potential to develop this idea further, perhaps using some of the other available tools. The use of such tools is not new, but their integration with local conditions and staff planning is a new development. By using a simple tool such as a spreadsheet, it was possible to discuss and negotiate, without imposed preconceptions about pedagogical models or the shape of the course activities, and to map out the practical resource implications while discussing options.

The two pilot projects were designed using an underlying conceptual framework that treats disciplinary learning and teaching as a complex adaptive system. The outcomes have illustrated some specific benefits in this approach. In particular, the CREWED process allows for pedagogical models to be negotiated in a curriculum development context, and for activities using learning technology to be designed and adapted along with the development of team processes. Whereas a focus on the quality of the learning design might have produced a better short term outcome for students in both courses, it could have done so at the expense of academic staff ‘burnout’, as described by Laurillard (2002) — had it been possible to engage the relevant academics in the first place. The CREWED approach aims to address the whole learning and teaching support system, so that academic staff can negotiate how much of their own time is spent, as part of a team. Linking with curriculum development priorities was essential for gaining formal support and resources, and the results provide evidence of value that can be used to argue for more resources.

The initial phases of this action research have therefore confirmed the necessity for research that deals with learning technologies as an integral part of a broader learning and teaching system — not just the pedagogical design, but also the academic context and staff workloads. The two pilot projects have extended the work done in UK universities to introduce teamwork and build capacity, by adding an explicit investigation of staff workloads and skills. This has given specific information to the two academic departments concerned, which will enable them to plan future online and blended courses more effectively, avoiding some of the academic workload barriers by using additional support staff and more teamwork. Without such explicit proof of the need for, and benefits from, extra support and skills, it is hard to argue for budget allocations and staff time.



While the pilots themselves are limited in scale and scope, they have laid some groundwork that can be built upon within the UNSW Faculty of

Engineering, and further afield. The Chemical Engineering course also exemplified barriers to teamwork across departmental boundaries, in service teaching arrangements. This is a central curriculum development issue in

Engineering, which also occurs in other professional disciplines.


Argyris, C. (1999) On Organizational Learning ( 2nd edition ed.). Oxford UK: Blackwell

Borrego, M. (2007) Conceptual Difficulties Experienced by Trained Engineers Learning

Educational Research Methods. Journal of Engineering Education (Washington, D.C.),

96(2), 91–102

Diego, J. P. S., Laurillard, D., Boyle, T., Bradley, C., Ljubojevic, B., Neumann, T., et al.

(2008) Towards a user-oriented analytical approach to learning design. [research paper]. ALT-J

Research in Learning Technology, 16(1), 15

Kezar, A. and Eckel, P. (2002) Examining the Institutional Transformation Process: The

Importance of Sensemaking, Interrelated Strategies, and Balance. Research in Higher Education,

43(3), 295–328

Laurillard, D. (2002) Rethinking University Teaching: a conversational framework for the

effective use of learning technologies (2nd ed.). London: Routledge Falmer

Laurillard, D. (2008) London Pedagogy Planner (LPP). London: JISC

Mitleton-Kelly, E. (2003) Complexity research — approaches and methods: The LSE Complexity

Group integrated methodology. In A. Keskinen, M. Aaltonen & E. Mitleton-Kelly (Eds.),

Organizational Complexity. Helsinki: Tutu Publications, Finland Futures Research Centre

Russell, C. (2009) A systemic framework for managing e-learning adoption in campus universities:

individual strategies in an institutional context. ALT-J Research in Learning Technology, 17(1)

Salmon, G. (2000) E-moderating: the key to teaching and learning online. London: Kogan Page

Salmon, G. (2001) Approaches to researching teaching and learning online. In C. Steeples & C.

Jones (Eds.), Networked Learning. London: Springer-Verlag

Salmon, G. (2004) E-moderating: the key to teaching and learning online (2nd ed.). London and

New York.: Routledge

Salmon, G., Jones, S. and Armellini, A. (2008) Building institutional capability in e-learning

design. ALT-J Research in Learning Technology, 16(2), 15

Trevitt, C. (2005) Universities learning to learn? Inventing flexible (e)learning through first and second-order action research. Educational Action Research, 13(1), 5783

Uys, P., Buchan, J. and Ward., L. (2006) Digitally enhanced blended learning: a comparative

study of Australian Universities. Paper presented at the ASCILITE

Zuber-Skerritt, O. Y. T. (1992) Action Research in Higher Education: examples and reflections.

London: Kogan Page



Sarah Cornelius, Carole Gordon,

Margaret Harris

School of Education, MacRobert

Building, University of Aberdeen,

King’s College, Aberdeen AB24 5UA [email protected], [email protected], [email protected]


Advocates suggest that anonymity allows all learners to have an equal voice in a learning environment, and that it encourages participation. This paper explores tutors’ and learners’ experiences of an anonymous, synchronous role play activity conducted using online discussion forums. A qualitative study was undertaken to investigate the experiences of ive groups of learners and four tutors. Data were obtained from an online questionnaire and interviews with students and tutors.

The indings reveal a huge diversity in responses to the activity. Learners’ emotions before the activity ranged from

‘conident’ to ‘panic’. Afterwards many stated that ‘anonymity’ was the best thing about the activity, suggesting that it ‘loosened inhibitions’ and allowed

‘unfettered expression of thought’.

At the same time, some respondents admitted trying to guess the identity of participants, and played their roles with varying degrees of conviction and engagement. Some participants may even have refrained from playing any part in the activity, hiding behind their anonymity. For tutors issues of control were signiicant and issues of facilitation were raised, although inappropriate behaviour was rare.

This study has revealed the diversity of learners’ responses to online role play, and the generally positive attitude towards anonymity. It also highlights the potential for anonymity to contribute to inequality in participation and raises the question of whether genuine anonymity can be useful or achievable.

Key indings with signiicance for future implementation of similar role play activities are presented here.

0055 ‘Unfettered expression of thought’? Experiences of anonymous online role play


There are many reported advantages of anonymity for online learners, including equality of opportunity, increased choice, high participation rates, enhanced disclosure, and the removal of gender and cultural expectations

(Chester and Gwynne, 1998; Freeman and Capper, 1999). Sullivan’s (2002) study of female American college students illustrates these benefits and identified anonymity as the most important aspect of learners’ online experience — equalising advantage, increasing openness and honesty, developing trust, and removing stereotyping, bias and fear.

Role play is regarded by some as an enjoyable, engaging and effective learning activity, which in the online environment is emotionally safer and lower risk than a face to face equivalent (Freeman and Capper, 1999; Bell,

2001). Vincent and Shepherd (1998) provide an early example with a team simulation using email to address issues of Middle East politics. Other applications in areas as diverse as business, learning and natural resource development have also been effective (e.g. Freeman and Capper, 1999; Bell,

2001 and McLaughlan et al., 2001) with benefits including the increased awareness of different perspectives and the development of soft skills

(McLaughlan and Kirkpatrick, 2005). Project EnRoLE, ( cedir/enrole/index.html), set up to assist dissemination of information about role play in university teaching, provides a comprehensive set of resources and references.

There are, however, challenges associated with combining anonymity and role play. Freeman and Capper (1999) observed ‘playfulness’ in a group of anonymous postgraduate learners, and there is the potential for this to slip into anti-social behaviour and even harassment. Chester and Gwynne (1998) and Freeman and Bamford (2004) report ‘theft’ of identities and subsequent use to flame or ‘denigrate’ other students. In addition, role play itself can induce feelings of fear, anxiety and guilt amongst participants and tutors and disengagement with roles can be an issue (Bell, 2001; Freeman and Capper,


Bell (2001) called for further work to investigate levels of involvement and role engagement in online role play, and the effects of asynchronicity and anonymity. Most work on online role play to date has reported on asynchronous activities and often highlights the benefit of time for reflection before responding to messages posted by others (Wills and McDougall,

2008). This paper contributes to the exploration of anonymity through consideration of a synchronous online role play which requires quicker responses, replicates more closely a face to face equivalent and overcomes the problem of the time taken for students to post (Douglas, 2007). The issue of equality of opportunity, raised by many as an advantage of anonymity, is explored, and questions are raised about the authenticity of anonymity in online activities.

The online role play activity

A synchronous online workshop was designed as part of the Teaching

Qualification Further Education (TQFE) programme for in-service lecturers



1 in Scottish Colleges (an SCQF level 9 qualification). The aim of the two and a half hour workshop was to explore issues of quality in Further

Education through two main activities: discussion of a reading and a role play activity. Both activities were facilitated using a WebCT discussion forum. For the discussion of the reading respondents posted messages in response to questions posed by the tutor. This activity was not anonymous and allowed the tutor to ensure that everyone was present in the online space and able to contribute. At the start of the role play activity, learners were split into groups of 4–6 and each individual allocated a role (student, tutor / lecturer, manager or support staff). The groups were provided with a discussion thread in which to consider the question ‘What is quality in

Further Education?’ Individuals were asked to provide a perspective on the question from their allocated role. At this stage the forum was ‘switched’ to anonymous, so that the author of any posting could not be identified.

Towards the end of the workshop the tutor set up new discussion threads to promote exchange of ideas between groups and reflection on the content and process of learning. The activity was not formally assessed but subsequent face to face discussions provided valuable informal formative assessment opportunities.

Following successful piloting of the workshop in the previous academic session, in 2007/8 four tutors facilitated similar online workshops for sixtysix learners in five groups. Each group consisted of up to twenty lecturers from one or more colleges.

A mixed methods study was conducted to allow description and exploration of learners’ experiences of the online role play. Forty five participants provided usable responses to an online questionnaire immediately after the workshop

(response rate = 68%). Sixty two percent of respondents rated themselves as regular users of WebCT prior to the workshop, whilst another eleven percent had used it for a previous TQFE online workshop. Sixty nine percent of respondents rated themselves ‘fairly confident’ computer users, four percent were ‘very occasional’ users and another four percent ‘completely at home online’. These findings suggest that almost three quarters of the group surveyed could be regarded as competent computer users.

Five of the students who responded to the questionnaire also participated in a telephone interview. The sample of interviewees selected provided a crosssection of colleges (e.g. rural / urban), and IT experience (novice to expert).

The four tutors facilitating the online workshops were also interviewed.

Analysis of data was informed by a grounded theory approach which allowed findings to emerge from the data rather than being influenced by any preconceptions. Interview data were transcribed and then subjected to thematic analysis using a constant comparison approach.

Qualitative data from interviews and questionnaires were then combined to provide evidence of:

■ feelings at the start of the role play;

■ experiences of the role play;

■ help and support requirements;

■ the most significant learning from the activity; and

■ comments and suggestions.

This paper draws principally on data from learners’ and tutors’ experiences of the role play and focuses specifically on the issue of anonymity. The research raised other issues of interest, for example the effectiveness of the role play in promoting learning about quality issues, and these aspects have been reported elsewhere (Gordon et al. 2009).


Experiences of the role play

Overall the findings reveal a diversity of experiences and responses to the activity. Learners’ emotions before the activity ranged from ‘confident’ to

‘panic’. Afterwards the most commonly mentioned feature of the role play was the anonymity. Some learners suggested that anonymity had ‘loosened inhibitions’ and allowed ‘unfettered expression of thought’. Others were less convinced of the role of anonymity in the success or otherwise of the activity, whilst some appeared to hide behind the anonymity provided and refrain from contributing. Several respondents noted that they had tried to guess the identity of participants, and there is evidence of roles being played with varying degrees of conviction. Some tutors confessed to being anxious about the activity, and although in most cases this was a very successful activity (from both tutors’ and learners’ perspectives), it raised issues about inappropriate behaviour, level of commitment and the value of anonymity in learning. These issues are illustrated and explored further below.

Throughout the abbreviation QR is used for ‘questionnaire respondent’, I for

‘interviewee’, and T for ‘tutor’.

Beneits of anonymity

Anonymity was identified by forty percent of questionnaire respondents as one of the ‘best things’ about the role play activity. Respondents considered that anonymity helped them play their roles:

“[anonymity] seemed to help me get more ‘into’ the role’ (QR3)

“People feel free to take on roles due to anonymity” (QR12)

Anonymity allowed a sense of freedom and lack of inhibition which clearly had an impact on some learners:

“[the best thing about the online role play activity was] feeling safe and free enough to be honest” (QR16)

“the anonymity […] was a clever idea that allowed for the unfettered expression of thought” (QR23)

The opportunity to speak freely without fear of identification led to increased participation and the opportunity to consider a wide range of perspectives:

“I think people said more than they would normally because it couldn’t be traced back to the individual” (Q13)

“[the best thing about the online role play activity was] stating a point of view and reading others’ points of views without knowing who they were. It led to a very open discussion” (Q44)

One of the tutors also felt that anonymity had been beneficial in terms of participation:

“I think the anonymity that students had allowed them to participate better than they would in a face to face situation” (T2)

The lack of sound contributed further to the anonymity “because no-one can hear your voice” (I3). For one the anonymity only worked because “we knew each other” (I1), whilst another seemed more ambivalent about the role of anonymity:

“I don’t think [anonymity] mattered very much, but it probably helped. It was probably better to be anonymous” (I5)

Playing roles

This activity required learners to take on specific roles, and success depended in part on how well they did this. Whilst anonymity may have been important in providing an atmosphere in which roles could be played in



1 safety, there were inevitably mixed reactions to the effectiveness of learner engagement with the role play. In particular the nature of the role (student, tutor / lecturer, manager, support staff) that participants adopted appears to be significant, with a lecturer / tutor role easiest to play:

“Role play was ok because I was a tutor, so I was already in role, so it didn’t feel terribly strange” (I2)

“people did manage to get into their role […] because of the topic which was close to people’s hearts. We know a lot about it, and felt quite strongly about it, and I think that’s the reason why it was so effective” (I1)

“some people got right into their role — that of manager” (I5)

For some individuals playing their role was a challenge:

“I found it very dificult [to get into role], because my role was a support worker” (I2)

Difficulties with playing roles may have resulted in certain roles dominating whilst others were missing in some groups:

“I would say that the lecturer and student input was much greater than the others and it was the same in the other group […] not everyone became involved […] there were roles missing”” (I4)

One tutor noted different styles of engagement, ranging from full involvement: “some people take it very seriously” (T2), to a more surface approach. Another incident revealed how a prompt from the tutor was necessary to get an individual into their role:

“One member of the group who I had given a student role […] spoke as if he was a lecturer […] so I posted a message to him saying “what’s the perspective of a student? […]” but he didn’t pick that up and he didn’t move in it at all” (T4).

A student also noted similar difficulties in playing their roles:

“People weren’t really divesting themselves of the lecturer role. Almost subconsciously people were still operating from the lecturer role” (I2)

A lack of involvement and role playing by some learners was an important issue which impacted on the experience. In one group the management role was absent despite a student being allocated this role. The tutor commented

“there were one or two I’m sure didn’t participate. I know it’s anonymous, but I have this feeling that they did not participate at all” (T1). A possible explanation was provided by the tutor:

“when they started the anonymous part, something went wrong with one of the groups because we ended up with having two managers in the one group. I think what had happened was that one person who should have been in one group had gone into the second group […] so the manager was missing in one group, which is an important point, because if you wanted to get a varying view, you needed the management view. So the irst group didn’t work at all.” (T1)

In response to this situation the tutor posted a message to group one to say that that management weren’t saying anything. Someone from group two noticed, and volunteered to help out in group one. The tutor continues:

“That showed that they had been looking at [the] other [group’s] postings, which they weren’t supposed to be doing. And in group two someone recognised the bloke who was playing the manager, and mentioned him, so everybody know who it was

[…] That spoiled it. The anonymity didn’t really work in that group” (T1)

Another tutor anticipated individuals not playing their part and tried to preempt any problems by careful allocation of roles — “if there were six in a group instead of four there was more chance to get people involved and it did not matter if roles doubled up but it coped with some of the problems of people not participating” (T2).


One strategy for dealing with quiet or non-contentious groups, which was possible due to the anonymity of the activity, was for the tutor to step in and post a message in one of the allocated roles. However, some tutors were hesitant about doing this:

“At one point I saw group one, they weren’t participating, and I thought I’d put in a few contentious things to start getting people to argue. In the end I decided not to […]

I decided that I would take no part in putting the comments in, so that if something did come back later I could say, “it wasn’t me — don’t take this out on me.” (T1)

Very occasionally tutors ‘seeded’ the discussions with messages, particularly in an attempt to get things started, perhaps in response to their own anxieties about the activity:

“I get very nervous, after I have put them into the roles, when nothing happens. Some will be typing away furiously. Others may be thinking and others may be confused.

[…]. Perhaps I should be keeping track of who is who and who is saying what” (T2)

Bell (2001) found almost two weeks elapsed before participants made contributions to her asynchronous role play activity, so it may be that there is an initial reluctance to start a role play discussion amongst learners, perhaps related to issues of fear and anxiety (Freeman and Capper, 1999).

The same tutor who expressed anxiety above, later commented on the evidence of her group playing their roles effectively:

“I […] was surprised that when the managers start speaking you get ‘management speak’ and students’ misspell (whether deliberately or not). They seem to adopt the roles quite effectively and they all seemed to participate” (T2)

Appropriate and inappropriate behaviour

Aside from not playing the allocated role at all, issues of appropriate and inappropriate behaviour arose. In some groups all went well:

“Everyone behaved themselves. There were times when it got a little bit heated.

Because we knew each other we were able to do that with each other and not take it too far […] people did not get upset because they realised it was a role play, and they were having a bit of fun” (I1)

T3 commented on initial problems with people knowing which role they were playing and exchanges of emails for confirmation of these, but then the actual role play went well:

“I think people forgot who the other people were, they got into role and simply responded to people […] in their individual roles. […] it all seemed to be happening according to plan.” (T3)

The value of knowing your peers in a group has already been identified, and the same respondent noted this factor as a possible reason for appropriate behaviour:

“[…] humour helps within the group, and knowing each other helped. If we hadn’t known each other, then possibly it could have got out of hand” (I1)

As with any role play, some participants played up their roles:

“It was a laugh, some of my peers took their roles to extremes” (Q39)

“I think some people got a little bit carried away at times as you tend to do in a role play knowing that there’s not going to be any consequence from this” (I3)

In some groups, issues became a little ‘sillier’ or more heated:

“they were just giving a few words in an answer and then replying to one another. So there wasn’t the depth in it. And I thought we might lose it altogether at one point.

However […] it all came back together and people started giving proper answers and it ended really well” (T4)




“There were a couple of off-the-cuff comments, there was one use of bad language

[…] I took it very much as tongue in cheek.” (I2)

It appears that some of the problems encountered resulted from a misunderstanding over instructions (for example some students clearly thought they could choose their own role), whilst others had difficulty posting messages in the correct thread, or following instructions to engage only with their small sub-group. Some learners took time to compose responses in a word processor before posting these for the group, whilst it was acknowledged by tutors that more confident IT users appeared to be

‘flitting around’ reading and contributing shorter, less thoughtful messages in a variety of forums.

Significant inappropriate behaviour of the type reported by Chester and

Gwynne (1998) was not experienced during this online role play, although what is regarded as inappropriate by one tutor or group of learners may be different from that which perturbs others. Perhaps more significant from the tutors’ perspective were the difficulties in monitoring an anonymous activity, particularly in terms of identifying non-contributors and those who have misunderstood instructions or need support. As Chester and Gwynne (1998) note ‘silence is not easily interpreted’, and there is the added complication that silent participants cannot be easily identified in an anonymous activity.

It may be useful for tutors’ to be aware of participants’ identity, although this in turn may impact on learners’ feelings of freedom. Awareness of identities by tutors would, however, allow the provision of quick and appropriate support to encourage engagement and open up opportunities for learners facing difficulties. McLaughlin and Kirkpatrick (2004) suggest that data on participant logins collected by software can be used to detect inactivity, but this may not be practicable in a synchronous activity when a moderator is simultaneously trying to monitor discussions.

Despite some of the challenges mentioned, overall the workshop was generally regarded as effective, particularly in the way that it allowed different perspectives on the issue of quality to be appreciated:

“There wasn’t very much silliness in this group, although there was more political stuff, soft of getting things off their chest about their organisation, sort of tricky things going on there, but in general absolutely super […] they raved about it, they thought it was a really powerful experience, they’d learnt a lot, so that was very positive” (T4)

Uncovering identities

Another management issue was the authenticity of the anonymity. Although

Freeman and Capper (1999) suggest that one of the advantages of online role play is that it can offer anonymity, despite tutors’ best efforts it is clear that learners were often keen to know with whom they communicating:

“I spent a lot of time looking and reading what others had said and trying to imagine who had said what […] I think we all did a bit of that, trying to guess who had taken on what role” (I3)

Some learners even developed strategies for uncovering the identity of their peers (for example, one learner informally told their tutor that they had printed out the list of roles provided in the forums before this was deleted).

The tutor recognised that “students are quite crafty that is why I try to take off the group list before they catch on to who is doing what […] we need to make sure that it is really anonymous” (T2)

The issue of genuine anonymity was also raised by another tutor:

“Although they were anonymous I was afraid that they might trace it to somebody and therefore be aware of who had been speaking. I think they might have got a bit over-excited about being anonymous and actually felt it was genuine anonymity whereas I suppose I don’t think there is any such thing at the end of the day” (T4)


One tutor had devised a method to ensure that participants could not guess the identity of other participants. She allocated everyone a number before the workshop then on the day showed them which role each number represented. “My reason for doing that was that I didn’t want them to be able to identify anybody and I thought that the chances of them remembering someone else’s number was pretty slim. In fact the chances of them remembering their own number was pretty slim too. We did have phone calls on the morning because they had forgotten their number.” (T3)

Tutors also commented on their feelings and experiences of managing an anonymous online activity. The lack of non-verbal feedback to reassure them that learners are OK was one issue which induced anxiety:

“I think if there is any sort of joking around in the [face to face] classroom I am pretty skilful at managing that. I would deinitely have a laugh over whatever it was, but I might say something just to turn the conversation or to move it in a different direction and it wasn’t possible to do that online. That’s maybe why I felt a bit nervous about it” (T4)

Where anonymity was compromised this also had an impact on the activity.

In one group three students were working together in the same room, and when they were ‘thrown out of that room’ had to work together from one PC:

“They then posted one message with student 1,2,3 and their names attached, so it was not anonymous. […] It did affect the role play. I had to post a message to tell them not to put their names on and to delete the message already posted. It spoilt the beginning […] I thought [the role play activity] was collapsing around my ears.

But it did get better.” (T2)

The software itself also created problems. Two tutors commented that they would have liked to return to using real names at the end of the workshop or be able to “turn on and off the anonymous when we wanted to” (T3), but the software used prohibits this. In order to keep track of individuals ‘behind the scenes’ monitoring of student input was undertaken by several tutors.

T3, for example, had a “sheet with all the different tasks and I was ticking them off — I ticked them off when they were in, I ticked them off when they responded to the reading. Once we went anonymous of course I couldn’t do that” (T3).

Whilst being regarded by learners as an important and engaging aspect of online role play, ‘anonymity’ should perhaps be regarded as a relative concept. Given the use of technologies which will allow the tracing of participants, there is always the possibility that those disrupting online role play activities could be brought to task, for example by tracking harassing message senders through IP addresses (Freeman and Capper, 1999).

In addition, our findings suggest that learners themselves are often keen to

‘crack’ the anonymity and will search for clues and develop strategies to uncover the identity of their peers in a role play activity. Whilst this may be a situation influenced by the fact that the groups involved in the role play knew each other in person, this is an important issue. Anonymity has been suggested as a having the potential to reduce social presence (Bell, 2001;

Gunawardena and Zittle, 1997), that is to reduce the degree to which other people are real. The fact that all participants were online at the same time may have increased social presence and led to a desire to find out who peers were. Bayne (2005) has also suggested that even in an anonymous online context “we cannot simply throw off the ways in which who or what we can be online is informed by our existence as subjects with bodies.”

Conclusions: does anonymous online role play permit the ‘unfettered expression of thought’?

Despite the widespread promotion of the advantages of online role play as



1 a way of providing a more level playing field than face to face role play, in particular through the advantage of anonymity, this study has raised some interesting issues which provide an alternative perspective. Clearly from such a case study generalisation to any larger sample of learners or tutors is not possible, however, some of the key findings may be of interest or help to others designing or facilitating online learning:

■ tutors and learners have anxieties about online role play

■ technology issues influence the ability of learners to participate fully

■ barriers to role engagement go beyond the cultural and language difficulties identified by other researchers (e.g. Bell, 2001) and may also be related to confidence with IT and the ease of identification with the role allocated

■ the reliance on written contributions promotes reflection and encourages some learners to produce thoughtful written contributions, whilst others will take a more surface approach

■ moderation and monitoring by tutors is challenging, for example it is difficult to identify those who are silent and the reasons for their silence, and therefore difficult to offer appropriate support.

As a result the experiences of learners in online role play appear to be diverse and difficult to characterise. Whilst some may experience the activity as an opportunity to provide open and honest comment on the scenario from the perspective of their allocated role, others may be prevented from engaging in the same way by issues of confidence, identification with their role, and technical difficulties.

The issue of anonymity has been explored in particular detail as this was raised by participants as one of the best features of the role play. Anonymity provides a number of benefits for learners, including the opportunity for openness and equity in an activity, however, it is clear that not all learners will share this experience. Some will find engaging with roles difficult and may not contribute as a result. The use of anonymity also raises issues for tutors, not just in the management of appropriate engagement, but also in monitoring contributions and providing appropriate support for learners. In addition it is clear that anonymity is in this context a relative concept, with no absolute guarantee of ‘invisibility’ possible.



Bayne, S. (2005) Deceit, desire and control: the identities of learners and teachers in cyberspace.

In Land, R. and Bayne, S. (eds) Education in cyberspace. London: RoutledgeFalmer

Bell, M. (2001) Online role-play: anonymity, engagement and risk. Educational Media

International 38(4): 251

Chester, A. and Gwynne, G. (1998) Online teaching: encouraging collaboration through

anonymity. Journal of Computer Mediated Communication, 4(2). issue2/chester.html

Douglas, K. (2007) Mediator accreditation: using online role plays to teach theoretical issues.

Australasian Dispute Resolution Journal 18(2): 92–100

Freeman, M. and Bamford, A. (2004) Student choice of anonymity for learner identity in online

learning discussion forums. International Journal on E-Learning, 3(3): 45–53

Freeman, M.A. and Capper, J.M. (1999) Exploiting the web for education: An anonymous

asynchronous role simulation. Australian Journal of Educational Technology, 15(1): 95–116

Gordon, C., Cornelius, S. and Harris, M. (2009) Quality matters: online role play for Scottish

Further Education staff. Proceedings of International Conference on e-learning in the

Workplace. Columbia University, New York, June 2009

Gunawardena, C. N. and Zittle, F. J. (1997) Social presence as a predictor of satisfaction

within a computer mediated conferencing environment. The American Journal of Distance

Education 11(3): 8–25

McLaughlan, R. G., Kirkpatrick, D., Hirsch, P. and Maier, H. R. (2001) Using online

roleplay/simulations for creating learning experiences. CAL-laborate 7.


Mclaughlan, R. G., and Kirkpatrick, D. (2005) Online text-based role play-simulation: The

challenges ahead. Simtect, Sydney: Simulation Industry Association of Australia. www.siaa.

Sullivan, P. (2002) “It’s easier to be yourself when you are invisible”: female college students

discuss their online classroom experiences. Innovative Higher Education, 27(2): 129–144

Vincent, A. and Shepherd, J. (1998) Experiences in teaching middle east politics via internet-

based role-play simulations. Journal of Interactive Media in Education, 98(11): 1–35

Wills, S. and McDougall, A. (2008) Reusability of online role play as learning objects or

learning designs. In Lockyer, S., Bennett, S., Agostinho, S. and Barper, B. (eds.) Handbook of Research on learning design and learning objects: issues, applications and technologies.

IDEA Group PA. pp. 761–776





0084 Between analysis and transformation: technology, methodology and evaluation on the SPLICE project


E-learning research as a subset of educational research has adopted many of the methodological approaches of the social sciences towards evaluation.

Typically these tend to be characterised by evidence-based approaches to identifying project performance against indicators, outcomes and baselines (JISC, 2007). Despite an increasing awareness of the importance of continual evaluation throughout a project’s duration — particularly with regard to agile and iterative projects, such approaches remain close to the classic model of quasi-experimental research and evaluation drawn up by

Campbell and Stanley (1963). Such techniques rely on traditional social science methods of data collection including questionnaires, focus groups and interviews, with little use made of specific technological innovations to support the evaluative methodology.

The situation is different in fields outside e-learning, notably Operational

Research, where much work on methodological development in evaluation and research has been conducted in recent years, with not only concepts and techniques but new technologies arising. These have included various techniques for problem specification deriving multiple stakeholder viewpoints from Soft Systems (Checkland, 1990) approaches (for example, the Strategic

Options Development and Analysis (SODA) technique which is designed to aid participative problem definition with the use of modelling software

(Eden and Ackermann, 1989)). Different kinds of software innovation helped

Beer’s Team Syntegrity (1994) take an alternative approach to Operational

Research which seeks to gain consensus about problem definitions and solutions through guiding stakeholders through a highly structured set of activities to explore different viewpoints. Mingers argues that these techniques can be combined using multi-methodological approaches to deal with highly complex problems on condition that the different world-views associated with each technique are surfaced and evaluated (Mingers, 2004).

This paper argues that technology creates not only a context for research in e-learning, but also a context for understanding outcomes. As such, technological development necessitates methodological development as institutions seek better ways to understand and adapt to the transformational effects that technological and social change has on them. To this end, we report on the use of a methodologically innovative approach to evaluation,

Realistic Evaluation (Pawson and Tilley, 2002), on a JISC-funded project on social software, SPLICE (Social Practices, Learning and Interoperability in Connected Environments), and on the technological innovations that accompanied the use of this methodology.

The SPLICE project and realistic evaluation

The SPLICE project focused on the ways individuals and institutions change their technological habits with social software, and the things that can be done to effect these changes. Over the course of 18 months, learners, teachers and industrial partners in the creative industries were exposed to


Mark Johnson, Peter Lager, Bill Pollard,

Graham Hall, Miranda Edwards, Pete

Whitield, Rupert Ward

University of Bolton, Deane Road,

Bolton BL3 5AB [email protected]


This paper concerns the ways in which technological change may entail methodological development in e-learning research. The focus of our argument centres on the subject of evaluation in e-learning and how technology can contribute to consensus-building on the value of project outcomes, and the identiication of mechanisms behind those outcomes.

We argue that a critical approach to the methodology of evaluation which harnesses technology in this way is vital to agile and effective policy and strategymaking in institutions as the challenges of transformation in a rapidly changing educational and technological environment are grappled with.

With its focus on mechanisms and multiple stakeholder perspectives, we identify

Pawson and Tilley’s ‘Realistic Evaluation’ as an appropriate methodological approach for this purpose, and we report on its use within a JISC-funded project on social software, SPLICE (Social Practices,

Learning and Interoperability in Connected

Environments). The project created new tools to assist the identiication of mechanisms responsible for change to personal and institutional technological practice. These tools included collaborative mind-mapping and focused questioning, and tools for the animated modelling of complex mechanisms. By using these tools, large numbers of project stakeholders could engage in a process where they were encouraged to articulate and share their theories and ideas as to why project outcomes occurred. Using the technology, this process led towards the identiication and agreement of common mechanisms which had explanatory power for all stakeholders.

In conclusion, we argue that SPLICE has shown the potential of technologicallymediated Realistic Evaluation. Given the technologies we now have, a methodology based on the mass cumulation of stakeholder theories and ideas about mechanisms is feasible. Furthermore, the summative outcomes of such a process are rich in explanatory and predictive power, and therefore useful to the immediate and strategic problems of the sector. Finally, we argue that as well as generating better explanations for phenomena, the evaluation process can itself become transformative for stakeholders.

various interventions to encourage adoption of online social habits. The interventions included the use of social forums, micro-blogging, and specific learning activity designs. The project was focused on those interventions which produced change in which circumstances. In some cases, the interventions ‘worked’ (e.g. learners or teachers transformed their habits); in some, they didn’t. In most cases, the interventions can only be said to have ‘sort of ’ worked. In addition to identifying what worked and where, the evaluation was concerned with identifying ‘why’ things did and didn’t work, and particularly what ‘sort of ’ working meant. This entailed an ongoing process of modelling outcomes throughout the project, with the ultimate aim of producing realistic models of change in habit which could then be used by institutions to plan strategy and policy with social software.

Realistic Evaluation was chosen as a methodology to support this. It asserts that discoverable mechanisms are responsible for social phenomena, and that better knowledge of these mechanisms can give greater control to practitioners, whether teachers, administrators or learners. In asserting the role of mechanisms in the social world, Realistic Evaluation is rooted in the philosophy of Critical Realism (Bhaskar, 1977). Pawson and Tilley argue that the job of evaluation is to uncover those mechanisms through a process which they (following Bhaskar) call ‘retroduction’. In essence, retroduction involves describing the context within which a possible mechanism might be responsible for producing a particular outcome. The relationship between

Context, Mechanism and Outcome is shown in Figure 1. In line with the

Critical Realist position, Pawson and Tilley argue that, whilst the experience of a project to any particular observer (or stakeholder) might be different

(or relative to the observer), those experiences are not that different. In other words, they may be the product of a common mechanism working within each individual context. Thus, in encouraging individual participants to articulate the mechanisms that they feel to be responsible for what they experience, it may be possible to consider overarching explanatory frameworks which describe mechanisms which are common to each. Such overarching mechanisms can then be considered for their explanatory and predictive power with regard to each individual outcome.

In SPLICE, the Realistic Evaluation approach was geared around identifying the causal mechanisms behind the impact of the interventions on the different stakeholders in different contexts in the project. With the identification of these mechanisms, institutions could be equipped with models that would predict the likely consequences of interventions around social software, and thus be in a better position to guide policy and strategy.

In other words, the ‘value’ of the SPLICE project could be realised in the form of knowledge that was practically useful to other institutions.

The practicalities of identifying mechanisms between the large variety of stakeholders on the project was challenging. In addition to basic questions like “how can a mechanism be captured or expressed?” or “how can common mechanisms be agreed?”, there were organisational problems concerning how all the different stakeholders together with the variety of different project activities could be represented. Unlike Operational

Research techniques, Pawson and Tilley’s Realistic Evaluation has not given rise to particular technologies. However, given that the central approach

Figure 1: Context, mechanism and outome in Realistic Evaluation






1 of the methodology is the identification of mechanisms, and its emphasis is on multiple stakeholder engagement, technology would appear to have something to contribute. In SPLICE technology, as well as being the object under investigation, also proved to be an important factor in the evaluation process, with specific tools developed to aid the realistic evaluation process.

The SPLICE context

Realistic Evaluation relies on sharing the contexts and outcomes from multiple perspectives. These different contexts and outcomes reflect the variety of stakeholder perspectives on a project. Like most learning technology projects, there were a large number of stakeholders in SPLICE, which included:

Technical developers.

Project managers.


Accounting managers.

Institutional administrators.

Funding body programme managers.

Creative Technology practitioners.


The stakeholders within SPLICE had different experiences of it. Those learners with whom the project interventions ‘worked’ reported significant personal transformations. These were, however, the minority. For most, the picture was more complicated, with some continuing to feel uncomfortable with social software, and others ‘dabbling’ without feeling they wanted to engage at a deep level. Amongst other stakeholders, institutional administrators varied in their experiences of the project, from simply managing the project money, to identifying key synergies between project outcomes and institutional objectives. Individual teachers varied in their experiences, from overcoming reticence to engage in new technologies, to transforming their teaching practices. Despite the variety of these experiences however, common patterns of experience were discernable. The purpose of the realistic evaluation approach was to elicit the nature of these common patterns between the groups of different stakeholders.

From the Realistic Evaluation perspective, each stakeholder could report an outcome (or a number of outcomes) from the project. With each outcome, a context for that outcome could also be established. Given these reports of outcomes, and identification of contexts, the evaluative task was to explore possible mechanisms which might be responsible in each case. The starting point for identifying mechanisms was to ask the stakeholder “what do you think is going on?”. Sometimes, this would produce interesting results, as individual stakeholder theories could be gathered, compared, shared and tested. At other times, theoretical explanations could be presented to the stakeholders to see if they felt that such explanations were meaningful to them.

This process of engaging stakeholders and encouraging theorising continued throughout the project. The project was designed around an iterative action research cycle of Plan



Review (similar to that articulated by

Argyris and Schön (1974)). This regular theorising helped to build models of “what’s going on?” as the project progressed. These models were then used to design the next cycle of interventions. As a starting point, the project began with some ‘ideal’ models and mechanisms which were used to plan the initial interventions. These were largely drawn from work on the Personal

Learning Environment (Johnson and Liber, 2008), and involved a model of personal organisation drawn from Stafford Beer’s Viable System Model

(1981). In the light of the results of actions taken against the backdrop of this model, both the model and action plans were modified. One advantage


of the iterative method was that, with many project partners, it was often appropriate to let individual partners shape their contribution to the project in the light of local conditions within their institutions. Within the Realistic

Evaluation framework, this was simply to provide different ‘contexts’ for actions, and to report on outcomes observed in the light of those contexts.

This process was repeated over the course of the project.

As the project progressed, low-level mechanism descriptions and theories needed to be cumulated into over-arching explanatory frameworks. For this, technology became an important means by which complex mechanisms could be articulated, and stakeholder opinions shared. Through the use of tools, a higher level of synthesis and cumulation of experiences, results, ideas and theories about “what was going on” could be shared and explored.

SPLICE tools for evaluation

In this second stage of the evaluation, stakeholders were consulted together and led through a process of sharing and exploring project outcomes and mechanisms. The basic structure of this process was:

1. Focused questioning to explore principle challenges.

2. Interactive modelling to explore possible explanatory mechanisms.

3. Reflection in the light of models, and repeating the process from 1.

These stages tended to be quite large-scale affairs owing to the number of stakeholders, and the initial step was conducted over the course of an

‘evaluation day’ with most of the project partners gathering to share their ideas. Owing to the large number of participants, and variety of different stakeholder roles, the coordination of this process required technological mediation. For a), a tool was developed for collaborative mind-mapping which allowed many participants to contribute their thoughts and theories in response to a particular question and then to share and reflect and ultimately vote on which question would then follow. To do this, an existing opensource mind-mapping tool was adapted to allow it to receive input from

Twitter (, a popular message sharing service.

The technology afforded agility and capacity to record and organise the views of stakeholders which ‘low-tech’ approaches to collaboration and brain-storming (for example, the use of post-it notes) did not. This capacity to organise allowed for the structured drilling-in on particular emerging themes, which aided consensus-building within the stakeholder group.

The basic design of the use of the technology was:

1. Allow for structured input of opinion and thoughts by stakeholders in a way which captured as much as possible.

2. Enable synchronous participation in the process for those not physically present

3. Allow for coordination and steering of the discussion towards consensus between stakeholders.

4. Through using other mind mapping tools, provide mechanisms for sharing the resulting mind map.

The results of this exercise and details of the technology are described in more detail below.

The second stage of the evaluation at 2 involved the creation of software to bring possible mechanisms ‘to life’ so that all stakeholders could grasp the emerging possibilities for mechanisms that might explain both their and others’ experiences. This software allowed for the creation of dynamic processes which could relate both to the ideas, theories and categories emerging from the mind mapping exercise at 1 and to broader established theoretical constructs from the social sciences. These mechanisms were



1 interactive, allowing participants to explore the results of particular actions. This deeper level of engagement by stakeholders served to lead conversations about causal mechanisms for the project to a deeper and more focused level. For example, where in the mind map exercise, a distinction was made by one stakeholder between different types of people within institutions (“energy creators” — the instigators of innovation, “energy neutrals” — those who are receptive to innovation, “zappers” — those who actively resist, and sometimes sabotage, innovation), using the modeller, more probing questions emerged (“what exactly happens when a zapper is introduced to a new idea?”)

Collaborative mind mapping through Twitter and


The mind-mapping exercise itself was conducted over the course of the

SPLICE ‘evaluation day’ and was used to hone-in on possible mechanisms through an iterative 3-stage process over the course of the day:

1. brain-storming and capturing possible answers to a question;

2. reflecting on results and voting for most effective answers;

3. drilling into chosen issues and repeating the process.

The process was repeated many times during the course of the day, coordinated by a facilitator whose job it was to ensure fair representation of all stakeholder views. Stakeholders submitted ideas and voted through Twitter.

By the end of the day, this exercise resulted in a large mind map whose basic structure can be seen in Figure 2. From this structure the questions that emerged from participant feedback can be identified as the successive roots of previous feedback. The initial starting questions were “How have you changed in your technological habits over the course of the project?” and “How have your institutions changed over the course of the project?”

The day was divided between exploring these two questions. To give an insight into how the technology worked, it is useful to demonstrate how the questioning developed during the day, and how the results of the questioning fed into the deeper identification of mechanisms.

The initial responses to the first questions included positive and negative comments from those present, depending on their experiences. On the positive side, some reported that they had changed through “following other professionals on Twitter” or being “more willing to let students dictate the agenda in the classroom” or by “connecting real life practice with the online environment”. On the negative side, some worried that “technophobes were getting left behind”, or were concerned about an “over-dependence on technology”. After capturing responses to the initial question, all stakeholders reflected on the responses gained and a vote was taken to decide which of these different responses would be pursued at the next

Figure 2: The basic structure of the twitter-generated mind map session

Focussed lines of questioning

Twitter ‘capture’


iteration of the investigation. The top-ranked ‘indicator of personal change’ was the realization that “I became more relaxed about what I put online”.

This was then pursued by repeating the data-gathering exercise and asking about the causes of this ‘increased relaxation’, or indeed what it meant.

Answers to this revolved around the emerging realisation that there was a large community of practice engaged in online social activity, with an increasing awareness that participation in online activity was an indicator of the social capital of an individual (“starting to judge other people by their online exposure”). The top-rated response in this iteration was that increased relaxation in putting things online was due simply to “realizing the value of online engagement”.

This raised the issue of “what is the value and when do you see it?”, since identifying ‘value’ appeared to be the principle cause for engaging with the technology. The iteration under this question produced responses suggesting that value lay in getting feedback and building relationships online. As the questioning progressed, theories were suggested, and at this point, there was an interesting correspondence with Bhaskar’s Transformational Model of Social

Activity where it was discussed whether the emergence of social connectivity online drives the increase in online habit. Such theoretical correspondences led themselves to deeper consideration in the second stage of the evaluation.

Deeper discussion at this point led to the consideration that what was ‘valuable’ was often what was not put online, and the group questioning continued down this path. This raised the question of the distinction between that which is deeply personal and that which people are happy for others to see, and following this, the question of whether the boundary between ‘public’ and

‘private’ life is changing in the light of technology. In turn, the differences between those who are disposed positively towards technology and those who aren’t became the focus of the next iteration.

As the process continued and the territory of the questioning became deeper, the mind map facilitated navigation back to where the questioning had started, and so helped make connections between the deep level (and increasingly theoretical) discussion, and the basic questions that it had begun with. As things progressed, the questioning focused on understanding the ‘relevance’ (as opposed to the ‘value’) of technology, together with the variation in the individual ability to change habits in the light of new developments. These issues of personal difference distilled to the differences between individuals who explored future scenarios in the light of new technological developments, and those who detected threats in technology to personal life. Finally, this led to a focus on the mechanisms whereby individuals organize themselves, with differentiation between those for whom priority was given to ‘future gazing’ and experimentation, and those who sought to remain in touch with embodied human experience and felt the need to ‘protect’ it from technology. This led to a discussion around the fact that the discussion itself was part of what technology does: that whether technology does or doesn’t work; whether users like or dislike it, there is still something to talk about. These discussion further suggested possible mechanisms, with some relevance to different mechanisms of the personal organization suggested by Harré (1984), Beer or Archer (2000).

Exploring possibilities: animated modelling of mechanisms

Stage 1 of the evaluation resulted in rich descriptions of things which happened to people, but did not go so far as to suggest common possibilities for mechanisms which might produce this. All stakeholders had at least some opinions about why things happened (some had more fully workedout theories). These theories and opinions were developed to different levels of sophistication. Thus the task began to identify those theories



1 which would fit the outcomes and mechanisms described by the mind-mapping. However, the risk with taking this approach is that a particular theoretical standpoint can be privileged over the real experience of stakeholders. Within the SPLICE evaluation process, since all theories are effectively descriptions of mechanisms, there was clearly a role for some technology to make the action of mechanisms more apparent and understandable so that all stakeholders could relate their experience to the mechanisms.

Figure 3: Inner State mechanism

A number of options are available for dynamic animated system models, including widely-used tools which are often employed in social simulations (e.g. STELLA (www. or VenSim ( However, in terms of producing rapid graphical representations of dynamic processes these tools can be over-complex. In our evaluation process we developed a simple javabased tool to animate mechanisms in the project called ‘InnerState’. The purpose of InnerState was to unite some of the theoretical models behind the project with the categories and new mechanisms that emerged from stage 1 of the evaluation.

InnerState allows for the construction and description of interactive mechanisms through the combination of a series of components. These include ‘conveyor belts’, ‘transformers’, ‘amplifiers’, ‘drop-points’, ‘collectors’ and ‘generators’. Across these components are passed ‘burdens’ — or

‘things to be processed’. These components may be arranged in any form, configurable through an XML file.

One use-case of InnerState in the evaluation of SPLICE was in the distinction-making between different responses to technology. In figure 3, the mechanism shown is a suggestion for the ways in which different sorts of interventions might be handled by individuals. The categories identified through the first stage of the evaluation were that people could be disrupted in their practice, or they could be coerced into doing something, or a new technological practice could be ‘exhorted’. Building on the organizational model suggested by the VSM, three different levels of personal ‘regulation’ were identified; ‘habits’, ‘organisation’ and ‘future planning’. The model developed showed the relationship between different types of intervention and the different ways in which those interventions would affect ‘habits’,

‘organisation’ and ‘future planning’.

The model was a starting point for a deeper discussion which also drew in the distinction between ‘zappers’, ‘energy creators’ and ‘energy neutrals’ which emerged in stage 1 of the evaluation. Questions arising from this included: “what do you do to a zapper to get them to change?”; “what do you do to an energy creator?”; “what are the problems with energy creators?”, etc. This led to more distinctions which mapped the level of personal regulation regarding ‘habit’, ‘organisation’ and ‘future planning’ with the identity of a person as a ‘zapper’, ‘energy neutral’ or ‘energy creator’: for example, the characterization of a ‘zapper’ as being low in ‘future planning’, but high in habitual responses, or an ‘energy creator’ as high in ‘future planning’ but less high in ‘day-to-day’ habitual responses. Discussions continued in the group at this deeper level.


In this stage of the evaluation process, a number of InnerState mechanism descriptions were produced and made available to the group, including animations of Bhaskar’s Transformation Model of Social Activity, Beer’s

Viable System Model applied to the person, and Harré’s theory of selfhood and positioning. By doing this, we were able to relate concepts reported by project stakeholders with concepts derived from theoretical descriptions, whilst engaging rich discussion with project stakeholders.

Teaching and learning in the evaluation process

The use of InnerState for animating mechanisms took the group discussion of mechanisms to a different level. From talking generally about ‘zappers’ in the institution, a more specific discussion could be had relating to the differences between the ways ‘zappers’ and ‘energy creators’ might be organized (in this case, by referring to Beer’s Viable System Model). As these discussions progressed, the group dynamics were also important. Unlike phenomenological methods of social research, in Realistic Evaluation the researcher is not considered to be neutral. Nevertheless, it was important to ensure that the explanatory power of the mechanisms that evolved satisfied all stakeholders in the process. Thus in place of theoretical neutrality, there was a dynamic of ‘mutual teaching and learning’. Usually, this dynamic was led by the researcher who taught other stakeholders about possible mechanisms using InnerState as a vehicle, those stakeholders situated their experiences, and through the interaction with InnerState taught their own theories confirming or challenging what they understood to be the mechanism the researcher was suggesting. Over time, the mechanism and categories within InnerState were refined.

This teaching and learning model for research is different from those positions advocated in phenomenological or evidence-based research. In the phenomenological case — notably in the popular Grounded Theory method of Glaser and Strauss (1967), the researcher aims to ‘bracket-out’ any initial presuppositions, with theory emerging from the categories identified by the phenomenological process (usually employing questionnaires, text analysis, coding, etc). In evidence-based research, often a particular theoretical model is considered against the evidence gathered from the research. The problem for Pawson and Tilley with phenomenological approaches is that they regard it as unreasonable to try to ensure that researchers have no initial theory as to what causes particular outcomes. Drawing on Bhaskar’s claim that

‘reasons are causes’, Pawson and Tilley argue that the theoretical views of not only researchers but all stakeholders are causal in the phenomena which result, and must therefore be surfaced through the evaluation process.

Furthermore, by engaging in the evaluative process, those theoretical views may change, and thus the evaluation process is itself transformative as well as analytical.

Evidence-based methodological approaches, for Pawson and Tilley, suffer from being overly prescriptive in their specification of ‘evaluation criteria’ at the beginning of a project. Drawing again on Bhaskar, they argue that the act of identifying evaluation criteria is to assert a particular view of the world which may or may not be grounded in reality. Since the purpose of the evaluation process for them is to identify the mechanisms at work in social systems, this imposing of a world-view before the project starts can be both causal in the project results and also not conducive to a critical engagement with the real mechanisms of the project.

The process of gaining agreement through the SPLICE evaluation was itself transformative on the ways in which stakeholders viewed the project, and had some causal effect in changing habits of those who engaged in the evaluation day. This causal effect was partly due to the way in which many different ideas for mechanisms and outcomes could be cumulated



1 and shared. This transformational aspect of the evaluation process is key to the Realistic Evaluation approach, since it is agued that the ‘value’ in the project rests on the efficacy of actions taken as a result of it. As stakeholders become better informed about the mechanisms at work in the project, their actions should become more efficacious as a result. Were this not to be the case, it would simply mean that the mechanisms identified were wrong.

Figure 4 below is an adapted version of a diagram Pawson and Tilley use to explain how different stakeholder perspectives may be cumulated. Basically, the process described is one which relates Abstraction (upwards movement) to Specification (downwards movement). At the ‘specification’ end, individual stakeholder perspectives identify Contexts, Mechanisms and Outcomes.

This results in larger scale general project outcomes which are then used as evidence for the establishment of middle-range theories. Using middlerange theories, new programmes are established. Finally, the methodological approaches used to govern the design of programmes may also be reviewed in the light of developments within projects.

Figure 4: Cumulation of evaluation results on SPLICE (adapted from Pawson and Tilley, 2004)












Middle-range theory

(contexts, mechanisms, outcomes)

Project outcomes






Stakeholder theories

(contexts, mechanisms, outcomes)


In the evaluation process on SPLICE, stakeholders were engaged in a process which took them from the low-end specification stage to middlerange theory, and through this process agreement could be established as to possible mechanisms in the project.

Conclusions: between analysis and transformation and the role of technology in evaluation

This paper has described the evaluation process on the SPLICE project.

In particular, it has focused on the tools used for the evaluation on the project. The construction of these tools was deemed necessary as a way of taking a different approach to evaluation which avoided the pitfalls of phenomenological inquiry and evidence-based evaluation. By adopting

Realistic Evaluation as an approach, we needed to find a way of dealing with descriptions of mechanisms as the principle ‘data’ of the evaluation process.

This has entailed the use of technologies for collaboration and technologies for the sharing, teaching and exploration of possible mechanisms. The question therefore remains: “to what extent does technological advancement affect methodological practice in evaluation?”


In answer to this question, some key distinctions need to be made between methodological practices in evaluation. Traditional evaluation practices tend to be analytical in their treatment of the thing to be evaluated. They treat that thing as something which exists independently of the evaluation process, which is not affected by that process in any way. Realistic Evaluation is transformational as well as analytical. It regards ‘value’ as being inherent in the actions which arise from a better understanding of the mechanisms of a project. If Pawson and Tilley’s arguments about evaluation are correct, then the role of technology in the methodological evolution of evaluation practice is fundamental.

Technology, as well as being (in the case of SPLICE, as with other e-learning projects) the ‘thing evaluated’ is also often the key medium through which

‘things are understood’. Social technology in particular provides a means by which stakeholder perceptions and theories may be surfaced as a project progresses. If ‘value’ is inherent in the actions that arise from a better understanding of what is going on, then the role of the means by which that understanding is gained is absolutely entwined in the process. As we have shown in the SPLICE evaluation, it makes possible processes that lead to greater shared understanding of common mechanisms: a true synthesis of different perspectives and experiences.

‘Synthesis’ is perhaps one of the greatest current challenges facing e-learning development. Funding bodies, both national and international, are awash with lengthy reports from thousands of projects from across the world. Many of these projects are doing similar things, many of them ‘sort of ’ work, and yet establishing any common consensus about ‘what is going on’ seems an elusive goal. Partly, this may be a failure of technology, or at least a failure to address the challenge of describing project outputs in ways which can be assimilated and synthesized into consistent descriptions of mechanisms. Sometimes these evaluative failures result in expensive ‘wrong turnings’ in funding programmes due to poor middle-range theory, or (more often) simply to produce programme-level outputs which avoid higher-level analysis altogether, preferring to cite specific instances of ‘good practice’.

With both of these, the institutional manager asking for specific programmelevel advice “If I do this too, what’s likely to happen?” can often get either an uncertain answer, or (worse) one which glosses over either negative or positive outcomes.

This might lead us, perhaps not unreasonably, to ask “what is the value in doing projects?” However, the knowledge of “what’s likely to happen” exists, but often does not get conveyed to programme-level. It is likely that if that same institutional manager asked their question to one of the key stakeholders involved in a project which has done something similar, the answer might be more realistic: “Well, a few learners will like it, a few won’t and for most it will ‘sort of ’ work. And as for the teachers, you must watch out for your ‘zappers’!” Behind such a reply is an awareness of the real outcomes, contexts, and some idea of a mechanism that lies behind the understanding of the original project. The value for the institutional manager lies in the realism of the advice they are given. Armed with an accurate prediction of what’s likely to happen, they can plan accordingly.

In the SPLICE project, we have investigated ways of identifying the value in the project through developing technologies to identify mechanisms at work. The value of the project lies in the effectiveness of the mechanisms identified. The actual mechanisms of SPLICE will be reported elsewhere, but the process of identifying them, and particularly the role of technology in that process, points to a way of addressing the problems of synthesis and value in programme-level outputs.





Archer, M. (2000) Structure, Agency and The Internal Conversation. Cambridge

Argyris, C. and Schön, D. (1974) Theory in practice: Increasing professional effectiveness, San

Francisco: Jossey-Bass.

Beer, S. (1981) Brain of the Firm Second Edition, John Wiley, London and New York.

Bhaskar, R. (1979) A Possibility of Naturalism Sage

Bhaskar, R. (1975) A realist theory of science Sage

Beer, S. (1994) Beyond Dispute: the invention of team syntegrity Wiley

Campbell, D. T. and Stanley, J. C. (1963) Experimental and Quasi-Experimental Designs for

Research Chicago: Rand McNally.

Checkland (1990) Soft Systems Methodology in Action Wiley

Eden, C. and Ackermann, F. (1989) Strategic options in development and analysis

(SODA) — using a computer to help with the management of strategic vision Knowledge based management support systems

Glaser, B.G. and Strauss, A. (1967) Discovery of Grounded Theory. Strategies for Qualitative

Research. Sociology Press

Harré, R. (1984) Personal Being: A Theory for Individual Psychology Harvard University Press

JISC (2007) Six Steps to Effective Evaluation: a handbook for programme and project managers

[accessed on May 22nd 2009]

Johnson, M. and Liber, O. (2008) The Personal Learning Environment and the Human

Condition: from Theory to Teaching Practice Interactive Learning Environments, v6, no. 1

Luhmann, N. (1995) Social Systems Stanford University Press

Mingers J. (2004) Realising Systems Thinking: Knowledge and Action in Management Science


Pawson, R. and Tilley, N. (2004) Realistic Evaluation Sage



John Dickinson

Newcastle Business School, University of Northumbria, City Campus East,

Newcastle upon Tyne NE1 8ST [email protected]


Modern day mass higher education presents challenges for both learners and teachers. Whilst digital resources, Web

2.0 technologies and online connectivity can add signiicantly to the learning opportunities of 21st century students, many cross programme Virtual Learning

Environment (VLE) provisions remain collections of somewhat disconnected and basic materials.

In its early development, organised e-learning has generally been the province of specialist programmes and individuals championing the new technologies. However along with the adoption of technology enhanced learning, there is a growing need to develop, design and embed more fundamental and far reaching strategic approaches that embrace the core of traditional university learning and teaching programmes.

This paper focuses upon a large undergraduate core module and discusses the implications of a practice based case study which explored how traditional campus based undergraduate learning and teaching could be redesigned and enhanced by the addition of online technology and e-pedagogy. It considers aspects of both pedagogical and technological design and examines how a VLE can support learners and teaching teams. Findings showed that students were extremely positive about the mix of onsite and online learning.

They saw anytime, anywhere access as fundamental, and valued the lexible access and collaborative opportunities offered by Web 2.0 and mobile resources.

Considerable operational beneits arose in supporting teaching teams and student marks increased. Drawing upon lessons learned from practice, and feedback gained from students and teachers, the paper examines how the approach may inform future curriculum delivery and programme speciication. It considers the contribution that blended learning may make in addressing the needs of 21st century learners in mass higher education, and relects on the implications of the case study in terms of aspects of design, practice and strategy.

0031 Technology enhanced learning in 21st century mass higher education. Aspects of design, practice and strategy for a necessary step change


Taking a case study approach, this paper looks to explore the contribution that blended learning can make to 21st century mass higher education. It uses the findings of a practice based case study to examine how traditional on-campus learning and teaching might be redesigned to embrace online technologies and e-pedagogy. It sets out to identify good practice that may enhance the learning and teaching experience of both students and teachers, and inform future programme and organisational development.

The case study focused on a large, business school based, undergraduate core module, and the adaptation of its traditional learning and teaching design to incorporate a mixture of online and onsite resources and activities.

Module pedagogy was underpinned by the principles of constructive alignment (Biggs 2003), and used a framework consistent with Mayes

Conceptualisation Cycle (Mayes & Fowler 1999) to integrate on-site and on-line elements across lectures, seminars, directed study, assessment and feedback.

The work was evaluated by collecting data using online questionnaires, focus groups, programme committee and module team meetings. The paper will provide an account of the methods and e-pedagogy employed in the case study module and then discuss and evaluate the blended practice in terms of its impact on learning and teaching. Within the analysis and conclusions will be a consideration of the contribution blended design and delivery might make towards addressing some of the current issues that studies have identified within modern day mass higher education.


Higher education has been in a period of considerable change since the early

1990’s. There has been a transformation in the way universities are financed and organisational objectives have shifted to focus on the generation of new income streams and increasing accessibility and participation. The resulting growth in mass education and international markets has generated the need to review the infrastructures and methods that support teaching and learning. Modern day mass education presents challenges for both learners and teachers. “The coming of mass higher education has brought larger classes, more diverse students and leaner unit costs, but keener interest in teaching quality and graduate attributes” (Entwistle, Hounsell et al. 2007, 1).

In parallel to such change we have seen the unprecedented growth in information technologies and the development of a whole new communication media and culture. Web 2.0 technologies have impacted fundamentally on the way we communicate and are changing the expectations of learners. Whilst universities have invested heavily in technology, there remains the need to adapt pedagogical approaches to



1 make the best use of this new infrastructure. Despite the opportunities and expectations it can remain difficult to change well established traditional methods. Laurillard (2002, 3) states “Higher Education cannot change easily.

Traditions, values, infrastructure all create the conditions for a natural inertia.” The recent Benchmarking and Pathfinder Programme has provided a valuable opportunity for institutions to take stock of their own e-learning provisions and practices. However it is necessary that they now translate this into further action for self-improvement and evaluation (JISC Benchmarking and Pathfinder Programme 2008).

In his report The Future of Higher Education Teaching and the Student

Experience, Ramsden (2008) identifies ICT as a key contributor to evolving expectations. However whilst digital resources, Web 2.0 technologies and online connectivity can add significantly to the learning opportunities of 21st century students, key messages about e-learning from the sector indicate many cross programme VLE provisions remain collections of somewhat disconnected and basic materials (Adamson and Plenderleith 2008). In her interview at the JISC Innovation Forum 2008, Sarah Porter endorsed the view of HEFCE Director John Selby, that the most important challenge now facing higher education was getting the work of experts and developers to the sector as a whole (Porter 2008). Whilst many institutions can point to examples of excellent practice, the quality of cross programme e-learning is often very inconsistent. More generally there is a need for the development of programme wide e-learning specifications designed to assist in the alignment of programme learning objectives and the student journey.

Whilst there are many theoretical models, those offering ‘learning as guided construction’ probably best fit with current scientific ideas about learning.

Guided construction gives the student an active part in their learning. It also gives an important role to external guidance, whether from a teacher, online resources or other learners. (EDNER Project Paper 1 2002)

Within the field of e-learning we can identify three significant models of learning. These are ‘Mayes Conceptualisation Cycle’, ‘Laurillard’s

Conversational Model’ and ‘Salmon’s Five Stage Model’. These models all put high value on active student learning and are in the ‘constructivist’ mode.

The concept of constructive alignment has been one of the most influential in recent higher education learning and teaching theory and practice.

It is important when developing e-pedagogy to focus not only on developing materials but also on the learning activities that will assist students’ learning and meeting the learning outcomes of the course. This is particularly the case when supporting student centred directed learning. “Learning and teaching in higher education is shifting towards a more student-centred model, in which the learner’s cognitive activity is acknowledged to be much more important than teachers’ historic pre-occupations about syllabus coverage. Educational technology development projects need to take this into account.” (EDNER Project Paper 3 2002, 1).

Mayes Conceptualisation Cycle (Mayes and Fowler 1999) stated that learning with technology involved a three stage cycle of; conceptualisation, where students are exposed to other people’s ideas or concepts; construction, where students apply these new concepts in the performance of meaningful tasks; and dialogue, where new concepts are tested during conversation with tutors and peers. Mayes stresses that focusing too much on primary stage ‘courseware’ will not provide sufficient support for learning. In order to ensure that learners are supported at all three levels of the conceptualisation cycle, a variety of teaching methods need to be within the course design. High level learning will not take place until there is two-way dialogue. This can only take place at the tertiary level — either using courseware or face-to-face methods of learning which are integrated with technology enhanced teaching.


The ESRC funded Enhancing Teaching and Learning (ETL) Project has enabled research into contemporary teaching and learning environments.

The project aims were to investigate what makes for effective teaching and learning in contemporary higher education and to use the findings to help bring about improvements in students’ learning (Hounsell et al 2005). The challenge was to respond to the impact of such issues as larger classes, increasing student diversity and leaner unit costs, whilst recognizing the need for greater teaching-learning quality and accountability. One element within this mix was the impact of information and communication technologies on the learning and teaching process, supporting both the provision of learning resources and communication between university teachers and their students. The project employed a set of key concepts which included the learning and teaching environment, constructive alignment, subject based thinking and practice and threshold concepts.

Alignment is seen as a cornerstone within the design of effective learning and teaching systems. In his model of constructive alignment Biggs (2003) applies a systems approach to learning and teaching, with the elements of the system all working towards the achievement of clearly defined objectives through the construction of learning. In this view the effectiveness of the system in achieving the learning outcomes will depend upon the alignment, or goodness of fit, of both the teaching and learning activities and the assessment methods. The implication is that such systems need to be deliberately planned and designed around these elements.

Whilst this representation of Biggs’ model provides a solid conceptual framework, it may be argued that it is an oversimplification, and that the greater complexities embodied in the wider teaching and learning environment cannot be ignored. Hounsell & Hounsell (2007, 94) observe “what is also evident from surveying the burgeoning literature on teaching-learning environments is the sheer breadth of potentially relevant contextual factors, embracing not only departmental, subject and institutional influences but also wider social, cultural and political ones”. McCune & Hounsell (2005) extended the scope of their study beyond teaching, learning and assessment activities to include the variety of resource with which teaching staff may interact. These encompassed curriculum aims, scope and structure; teaching-learning activities and learning support; assessment, guidance and feedback; course organisation and management; and students’ backgrounds, knowledge and aspirations.

Given this much wider scope they used the term ‘congruence’ rather than alignment to explain the degree of harmony achieved between high quality learning outcomes and the elements and strategies employed.

Hounsell et al (2005) found there were issues in the level of engagement with undergraduate students. They also identified differences between teaching and learning environments across first and final years of study. Final year modules tended to offer more student choice and have more varied approaches to teaching and assessment. They also had smaller class sizes and ‘better’ resources. Perhaps this is not altogether surprising given the extra demands put on first year modules in terms of scale, student diversity and unit costs.

Deep and surface learning approaches are often seen as ways of measuring student engagement and the quality of learning. The ETL Project identified

‘organised effort’, how students organise their study and whether they use their time effectively, as important in enabling students to achieve deeper learning. Scores for deep approach and organised effort were higher, and scores for surface approach lower, in final year courses. Another factor influencing this was the compulsory nature of core modules and the diversity of the student body. Issues such as perceived subject relevance and prior knowledge can effect student motivation and engagement.




In general the dynamics of learning and teaching will vary across different disciplines and settings. Hounsell et al (2005) found that the learning approaches supporting deep learning were markedly different across the four subject areas studied. The study by Reimann (2004) of economics learning and teaching environments found a number of issues. These included tensions between, students with and without prior subject knowledge; students intending to major in economics and students taking it only as a compulsory foundation module; and gearing the curriculum to the majority leaving minority groups overstretched or under-challenged. There were also concerns around disciplinary norms and student diversity, and difficulties interpreting assessment tasks such as exams and assignments. Many of these concerns can also be identified in finance and accounting learning environments.

As evidenced by the National Student Survey assessment and feedback is a common area of concern for students. The ETL Project found in terms of feedback there was variability in students’ experiences across the subjects and course settings. A wide range of concerns expressed uncertainty about expectations for set work, dissatisfaction with the variable quantity and helpfulness of feedback comments from staff, and frustration with delays in receiving feedback. (Entwistle, Hounsell et al. 2007).

This research will explore further the issues of organisational change, pedagogical practice and contemporary learning and teaching environments.


The work examined in this paper was undertaken within the author’s own institution and as part of his own practice. The development of the blended e-learning pedagogy involved an ongoing process of diagnosis, planning, action and evaluation. This took place over a three year period with the broad objective of identifying effective learning and teaching strategies through the use of technology. The strategy adopted had similarities to action based research and was underpinned by a philosophy embracing a mixture of pragmatism and interpretivism. Given the practitioner based method within an academic setting, the approach may also be seen as falling within the realm of applied practice-based educational research as defined by Furlong and Oancea (2005).

The practice based nature of the work lent itself to a case study strategy which was at the centre of the research method. Data was collected using a mixed method, making use of a questionnaire to gather quantitative data, and focus groups to gather qualitative data. Whilst the main time horizon of the questionnaire was cross-sectional, the focus group feedback was received at various stages over the three year period of developing practice. The data collected was subjective in so far as it focused on the values and opinions of students and teachers. This led to an interpretative and inductive approach in the analysis and evaluation of the findings.

Whist there are limitations associated with a single case study strategy it was justifiable in this instance because it provided an opportunity to investigate new and developing practice. It was also typical, in the sense that the case study exhibited many of the more general issues found in contemporary large module teaching and learning in mass education.

The questionnaire was constructed to collect data from students regarding the module pedagogy. It was decided to build the questionnaire in the form of an online survey made available at the year end via the module site on the VLE. The questionnaire was released immediately following the close of teaching in a three week window between completion of semester two delivery and the year end examination. This was a time likely to gather the maximum response as most students would be using the VLE to support their revision. The survey was kept relatively short and it was made clear all responses were anonymous.


In addition to the questionnaire, qualitative feedback was obtained from both students and teachers via programme committee meetings, module team meetings, focus groups and discussion sessions within module delivery.


The case study focused on a business school core module that provided common input in financial accounting for 600 first year undergraduate students from a variety of programmes, including business studies, marketing, tourism, strategy and human resource management. It was year long and delivered over 24 weeks. There were three mass lecture cohorts of 200 students each, and approximately 30 seminar groups containing 20 students. Contact time for an individual student was 36 hours a year, made up of a 1 hour weekly lecture and a 1 hour fortnightly seminar.

The teaching team consisted of eight academic staff from the accounting and finance subject area. In addition to full time, the module was offered part time and franchised overseas. However the study data did not include franchise partners.

The VLE was fully integrated into module learning and teaching. It was presented as one of five key learning resources (Figure 1).

Within the on-site and on-line design a conscious effort was made to align curriculum objectives and intended learning outcomes to learning and teaching activities and assessment tasks. Using a framework consistent with Mayes Conceptualisation Cycle, the learning activities embodied in lectures, directed study, seminars and online learning were brought together into a ‘blended learning cycle’. Lectures were followed by active student learning through on-line directed study activities. This directed study was at the heart of the learning process. Teacher student collaboration and

Figure 1: Blended learning pedagogy — onsite and online learning and teaching


Weekly topics

Workshop activities

VLE resources


Fortnightly activities

Directed study review

Feedback and discussion



Study information

Materials and activities

Communication and collaboration

Assessment and feedback


Selected hard copy key notes

Workshop proforma

Directed Study

Weekly learning activities

Discussion forum support


1 feedback was then provided in the follow up seminars over a fortnightly cycle. For diagnostic and formative purposes, suggested answers were also made available online each week and students were expected to assess their progress by comparing and amending their own work. This formative feedback mechanism focused on increasing large group formative assessment within a manageable overall workload.

In addition to directed study activities, students were expected to use the e-materials and media presentations to follow up lecture topics, and reflect upon and consolidate their learning. Students were also required to complete interactive online computer assisted learning modules.

Support was also provided for formative and summative assessment. The mid year assessment was a multiple choice exam and this was aligned to a series of online formative progress tests with feedback during the first semester. Questions in the final exam align to semester two directed study and seminar activities. Preparation for this assessment was supported by guidance sessions and materials that explored previous year questions and marking criteria.

The questionnaire was completed by 201 students. This was approximately

40% of those studying the module in full time mode.

The students were generally supportive of online learning. 77% of students agreed or strongly agreed that the VLE had helped them study the module, whilst only 12% disagreed.

For the blended pedagogy to be effective, students needed to access the VLE regularly and preferably on a weekly basis. Survey findings showed 79% of students visited weekly or more frequently, 16% at least every two to three weeks and only 5% less than that. It was interesting to see that only 2% of students stated that they did not find the VLE easy to use. This did not necessarily mean the student fully understood the best ways to use the VLE to support their study. This was why induction and tutor led example was so important. A clear distinction needed be made between operating the VLE and managing the module e-pedagogy.

The robustness of the technology was fundamental to the success of online practice. 13% of students reported having experienced technical difficulties.

Whilst this was encouraging it should be noted that, from a teaching perspective, a number of staff did feel that the platform was not always reliable.

Students were asked to rank the relative usefulness of lectures, seminars, the VLE and workbook. Whilst all four areas scored 19% or greater, it was apparent that students put the greatest value on seminars (30%) and the workbook (33%). This was encouraging as seminars played a key role in support and feedback for active learning activities. On the first level of ranking the VLE and lectures seemed equal, however third and fourth level rankings showed lectures were the lowest rated elements. Some tutors felt this supported concerns that the blended approach undermined lectures leading to poor attendance.

In terms of student usage of the functional areas of the VLE, the content area was used the most with a score of 88%, followed by information 79% and assessment 71%. The least used area was communication at 45%. This pattern was reinforced by the student’s ‘usefulness’ ratings, where content was a clear winner and discussion / communication the loser. These responses may have reflected the blended design as development had initially focused on content, information and assessment. Directed study was managed through a ‘directed study forum’ and students were encouraged to use this forum to participate in peer support. However student contributions were minimal and this may be a characteristic of campus based environments. It is recognised that for students who are based on campus, a major part of their learning comes from the everyday face-to-face social interactions (JISC InfoNet 2005).


Students were asked directly if they felt online support affected their attendance. 67% said not, and a further 7.5% stated that their attendance actually improved — possibly through increased involvement and better understanding of the overall pedagogy? Nevertheless 23% did feel they attended less. 10% felt they attended fewer lectures, and more questionably

6% attended fewer seminars. Whilst this aspect can be viewed as disappointing it may underlie enhanced learning opportunities, as students felt that they benefited from having a greater choice as to how and when they studied.

Directed study was a vital ingredient in terms of active learning and the module’s e-pedagogy. Effective feedback and reinforcement of learning through seminars was dependent upon this. It was encouraging that only

12% of respondents did not feel that the VLE supported there directed study well. This would indicate the blended learning process was successful overall. However of the remaining 88%, 26% were neutral in their response, implying there was still room for improvement in terms of the timely completion of directed study and seminar attendance. Assumptions can be wrongly made that first year students understand how to study and use the available resources. The teaching team believed it important to allocate contact time within the study programme to explain to students how best to use the resources and study routines.

Students were invited to attach up to twenty qualitative attributes to their blended learning experience. None of these questions were mutually exclusive and students were free to pick as many or as few as they chose.

Students were most positive about access, independence, time saving and convenience. It is clear students appreciated anytime anywhere accessibility.

There was also evidence of pedagogical awareness with students flagging enhanced learning, being in control, motivation, explorative and active learning. Whilst many of the negatives got low responses, they were still chosen by a few students who felt it time consuming or isolating.

Students were asked if they felt the need for online support was greater on large core modules in mass lecture mode. More than half said yes. It may be the case that students engage less through sheer size and anonymity, or lack of identity with subjects outside their main focus.

With regard to the balance of online and traditional teaching elements, 61% felt the balance was about right and 24% wanted more online learning. 14% wanted more contact time with tutors. The initial conclusion seemed to be that the overall balance was about right. However students had limited direct comparisons available and the module was more developed than others in their programme.

Feedback from teaching staff and students recognised the valuable role that the VLE could play in supporting the large teaching team and module delivery. Tutors thought the shared online resources offered significant benefits in co-ordinating and aligning their teaching. They felt the blended provision resulted in better organisation and forward planning. They found it helpful that teaching materials could be shared in advance, and then made available to students as required.


From the case study practice and feedback gained from learners and teachers, it was possible to identify aspects of design and practice that were seen as supporting contemporary learning and teaching environments.

The blended approach required the full integration of on-line and classroom environments. It was important that students understood the role of technology in their learning and the implications for their study strategies



1 and engagement in learning activities (Sharpe et al 2006). This was explained in the opening lectures. Students that did not understand their role as learners and the interrelationship between lectures, directed study and seminars, were generally more disorganised and ill prepared . Conversely students that followed the study routines were usually well organised, completed directed study work and were prepared for seminars. Ongoing integration was assisted by accessing online materials in class, previewing directed study activities and related online material. Teaching sessions were started with a brief reference to the online study plan to focus the topic objectives and learning resources. These techniques enabled students to remain familiar with the VLE and its functionality.

It was clear that issues of site design were important. Students found many

VLE site structures within the wider provision confusing. The same type of materials could be in different places and materials difficult to locate. In response to this, the case study site design kept to a standardised structure with clearly labelled menus to assist navigation. Sections for materials, directed study and assessment were laid out to reflect the study plan, with course links to connect related learning elements. Consistent design was important when building across programme wide provisions and the student journey.

Prior to the blended approach the case study module had demonstrated many of the issues associated with large and diverse student groups that had been identified by the ETL Project. Entwistle et al (2007, 2) had stated,

“the issues challenging first year courses included the risks of impersonality in large classes, inconsistencies between tutors where course teams were also large and diverse, and curricula that, while well suited to a majority of students taking a unit, could disadvantage or demote others with different aspirations or depth of background knowledge in the subject”. It was interesting to consider to what extent these issues had been addressed by the technology supported delivery employed in the case study.

It may well be that, under a mass education model, first year modules will be larger and more diverse, and congruence more problematic than in later years. It is also likely that student motivation and engagement will be at its greatest in the final year of study. However it is important to ensure that resources are equally rich at all stages of learning, and technology can offer a wealth of valuable support and opportunity in this direction.

It is necessary to understand and respond to the large class numbers that are now a feature of mass education. Trow (1973) recognised that a large scale shift from elite to mass education would have a significant qualitative and well as quantitative impact on university learning and teaching. The blended approach offered a framework of shared learning resources and activities within the VLE which was supported by online guidance. This significantly reduced inconsistencies in delivery across the large course team and improved the student experience. Within the case study module both teachers and students reported operational benefits in terms of coordinating the teaching team and delivery. The communal module site allowed seminar tutors to stay abreast of the lectures and directed study, align seminar delivery and share teaching materials across the team. Answers could be made available for teaching purposes prior to being released automatically to students.

Technology and e-learning helped engagement by involving the students more. The use of collaboration and social networking tools reduced impersonalisation allowing students ready access to tutors and supporting peer group contact and activities. Overall students saw anytime, anywhere access as fundamental, and valued the flexible access and collaborative opportunities offered by Web 2.0 and mobile resources.

One of the most valuable aspects of the blended pedagogy was its contributions towards ‘organised effort’. This supported and developed the


student’s ability to manage their studies effectively, and contributed to them achieving deeper learning. The VLE provided an excellent medium in which to present the study programme and learning resources in an organised format.

The student journey was supported using teaching and learning plans, study guides and appropriately placed links to the key study support resources such as online databases, library search tools and electronic journals.

Whilst the target was to enable students to be deep learners, a significant proportion of surface learning did take place across the first year case study module. Factors influencing this included the compulsory nature of the core module and the diversity of the student body. This meant that some students had prior subject knowledge, whilst others were studying with low interest as their main business specialism was elsewhere. Electronic resources played a valuable role here offering avenues of additional support for weaker students and further study for stronger students. This was achieved in part by the use of online and mobile media, podcasts, and RSS feeds. These provided targeted talks by tutors and subject experts. Students liked these resources, which they felt helped them engage with topics and reinforce key concepts. When designing the study programme there appeared to be benefits in aligning learning and teaching activities and learning outcomes with the programme outcomes and level, rather than being overly driven by discipline and profession based norms. This helped students understand the wider relevance of the study topics and the related threshold concepts within the subject.

There were many variables and it was difficult to measure in quantitative terms how successful the e-pedagogy was in promoting student learning.

However the module mean scores did increase from 52% to 58% during the period the pedagogy was developed between 2004 and 2008. They were no other changes in curriculum and learning outcomes during this period that might have influenced this change.


The second half of the twentieth century has seen a remarkable expansion in student enrolments in higher education. The proportion of 18–21 year old undergraduates has tripled since the 1960s with over two million students studying at UK higher education institutions.

As pedagogy develops and higher education institutions invest in learning technology so expectations increase for traditional learning and teaching systems to adapt and change. Initiatives such as the JISC Benchmarking and Pathfinder Programme have made a valuable contribution to the sector’s understanding of the current role and potential of technology in learning. However, many students still experience wide variations and inconsistencies in the way technology is used across their programmes to support their learning. In practice a significant core of university teaching relates to existing programmes, designed for delivery using traditional teaching methods, within environments and communities where students have a physical as well as a virtual presence. Such programmes put different demands on e-learning, and there is a need to establish standards in design and practice which support the growing number of digitally native students.

By examining the blended learning practice within the case study this paper has set out to identify and discuss the contribution that technology enabled online learning and e-pedagogy can make to the enhancement of student learning in contemporary learning and teaching environments in mass education. Whilst the pedagogy described may be limited by the characteristics of the case study group, it was felt that many of the approaches were sufficiently generic as to be of value to wider practice and design. It is believed that there is strong evidence that the approach can provide added value to the learning and teaching experience. Whilst it



1 is the author’s view that blended learning can enhance all types of campus based teaching, it is suggested that such approaches can make a particularly important contribution in meeting the challenges posed by the large scale undergraduate programmes which prevail in mass higher education.

This work is currently contributing to the enhancement of school learning and teaching strategy. It has led to the establishment of school wide minimum standards. Aspects of the design and practice are also informing a blended learning framework addressing programme outcomes, differential study level strategies, group size and the student journey.


Adamson, V. and Plenderleith, J. (2008) Higher Education Academy/JISC e-Learning

Benchmarking Exercise Phase 2 Review. Higher Education Academy

Biggs, J. (2003) Teaching for quality learning at university: what the student does. Buckingham:

Open University Press. 2nd Ed

EDNER (Formative Evaluation of the Distributed National Electronic Resource) Project

(2002) How students learn: ways of thinking about good learning in HE. Issues Paper 1

EDNER (Formative Evaluation of the Distributed National Electronic Resource)

Project (2002) Changing conceptions of teaching in UK HE: some implications for DNER projects.

Issues Paper 3

Entwistle, N., Hounsell, D., Anderson, C., Beaty, L., Bromage, A., Cousin, G., Day, K.,

Hounsell, J., Land, R., Litjens, J., McCune, V., Meyer, E., Nisbet, J. and Reinmann,

N. (2007) Learning and teaching at university: the influence of subjects and settings — TLRP

Research Briefing 31. London: TLRP [Online]. Available at retrieve/2951/Hounsell%2520RB%252031%2520FINAL%5B1%5D.pdf [accessed Feb 2009]

Hounsell, D. and Hounsell, J. (2007) Teaching-learning environments in contemporary mass

higher education. In Entwistle, N.J., et al. (eds) Student Learning and University Teaching.

(Psychological Aspects of Education — Current Trends. British Journal of Educational Psychology Monograph Series II ) British Psychological Society, Leicester: 91–111

Hounsell, D., Entwistle, N., Meyer, E., Beaty, E., Tait, H., Anderson, C., Day, K. and

Land, R. (2005) Enhancing Teaching-Learning Environments in Undergraduate Courses. London:


JISC Benchmarking and Pathfinder Programme (2008) Challenges and Realisations from the

Higher Education Academy/JISC Benchmarking and Pathfinder Programme. Higher Education


JISC InfoNet (2005) Effective Use of VLEs.

Laurillard, D. (2002) Rethinking university teaching: a conversational framework for the effective

use of learning technologies. 2nd ed. London: Routledge Falmer

McCune, V. and Hounsell, D. (2005) ‘The development of students’ ways of thinking and

practising in three final-year biology courses’. Higher Education, 49 (3): 255–289

Mayes T. J. and Fowler C. J. (1999) Learning Technology and Usability: A Framework for

Understanding Courseware. Interacting with Computers

Porter, S. (2008, Dec 16) Podcast: The importance of innovation to the future of higher

education. JISC Podcasts. Available at [accessed on

December 20th 2008]

Ramsden, P. (2008) The Future of Higher Education Teaching and the Student Experience.

Department of Innovation, Universities and Skills. Available at education/shape_and_structure/he_debate/~/media/publications/T/teaching_and_ student_experience_131008 [accessed February 2009]

Reimann, N. (2004) First-year teaching-learning environments in economics. International

Review of Economics Education, 2 (1): 9–38

Sharpe, R., Benfield, G., Roberts, G. and Francis, R. (2006) The undergraduate experience

of blended e-learning: a review of UK literature and practice. Higher Education Academy

Trow, M. (1973) Problems in the Transition from Elite to Mass Higher Education. (Reprint of a OECD Conference paper on mass higher education in Paris in June 1973). Berkeley,

California: Carnegie Commission



Sue Rodway-Dyer, Elisabeth Dunne,

Matthew Newcombe

University of Exeter, Devonshire House,

Stocker Rd, Exeter EX4 4PZ

[email protected],

[email protected]


Feedback has been highlighted as the most powerful inluence on student achievement, but students are often less satisied with feedback than with other aspects of the student experience. It is hence important that ways of offering feedback are found that are useful both for improving learning and for gaining student satisfaction. This ongoing study was designed to explore and to improve feedback in a variety of differing contexts, two of which are reported here: i) audio feedback on a irst year undergraduate written assignment in

Geography (product-oriented feedback); and ii) video feedback from ongoing laboratory sessions with irst-year

Biosciences students (process-oriented feedback). These contexts have been selected as offering different ways of working and for highlighting a number of issues and areas for further development.

Student and staff views have been gained via surveys, focus groups, individual interviews and ‘stimulated recall’ sessions. Findings suggest that students have high expectations in relation to feedback; many anticipate the kinds of individual face-to-face interaction they experienced in school and are not easily satisied by other ways of working.

In addition, offering audio or video feedback that is supportive to learning in both affective and cognitive terms is not necessarily easy. In the context of written assignments there is still much to be learned about appropriateness of length, tone, the register of language, the balance between praise and criticism, and the best contexts and timing for audio feedback. In the context of large classes for laboratory sessions, further research is needed on how lecturers and demonstrators can give ongoing feedback that is useful when captured for replay in video form, and also about how effective video taken in class might be then used for training purposes in order to enable student demonstrators to be more effective and knowledgeable when offering feedback to students.

0207 Audio and screen visual feedback to support student learning


There has recently been considerable interest in using audio for feedback in

UK higher education (see, for example, Rotherham, 2008). Assessment and in particular feedback, are generally considered key factors in supporting the student learning experience and in gaining student satisfaction, and feedback has been highlighted as the most powerful influence on student achievement

(Hattie, 1987; Black & Wiliam, 1998). Yet the UK National Student Survey has outlined over several years that this is an area in which students are often least satisfied. It is hence important that ways of offering feedback are found that are useful both for improving learning and for gaining student satisfaction.

In 1992, Ramsden suggested that technology is changing the nature of university teaching, but time has shown that this does not necessarily happen easily and that technology cannot, of itself, promote enhanced learning.

George (2002) considers it ‘an enabler, not a solution’ and McGettrick et al (2004) believe that e-learning remains one of the ‘grand challenges’ for education. Laurillard (2002) helpfully argues that any study of new approaches to technology should fit firmly within sound pedagogic principles and practices, and Stiles (undated) suggests that no sustainable change will happen unless traditional pedagogy is adapted for more active approaches to learning.

In the context of audio feedback, rhetoric abounds: how much better to have ‘the tone of voice, emphasis on particular words, the effective use of pauses, and the warmth of an encouraging tone when critical comments need to be made’ (Race, 2008). The new interest in aural feedback has led to a number of small-scale practitioner studies, and they do suggest that intonation counts; also that digital feedback suits today’s student (Denton et al, 2007; Bridge and Appleyard, 2007); that video feedback is preferred; that immediate spoken observations on students’ practical sessions can serve as useful feedback (Epstein et al, 2002); that aural feedback tends to be more extensive, easier to access and understand, and with more depth (Merry et al, 2007; Gomez, 2008; Rotherham, 2008); and that it enables students to address their overall learning development (Ribchester et al, 2007).

Nortcliffe and Middleton (2007) describe an analogue-recorded feedback study wherein audio impacted on self-reflection and action, was preferred by students, and was less stressful and time consuming for staff; yet their most recent digital work warns that audio feedback does not necessarily support achievement; and Irons (2008) argues that using technology for formative feedback ‘is not a cheap or easy option’.

Aims and research design

The research described in this paper emanates from a one year, ongoing,

Higher Education Authority funded project exploring the use of existing and emerging technologies to improve feedback and to promote a feedforward culture wherein students listen and pay attention to feedback so as to improve their performance. The project captured digital audio and screen visual feedback within a number of contexts in three subject



1 areas — biosciences, geography, and medicine. National and institutionalbased surveys all suggested a certain level of dissatisfaction with feedback in these subject areas, especially in relation to other areas of University provision. In addition, these three subject areas offered a range of different contexts in which feedback is important to student learning, from feedback on student products such as written assignments and presentations, to feedback on ongoing processes of learning, such as during laboratory sessions or practical fieldwork, or to promote reflective professional development on learning in the workplace (as in a hospital ward).

The overall aims of the project are as follows:

To use existing and emerging technologies to improve feedback between tutor and student.

To refine understanding of the impact of technology-enhanced feedback methods on staff and students in order to inform future practice.

To encourage academics to respond to key factors in effective feedback in order to promote a culture of ‘feed-forward’ and engagement in feedback by students.

To test out specific research methodologies, such as ‘stimulated recall’.

To provide a collection of resources and items for dissemination that can inform research and practice both locally and within the sector more broadly.

The research methodology in each selected context is slightly different, dictated in part by the constraints of the one year duration of the project, but also specifically due to the differences between contexts and the ways of working of academic staff involved. It was deemed important to the success of the project that each subject area should be supported in gaining data that would be useful to that context and that would enhance what staff want to know and to achieve. Overall data-collection includes student surveys, focus groups, audio and video data, and individual interviews and ‘stimulated recall’ sessions with academic staff.

This paper provides insights into two of the contexts studied, highlighting differences in ways of working and in the nature of outcomes and recommendations for future practice:

1. Audio feedback on a written assignment offered to a sample of 73 firstyear geography undergraduates studying a first-semester introductory module on earth system science (product-oriented feedback).

2. Video providing ongoing feedback from laboratory sessions and made available to 180 first-year Biosciences students (process-oriented feedback).

In geography, hearsay evidence suggested that students do not give consideration to written feedback on assignments, do not carefully read points made, and do not use it for developing their learning. It was anticipated that audio feedback might be more detailed and helpful to learning. In Biosciences, staff similarly suggested that students in laboratory settings do not pay detailed attention to the extensive verbal feedback they gain in this context, and that laboratory feedback is not necessarily remembered or heeded. In particular, academic staff suggested that students lack awareness of when they are receiving feedback, especially because it is transitory and not captured, and that video feedback might support students in recognising the value of class feedback.

Each of these contexts is outlined, in turn, below.


Audio feedback in geography

The context

In geography, there was an interest in using and evaluating audio feedback for written assignments in order to highlight and help students to become aware of the relationship of feedback to future assessed work (i.e. feedback as feed-forward). Each student was required to submit a 1500-word written assignment at the end of the fourth week of degree study, having been given detailed guidelines on assignment writing and on the assessment criteria.

Detailed audio feedback was offered via MP3 file, alongside a written feedback form with a grade and brief summary, and short comments were also written on the actual piece of work. Feedback was offered via MP3 audio files. All feedback followed the same format, with the mark for the student being given first along with its relationship to the grade criteria and descriptions; this was followed by general positive feedback comments and then a detailed analysis of the essay with constructive criticism on where it was not so good and exactly what was needed to improve. At the end of each file, a general summary comment was provided. The accompanying written feedback sheet contained a grade, space to comment on three good and three weaker aspects of the essay, and a space for the student to later write about one thing that they have done to improve as a result of the essay feedback (to encourage the concept of ‘feed-forward’).

After both feedback and assignments had been returned, a short, paperbased, retrospective questionnaire was used to gain quantitative and qualitative data on student views of the process. Two focus group sessions — one with a physical geography group and one with human geographers — allowed for more in-depth discussion about assessment in general. Informal individual interviews with all focus group students enabled deeper discussion. Six months later, all students were asked (via email questions) to reflect back on the experience of gaining audio feedback.

Questionnaire results

Survey results (with a return rate of 71%) highlighted student views.

The majority of students listened at least once, most students listened to it twice and some up to four or five times.

In comparing audio with the written feedback, the majority of students considered both audio and written feedback to be either useful or very useful (82% and 84% respectively), although about 20% did not find one or the other, or both, to be helpful.

The main advantage of audio feedback was considered to be the greater detail and depth (54%), and also that it was clearer and easier to understand. Perceived disadvantages focused in the main on difficulty in finding the point in the assignment to which the feedback related. Only one student reported difficulties with the technology.

In contrast to the suggestion that students like the ‘friendly tone of voice’

(Race, 2008), some students found it a more negative experience, and were not always attracted to the tone.

Very few students thought it was an advantage to have an audio format because it was easy to listen to, easy to pause, or easy to access on their computer in future.

Equally few thought that it could be misheard, or that it would be difficult to listen to regularly, could be deleted by accident, or that it would be difficult to listen to regularly.

Over half the sample considered the main advantages of feedback written onto their assignments to be that it related to specific parts of the essay, as well as being easy to refer back to. However, one of the main problems with written feedback was in the legibility of handwriting (20%).

The majority of students thought they would achieve ‘somewhat better’




(76%) and 14% ‘much better’ as an outcome of their feedback. Ten percent suggested that the feedback would not have an impact on their future performance.

76% of students wanted face to face feedback from a tutor in addition to other forms of feedback. Over half felt that feedback from peers would be to some degree useful.

When asked to reflect on the audio feedback at the end of their first year, those students who responded referred to it still as having been a negative or upsetting experience, though all agreed that it had helped them to improve.

Focus group discussion: students

Two focus group sessions, with six students overall, highlighted that experiences of previous class sizes in schools and colleges had varied from between 3 and 20 (hence they were not used to large groups), and essays had always been marked traditionally with the opportunity readily available for students to talk to the teachers should they want further help. None had ever received feedback in the form of an MP3 file before and, although the technology posed no problems, they had found it a shock as they did not know they would be receiving feedback in this way. All students commented that, as it was their first essay, they had not known what to expect in terms of university-style feedback, that it was different from school and that it was a jump to university standards. None had expected to fail or just scrape a pass, especially having achieved good A Level grades. (Many students did not perform well in this assignment: overall grades ranged from 10% to 75%, with a mean of 46%). However, none of the students in the focus group had looked at the marking criteria, despite frequent requests to do so.

All considered that their feedback focused on the negative rather than the positive and they did not like some of the terminology used, perceiving it to be very negative, as was the tone of voice. One student stated that it was their first essay at a time when they were trying to adjust to living away from home and making new friends, and that this made any negative feedback more difficult to cope with. All would have preferred face-to-face feedback.

All reported gaining better marks since this first assignment. They had all later found that relating their grade to the marking criteria had helped them to understand what exactly was meant by these criteria. They acknowledged that the lecturer had obviously spent an enormous amount of time giving them detailed feedback and, after the focus group discussion, all stated that they would go and listen to the feedback again to actually learn from it. They also suggested they would be happy to get more audio feedback, as long as they also received written comments.

Teacher relection: stimulated recall

The stimulated recall session had two main aims:

To allow the lecturer to explain his thought processes as he had recorded the MP3 files for the students;

To allow the lecturer to hear himself and reflect on how the students will have responded to his comments, and whether there is room for improvement.

Two examples of audio feedback were used for the stimulated recall exercise — one to highlight feedback on a very poor assignment and one for an average assignment. Several excerpts from the audio feedback were played at intervals and the lecturer was asked to comment reflectively on the rationale and appropriateness of his feedback statements. He stated that the structure of the feedback had ‘sort of evolved’ initially, relating to the feedback sheet, giving the mark (‘the thing they are most interested in’) and then the justification. He reported trying to make the link between performance and mark very clear by referring to the marking criteria. He also recognised


that he had felt frustrated that he had told the students exactly what they needed to follow to achieve a good mark and that they had not made use of this information, meaning that the same points continually had to be pointed out as weaknesses in the assignments. He immediately picked up on the use of what he now considered to be inappropriate language and terminology, which he could avoid in future. He explained that he purposely read aloud the comments he had written on students’ essays because he knew it is a common criticism that students cannot read lecturer’s writing. He felt that reading it out loud reinforced comments to help the student, and he also believed that the audio and the essay need to be gone through together, not studied in isolation.

He considered that he gave a very detailed analysis, and hoped from a student’s perspective that the points would have been clear as they related to evidence on the essay. He recognised that some comments might not have been taken in the same way in a podcast as they might have been in the face-to-face context, especially with students direct from school.

In some instances words were repeated and emphasized, such as ‘not

relevant’, to be the equivalent of underlining on the essay text, which may not have come across as intended. The lecturer accepted that his feedback could be perceived as negative, although he reinforced that dealing with realistic feedback is a necessary experience in ensuring that students adapt to the standards required at university. Overall, however, the stimulated recall session persuaded him that he would make changes in the style and organisation of audio feedback in future.

Screen visual feedback in biosciences

The context

As outlined above, the context for the study of feedback in biosciences was very different. In order to gauge student views on feedback within laboratory settings, a questionnaire survey was designed in collaboration with academic staff in microbiology. It was hoped that the survey would enable students to become aware of the different kinds of feedback they receive and the various situations within which they receive it. This latter factor was considered of particular importance given larger classes than in previous years and the importance of ensuring that students feel satisfied with their feedback experiences. In addition, ongoing verbal feedback between the lecturer and students, and demonstrators and students, was videoed during laboratory sessions on a first year undergraduate course with 180 students, 1 lecturer, 10 demonstrators and one graduate teaching assistant. Additionally,

2 workshop sessions with third year undergraduates with a lecturer, assistant and 35 students were videoed within microbiology. This material was then available for creating edited exemplars of practice that could be used in future for training purposes. Two third year undergraduate revision feedback sessions were also produced as film clips, and shown in the laboratory for students to use as a revision aid.

Questionnaire results

The short, anonymous questionnaire allowed for quantitative and qualitative data collection and was distributed to first and second year Bioscience students during second semester laboratory classes — that is, all students had already experienced one semester of laboratory sessions in their current academic year. A total of 141 students completed the questionnaire (45%),

55% of these being from Year 1. Only six students were over 25. Most are studying for the BSc in Biological Science (59%), with others specialising in biological and medicinal chemistry, human biosciences, molecular biology or biology and animal behaviour.




Responses outlined student perceptions.

The majority of students perceived they had received verbal feedback in a variety of ways in the laboratory setting, and had gained some kind of feedback within every practical, although one student stated that they had not received any.


he majority of students considered feedback from lecturers, demonstrators and other students to be a positive experience, although there was the occasional exception.

80% stated that they received feedback every practical session from the lecturer to the whole class. However, this leaves 20% of students who do not perceive they gain whole class feedback on a regular basis, or who do not interpret whole class interaction as ‘feedback’.

The lecturer was not considered to give individual feedback on such a frequent basis (understandably, given the large student numbers and the size and layout of the laboratory), with 5% responding that they gained individual feedback every session, 22% every other session, 32% on occasion and 39% stating that they never received individual feedback.

86% of students agreed or strongly agreed that lecturer feedback to the whole class was a positive experience, but 19% of students were ‘unsure’ about the nature of the lecturer feedback when experienced individually, and 12% when with the whole class.

Demonstrators were also perceived key to feedback, with 84% of students suggesting they were offered individual feedback at least every, or every other, practical; and 79% considering that they receive feedback as part of a group equally frequently. Only 4% claim not to receive individual feedback from demonstrators, and 11% not within their group.

Demonstrator feedback to the group and to individuals was considered positive (84% and 90% respectively), with 42% strongly agreeing that demonstrator feedback to them as individuals was a positive experience.

Over 70% of students suggested they gained feedback from their peers on a regular basis, often every session. Seventy three percent thought peer feedback to be positive, although a quarter of the sample were unsure about this.

Almost all students considered that feedback in the laboratory context included ‘questioning to make you think’.

Negative comments regarding demonstrators covered the perceived lack of available demonstrators, their lack of understanding, and the fact that they did / could not answer student

There were varying views regarding approachability of lecturers for additional feedback, with first years being concerned at wasting the teacher’s time or ‘wanting to impress’ rather than admit a lack of understanding.

One third of students reported liking verbal feedback, whereas a third preferred written.

Most students thought that verbal (recorded audio) feedback might be useful for other forms of work such as essays, although 18% percent specifically did not agree with this.

A third consider constructive criticism and suggestions for improvement as the most helpful. With others liking ‘questions’ or ‘support for identifying

errors’ or ‘being pointed in right direction’ Some students said ‘any feedback is

helpful’ or ‘all feedback’. Both verbal and written feedback were considered important, but ideally on a one-to-one basis. Twenty percent wanted individual face to face feedback and believed this to be more helpful than anything else.

In relation to the issue of concern — whether students recognise ongoing lab processes as offering feedback — all students agreed to expecting feedback in laboratory settings; the majority felt that feedback was clear and understandable; and most thought that verbal feedback was immediate and


timely within laboratory sessions, although more first year students (14%) disagreed or strongly disagreed with this compared to second year students

(5%). The majority of students agreed to some extent that verbal feedback within laboratory settings is crucial to degree performance although more second year students (30%) disagreed in comparison to first years (20%).

Further, the majority (76%) disagreed with the statement ‘I do not count verbal feedback in laboratory settings as feedback’, indicating that most agree that laboratory settings do provide them with feedback.

Focus group discussion: student, demonstrator, and graduate teaching assistant perceptions

Twenty-two students were interviewed about their views on feedback during the laboratory sessions. All felt that feedback during the sessions had been good with responses varying between “quite good” and

excellent”. The feedback was said to be “helpful” with “good explanation” from both the lecturer and the demonstrators. The students reported the laboratory sessions as being well organised with good instruction sheets, plenty of demonstrators around and clear relevance to the lectures, with demonstrators continually asking questions. At the point of writing this paper, feedback has not been gained on first year responses to having video available, though all this group thought that video clips showing skills and information from the laboratory sessions on the web would be helpful for revision and better understanding. Third year students all reported that video of their feedback for revision sessions had been helpful; for example:

‘Cannot write everything down in detail’.

‘Want to be looking and writing at the same time in the laboratory’.

‘Can get bored just reading notes, more interesting on video with animation and intonation’.

‘You have time to hear other people’s questions and learn from each other’.

‘It is possible to remember what we hear and see more than just reading something’.

Demonstrators in laboratory sessions suggested they had learnt a number of skills, including listening properly to students, problem solving, being helpful, patience, making students think and having confidence in what they are doing: “You have to be really good at explanation and have really good

knowledge. All demonstrators thought that the quality of feedback within the sessions was good for the students, made easier by the fact that it is a topic which tends to be well known about and understood by demonstrators.

Three demonstrators felt that they would have appreciated briefing sessions before the day and that they are “a little bit thrown in at the deep end, which

then makes it frustrating when students say negative things….and you want to

provide and do a good job.”

The Graduate teaching assistant suggested that standards were rising in the first year and that students were motivated and interested in practicals.

However, she was concerned regarding both her own and the demonstrator’s training and that what she received was not particularly helpful and that she had learnt the most from helping to teach students. She highlights:

demonstrator briefing sessions take place prior to the practical, sometimes just

half an hour before the practical session begins, which can cause problems if the subject area is very different to a demonstrator’s background, so a bit more time to gain some understanding with the help of the practical co-ordinator would help, rather than in some cases learning new techniques, etc, just before we’re supposed to teach … I know last year there were some problems with people saying the demonstrators weren’t very good and I know some of us felt like we weren’t given a good enough briefing so it is like we were getting the blame but really we didn’t

understand the practical.’ The School had in fact already recognised this as an issue, and it is anticipated that video from the project will be drawn into a more detailed training package for demonstrators in future. The idea of



1 putting videos of laboratory skills such as pipetting onto the Virtual Learning

Environment for both demonstrators and students to access was thought of as extremely helpful, even “a brilliant idea” by all demonstrators.

Discussion and recommendations

Overall, it seems that there is considerable potential in using audio and screen visual feedback to support learning in different ways in the two different contexts described. A major benefit of any kind of audio and video feedback is that students report not needing to struggle with illegible handwriting, at least suggesting that forms of technology-supported feedback should become a priority. In general terms, students strongly value individual, face-to-face feedback — especially those who come directly from the highly supportive contexts whereby feedback is offered in secondary schooling. The change from school to university seemed to create difficulties for some students and it may be that audio-feedback on the first assignment was somewhat difficult for some students to deal with, especially as they seemed unprepared for this. Equally, some students seemed unprepared for the need to work without constant attention and feedback in large laboratories, More research is needed into whether, for example, a first assignment is a good time for setting different expectations and new ways of working, and to what extent — and how — students can best be prepared for new experiences. Both cases described above suggest that ensuring that students have realistic expectations of mass higher education is a crucial factor, and both suggested that students are better equipped to deal with this as they move through their first year and into the second year. Students are certainly not averse to audio and video feedback: indeed they might enjoy and value it if the content and circumstances are appropriate to their needs. However, the fact that some Biosciences students (even if a small percentage) report not receiving whole class or group feedback is of concern, and continues to beg the question: ‘what do students perceive to be feedback?’

A number of factors highlighted by this study are worthy of evaluation and further research. For example, there needs to be further work on what might be the optimum time length for this kind of feedback, whether listened to or watched; on the style that students appreciate, and on the balance of negative to positive feedback — so as to ensure affective as well as cognitive benefits. To accompany this, academics may need to explore the register of language that is most appropriate to spoken feedback, especially as this is an area wherein they are not well rehearsed. It may be that audio feedback is different in style both to written feedback and to the more colloquial language often used in face-to-face interaction. The question of what is appropriate, or high-quality, feedback is not always easy, but practical outputs from the project, such as a good practice guide to assignment feedback, or video clips showing examples of good practice between the lecturer, demonstrators and students, may enable development of feedback skills and enhanced practice. More research would be useful on whether listening to feedback supports learning, or is better attended to, or better remembered, than reading written comments, and whether this applies to some students more than others

Overall, students did think that audio or screen visual feedback would enable them to improve future performance, but i) it is not clear whether this performance is supported better by audio feedback than by written; ii) whether students will regularly listen more readily and more repeatedly to audio and video feedback than they would written; and iii) any feedback will not improve future performance unless students are asked to attend to it and to specifically draw on that feedback in future activity.



Black, P.J. and Wiliam, D. (1998) Assessment and Classroom Learning. Assessment in

Education, March, 7–74

Bridge. P. and Appleyard, R. (2007) A comparison of electronic and paper-based assignment

submission and feedback. British Journal of Educational Technology Volume 36, Number 4,


Denton, P., Madden, J., Roberts, M. and Rowe. P. (2007) Students’ response to traditional

and computer-assisted formative feedback: A comparative case study. British Journal of

Education Technology, Volume 39, Issue 3, 486–500

Epstein, M. L., Lazarus, A. D., Calvano, T. B., Matthews, K. A., Hendel, R. A.,

Epstein B. B. and Brosvic, G. M. (2002) The Psychological Record. Volume. 52, no.2,


George, R. (2002) The challenge, as seen by employers. Keynote speech, e-Skills, Summit,

Greenwich University, 29 May

Gomez, S. (2008) Making assessment feedback meaningful and rapid. Sixth Annual Conference and Exhibition from Assessment Tomorrow. Innovative Use of Technology to Assess and

Support Learning. London 12–13/3/2008.

Hattie, J.A. (1987) Identifying the salient facets of a model of student learning: a synthesis of

meta analyses. International Journal of Educational Research, Volume 11, 187–212

Irons, A. (2008) Enhancing Learning through Formative Assessment and Feedback. Routledge,

Oxon, USA and Canada

Laurillard, D. (2002) Rethinking University Teaching. London: Routledge.

McGettrick, A., Boyle, R., Ibbett, R., Lloyd, J., Lovegrove, G. and Mander, K. (2004)

Grand Challenges in Computing: Education. Swindon British Computer Society

Merry, S., Orsmond, P. and Galbraith, D. (2007) Does providing academic feedback to

students via mp3 audio files enhance learning? HEA Centre for Bioscience www.bioscience.

Nortcliffe, A. and Middleton, A. (2007) Audio Feedback for the ipod Generation. Proceedings of International Conference on Engineering Education 2007, Coimbra, Portugal.


Race, P. (2008) Assessment, Learning and Teaching Reflections, for Leeds Metropolitan ‘Sounds

Good’ week.


Ramsden, P. (1992) Learning to Teach in Higher Education. London: Routledge

Ribchester, C, France, D., and Wheeler, A. (2007) Podcasting: a tool for enhancing assessment

feedback? The 4th Education in a Changing Environment Conference, Salford. 12–14/9/2007

Rotheram, B. (2008) Sounds Good: Quicker, better assessment using audio feedback. JIScfunded project, Jan-July 2008. and_innovation/soundsgood.aspx

Stiles. M. (undated) Strategic and Pedagogic Requirements for Virtual Learning in the Context of

Widening Participation. Staffordshire University.





0288 Students and mobile devices: choosing which dream


Mobile devices include smart-phones, games consoles, digital cameras, media players, netbooks, in-car sat nav and handheld computers. Almost every student owns one and uses one, often more than one. Not only do they own them and use them but they also invest considerable time, effort and resource choosing them, buying them, customising them and exploiting them. These devices express part or much of their owners’ values, affiliations, identity and individuality through their choice and their use.

They are both pervasive and ubiquitous, both conspicuous and unobtrusive, both noteworthy and taken-for-granted in the lives of most — but not all — students.

This is new and is completely different from older, static and less personal information technologies such as desktop computers and TVs. It is a quantitatively different phenomenon and the statistics are commonplace: mp3 downloads outnumber CD sales, camera-phones outnumber cameras, smart-phones outnumber laptops, mobile phone ownership is reaching saturation and the British send over a billion texts a week.

Mobile devices are however also a qualitatively different phenomenon.

Students no longer need to engage with information and discussion at the expense of real life but can do so as part of real life as they move about the world, using their own devices to connect them to people and ideas, ideas and information of their own choosing, perhaps using their own devices to generate and produce content and conversation as well as store and consume them. This is changing how students relate to technology. It is also changing how they relate to other students and to the content and conversation facilitated by the technology, so consequently it is changing how they relate to learning and to education.

This thought piece looks at these devices in the hands of so many students and the challenges and opportunities that these devices represent for the support and provision of learning, and indeed for the meaning and nature of learning. The phrase student devices is used to signify not mobile devices in general nor the purely technological characteristics of specific categories of mobile devices nor those mobile devices that might be especially suited to learning or already used in education. The phrase is used emphatically to explore the educational and institutional implications of students’ choices. It is understandable that much of the discussion will focus on mobile phones considering their massive dominance in students’ lives, but the increasing functionality and power of the mobile phones that students buy mean that very few mobile technologies are not coming into the hands of most mobile phone owners and thus into the hands of most students. Of course, a much wider range of mobile devices is in circulation but we need to remember the demographics of all these various devices and acknowledge the primacy of the mobile phone amongst the less privileged.

The devices themselves are important, as are the systems, networks and infrastructures that support them. The probable trends in functionality, availability, ownership and use are also important, as is the operation of the market-place through the networks, the content providers, the service providers and the hardware manufacturers, in determining what gets promoted and what gets ignored.


John Traxler

Learning Lab, Priorslee Campus,

University of Wolverhampton, Shropshire

TF2 9NT [email protected]


There is a crisis looming and a paradox emerging. Many educators advocate, promote and encourage the dreams of agency, control, ownership and choice amongst students whilst educational institutions take the responsibility for provision, equity, access, participation and standards. The institutions traditionally procure, provide and control the technology for learning but now students are acquiring their own personal technologies for learning and institutions are challenged to keep pace. These allow students to produce, store, transmit and consume information, images and ideas; this potentially realises the educators’ dream but is for institutions potentially a nightmare, one of loss of control and loss of the quality, consistency, uniformity and stability that delivered the dreams of equity, access and participation. This paper traces the conlicting dreams and responsibilities.

These are factors that put devices in the hands of students, and constrain and position their use. At that point, other factors come into play, those factors are part of an evolving dynamic between technology, on the one hand, and society, including education on the other, as students, communities and institutions adapt and evolve around the technology. The particular significance of widespread mobile devices in this respect is their impact on ideas about information and knowledge, and about the nature, support and delivery of learning, and on how these evolve.

This thought piece explores these issues and specifically looks at the challenges, from the practical to the philosophical, that universities face if they are to move in a direction that is positively aligned to this dramatic rise in students’ own devices.

The technology, and its ownership, access and use

If we look at mobile devices and technologies, especially if we make a comparison with desktop technologies, what we see is diversity, transience and incoherence. There is no standard footprint or format. The devices come in all sorts of shapes and sizes, from slim matchbox to sturdy paperback book, landscape or portrait. They may open out, slide open or neither; they have all sorts of keyboards (some virtual, some real) and screens; they may respond to touch, gesture or stylus, they may capture or play various media and connect to various networks and peripherals. They run various operating systems, applications, networks and connectivity, any of which will change overnight, even if those are supposedly stable and standard.

These devices are developed and designed for various retail niches and corporate markets, certainly not for learning, however informal. This should not be a surprise; educational technology has always been parasitic, originally co-opting desktop computers intended for corporate business customers and now trying to co-opt mobile devices intended for individual lifestyle customers. This process continues today (Hemmi et al, 2009) and has been rigorously explored (Bar et al, 2007). Not one of these technologies was intended for educational use and so they continually challenge educationalists to develop educationally sound applications

From a purely technical perspective, we could explore new mobile technologies coming to maturity and perhaps coming to market; the issue however is not technology per se. The issue is how technology is packaged, presented and marketed. Given current trends, it seems inevitable if there is a business case for these or any other features then they will be marketed around mobile phones, though extra features will also go into media players and games consoles too.

Having looked at mobile devices and technologies, we see some underlying differences with desktop PCs. The design and manufacture of mobile devices produces a closed box and, unlike PCs, manufacturers cannot adapt to evolving markets by putting in extra cards for graphics processing, increased memory, enhanced connectivity or games functions and cannot easily plug extra or improved peripherals such as better screens, joysticks or concept keyboards. This inflexibility may mean manufacturers are conservative and target discrete segments in the market, the youth market being one of them.

Images of the inside of any mobile phone illustrate that mobile devices are not designed to be upgraded, serviced or even opened, just used and discarded.

Sales figures (for example, Kumar, 2004) show that many buyers and users clearly prefer specialised, dedicated devices such as the Apple iPod, the

Tom-tom, the RIM BlackBerry or the Sony PSP rather than any generic and more general-purpose device and clearly many buyers echo Rolt’s (1947) remark that, ‘Manifestly it is better to use simple tools expertly than to possess a bewildering assortment of complicated gadgets and either neglect



1 or use them incompetently.’. Therefore, whilst we have seen the migration of most PDA functionality into phones, this has not lead to the emergence of some generic converged device or even some generic converged platform or architecture and the market is segmented by “understandings of the consumer held by those in the mobile operators industry” (Green et al,

2001:1). Furthermore, consumer choice favouring divergence, individuality and constant innovation coupled with device design and manufacture targeted at niches and an architecture based on dedicated closed boxes means that this situation will not change. We can say only that the devices owned by students will be, at best, poorly suited to learning, will all be different and will all be changing, often for reasons that are not technical, not educational and probably not even rational or foreseeable.

This is not a helpful picture for universities hoping to plan around mobile devices. There is reassurance in prediction; it puts change in a context and gives a basis for planning.

The social aspects of mobile devices

The personal, cultural and social aspects of these trends hinge in many respects on the essential difference between desktop technologies and mobile technologies, a difference that means we can ignore the former but not the latter. Interacting with a desktop computer takes place in a bubble, in dedicated times and places where the user or student has their back to the rest of world for a substantial and probably premeditated episode. Interacting with mobile technologies is different and is woven into all the times and places of students’ lives. Mobile phones have created “simultaneity of place” (International Telecommunications Union,

2004:20, paraphrasing Plant, 2002): a physical space and a virtual space of conversational interaction, and an extension of physical space, through the creation and juxtaposition of a mobile social space. This affects people’s sense of time, space, place and location, their affiliations and loyalties to groups and communities, the ways in which they relate to other individuals and to groups, their sense of their identity, and their ethics, namely their sense of what is right, what is approved of and what is inappropriate.

When we say we can ignore desktop technologies but not mobile technologies we mean that desktop technologies operate in their own little world, mobile technologies operate in the world.

Mobile devices demolish the need to tie particular activities to particular places or particular times (in spite of the ubiquitous ‘I’m on a train ...’ gambit). They are reconfiguring the relationships between public and private spaces, and the ways in which these relationships are penetrated by mobile virtual spaces. Virtual communities and discussions had previously been mediated by static networked PCs in dedicated times, places and spaces.

Now, mobile technologies propel these communities and discussions into physical public and private spaces, forcing changes and adjustments to all three as we learn to manage a more fluid environment. This is documented in the literature of mobilities, for example the new peer-reviewed journal of that name, and various authors remark that the private “is no longer conceivable as what goes on, discreetly, in the life of the individual away from the public domain, or as subsequently represented in individual consciousness”, (Cooper, 2002:22) “that massive changes are occurring in the nature of both public and private life and especially of the relations between them.” (Sheller & Urry, 2003:1) and that “The use of these mobile sound technologies informs us about how users attempt to ‘inhabit’ the spaces within which they move. The use of these technologies appears to bind the disparate threads of much urban movement together, both ‘filling’ the spaces ‘in-between’ communication or meetings and structuring the spaces thus occupied.” (Bull, 2005:334).


People comment on the use of mobile devices, often phones but more usually media players such as the iPod, to re-appropriate, public space or work time back into the private; with a mobile device, there is ‘no more dead air’ (Bull, 2005).

Mobile technologies are redefining discussion and conversation. Rather than these being set aside as something done at certain moments, for a delimited stretch of time, usually in a private space (or semi-private phone

‘box’ or ‘booth’), Sheller (2004:5) says there is now “a constant flickering of conversation”. Furthermore in order to manage the intrusions of mobile calls and conversations into real time and space (or vice versa perhaps), we are evolving a set of non-verbal actions and interactions with the mobile phone in public. In order to maintain discourse and connectedness across different spaces we are devising and learning new protocols. We are devising new

tie-signs (Goffman, 1971) in order to manage simultaneous conversations in real and virtual space, allowing us to service different types of conversation without offending either our real correspondents or our virtual ones. We have to manage enforced eavesdropping (Plant, 2002:47) and adopt civil

attention (Goffman, 1971) where our neighbour in the train or bus, for example, holds a private, intimate and probably embarrassing conversation with some unseen other and we have to make gestures that signal that we are not paying any attention, averting our gaze or shifting our stance.

Mobile devices eroding established notions of time as the common structure, for scheduling, co-ordinating and organising activities and events.

Various authors talk about the “approx-meeting” and the “multi-meeting”

(Plant, 2000:31), about ‘socially negotiated time’ (Sørensen et al. 2002:3) and the ‘micro-coordination of everyday life’ (Ling, 2004:69) alongside the

“softening of schedules” (Ling, 2004:73) afforded by mobile devices as we use them to adjust our schedules and our commitments on-the-fly as events unfold. Finally, Nyiri (2006:301) says, “with the mobile phone, time has become personalized” whilst Fortunati (2002) says, in a piece that addresses and analyses many of the issues covered here, that “The mechanical representation of time is more and more unacceptable at a social level. In other words, the abstract, uniform and unitary time of the clock is sinking further and further down in relation to electric and satellite time. With the possibility of perpetual contact, the mobile phone ends in fact by shaping time as a container of potentially continuing connection.”

Mobile devices are also eroding physical place as a predominant attribute of space. The phrase absent presence (Gergen, 1996) describes situations where groups of people physically together, co-located, are all connected elsewhere. Mobile devices now enable us to carry our various virtual communities with us but physical communities — the family, the town, the university, the cohort — become devalued. Mobile devices are creating communities and groupings, sometimes transient and virtual, arguably at the expense of existing and traditional ones. In some cases, this is because increased connectivity and functionality have meant that social networking sites such as Facebook have adapted and migrated to mobile devices, in other cases social networking sites native to mobile devices, such as Mxit,

Twitter and Jaiku, have developed and flourished.

Sometimes the device itself, the early Walkman (du Gay et al, 1997) and the first cell phones for example, signify membership of a community. In other cases, specific groups or communities use the devices in their own exclusive way: txtspeak in its early days served this purpose (Shortis, 2009 and

Thurlow, 2003 both give considerable context to this remark) and around the world different communities use the missed call differently (Donner,

2008). More significant though, mobile devices have catalysed a range of communities, transient and ephemeral perhaps, and sometimes described as smart mobs, groups of interconnected people forming a distributed



1 intelligence, around particular political, artistic or social issues (Rheingold,

2002). With each of these groupings come new norms, expectations, ethics and etiquettes and shifting ideas about the self and identity. Our social networks are part of the construction of our identities in the sense that we say who we are and we learn who we are by who we associate with and by who we are comfortable being seen associating with. Increasingly, online social networks are part of this identity construction and these are becoming mobile, perhaps reintegrating the virtual and the actual.

At the mLearn conference in 2007, Charlie Schlick, Product Manager of

Nokia, described company practice in talking of mobile phones as ‘our new private parts’. These devices are personal, universal and closely linked to identity and in talking about student devices we must recognize how closely they are bound up with a changing sense of self. Some authors describe personal mobile devices as becoming prosthetic; Raul Pertierra (2005:27) says, “Unlike desktops and other immobile technologies, mobile phones more closely resemble tools or prosthetic devices as extensions of the body.

They become extensions of the hand, allowing us to connect anytime, anywhere, with anybody. Bodies themselves become writing devices as phoneurs negotiate new urban spaces.” Other authors describe them as becoming embodied (for example, Rettie, 2005).

The educational implications of student devices

Many of the implications of these remarks for universities are still unclear.

However, we can tease out some of them which could be addressed at a number of levels. There is the purely tactical level; universities are fundamentally sound but need to tinker with perhaps timetabling, network security, outreach, staff development, assessment regimes, the wording of acceptable use policies or the constituents of blended learning and all will be well.

An obvious implication for working with students is the need to recognise that expectations about face-to-face interactions are now fragmenting more than ever, and that different groups of people will have different ideas about courtesy especially in relation to mobile phones; there will be different expectations about whether to answer a call or a text whilst in an interview, tutorial or lecture.

Mobile devices are defining and supporting new communities and their aspirations; attitudes and idioms must be understood and addressed if they are to have parity of access to university education. These transient and mobile communities have their own norms that determine what is acceptable. These norms might govern etiquette, taste, language, values and ethics, and the educators must understand these in order to work effectively within these communities.

The services, connections, discussion and content — and university education is all of these — are no longer seen as dependent on face-to-face contact at predetermined times. Educational provision is built around time and place: the timetable, hand-in dates, the classroom, the year-group, the deadline and the laboratory. These observations suggest that the education system, especially the formal university system, is getting out of step with how many students perceive the world they live in and that, irrespective of the significance and reaction to student devices, changes are needed to keep universities aligned to a changed and mobile society.

Physical locatedness is further weakened by the increase in cloud computing,

(as described in Wiess, 2007). This is the phenomenon of data, applications and processing moving away from specific hardware hosts and into the

Internet. The combined consequence for universities will be to challenge the primacy of institutionally controlled desktop computers. A different medium-term trend will be for these activities to move into the environment,


into buildings, furniture, vehicles or clothing, and to become ambient and pervasive (Satyanarayanan, 2001). The consequence for universities will be to accelerate the convergence of physical architecture and virtual architecture, and to blur the boundaries between institutional space, social space and personal space, and the outside world. At the same time, learning and knowledge are less anchored in physical artefacts. eBook readers and media players, for example, mean that books and records are longer necessary to store and transmit literature and music. Video-on-demand is another part of the transformation of live social performance into consumable artefact and now into disembodied asset.

These are all part of an epistemological revolution (for example, in the sense broadly outlined in Des Bordes & Ferdi, 2008), a phrase used to express the fact that computers and now mobile technologies are revolutionising what we know and how we know it, and hence what we learn and how we can learn it. In talking in these terms, we should however be careful not to obscure the nuances and differences between individual devices and technologies and the various ways in which different cultures and organisations with society adopt and adapt them. To portray the demography of ICT access as simply ‘digital immigrants’ and ‘digital natives’

(Prensky, 2001) is to over-simplify a situation where different technologies, desktop and mobile, are adopted by different communities, cultures and subcultures in different ways at different rates.

These factors are significant to learning and education, and to how the universities tackle the challenge of student devices, because they reveal how central these devices and technologies are to the lives of almost everyone in our society.

Ownership of technology, knowledge and learning

These changes and trends will cause significant shifts in the idea of

ownership, specifically the ownership of technology and of knowledge. We mean here that more students and a greater range of students will buy and possess mobile devices and access information. We also mean however that through this process, these students will gain greater confidence, agency and familiarity with the technology exemplified by mobile devices and with the knowledge mediated by them. Increasingly, they will feel less inhibited and less intimidated by knowledge and technology since they will form a greater part of their everyday lives, under their control and not the prerogatives of affluent students from more entitled social classes.

This is probably obvious in relation to technology but less so in relation to knowledge.

In the case of the technology, the increasing capacity, capability and functionality of mobile devices means that activities associated with landline telephones, analogue cameras, desktop computers, TV sets and music centres are now all converging on devices that have become as commonplace, personal and taken-for-granted as the wristwatch and the cigarette lighter. This has taken place over about 10 years. The impact of this on students’ attitudes to technology, especially to computer technology and digital technology, must be profound, though of course very different for different age groups, and hence different for mass-participation universities as opposed to traditional universities.

In the case of knowledge, and of course in the case of information, images and content in general, this is also true but we must distinguish between the consumption of knowledge and its production.

Mobile devices, especially connected devices, enable students to consume, that is, to access and store, all sorts of knowledge almost instantly and



1 almost wherever they are, with little or no effort compared to earlier technologies. Now practically all types of information, files and formats, available from Wikipedia, Google Scholar, Flickr, iTunes, YouTube,

Facebook, Google Maps and BBC iPlayer are easily accessible on mobile phones. Podcasts of academic courses are available from the world’s universities. This shifts the educational locus and authority away from faceto-face provision and delivery, and away from formal educational institutions.

Student devices are an integral part of these processes.

The changed sense of the ownership of technology and knowledge, just described, has practical implications for the actual ownership of technology and knowledge within education itself. We come to these later.

In addition to the changing ownership of knowledge, mobile devices deliver this knowledge chunked, structured and connected in very different ways from earlier learning technologies such as the lecture, the web and the book.

Knowledge is not abstract, unaffected by how it is stored, transmitted or consumed. In its earliest forms, knowledge and learning came from lectures, a linear format from an authoritative ‘sage-on-the-stage’ with no pause, fast

forward or rewind, and from books, substantial and linear but segmented and randomly accessed. The delivery of knowledge and learning by networked computers meant a break from linearity with the introduction of hyperlinks and new heuristics of usability that prescribed how knowledge and learning should be chunked and presented. With mobile technologies, using a small screen and a limited input medium, the usable chunks become much smaller but the navigational overheads become much larger. In essence, small pieces of knowledge and learning can be easily presented but their relationship to any others may be difficult to understand, thereby fragmenting and perhaps trivialising what students learn.

The patterns of use, that is, the various ways in which people interact with technologies, also differ dramatically if we compare sedentary desktop technologies with mobile personal technologies. The use of desktop computers, documented in the research literature of HCI, is well understood, well established and much more tractable than is the use of mobile devices (see Jones

& Marsden, 2006). Our understanding of how people engage with information as they walk down the street and perhaps share devices with friends is still relatively limited. Words like ‘lightweight’, ‘opportunistic’, ‘informal’,

‘spontaneous’, ‘episodic’, ‘private’ and ‘personalised’ are found in the literature but this is often impressionistic. Nevertheless, creators, publishers and providers of content (and navigation and organisation) must adapt to these findings as they emerge if the student experiences is to be optimal.

In the final panel discussion at the 2007 mLearn conference in Melbourne,

Professor Mike Sharples, with the other panel members, was asked about the role of universities in an age where mobile devices, student devices, gave universal access to facts and information. His answer, perhaps tongue-incheek, was that universities could at least still give degrees. This is another aspect of student devices in relation to the consumption of knowledge and at the very least, implies that assessment regimes, both what is being assessed and how is it assessed, are seriously challenged by the affordances of student devices.

Moving from the consumption of knowledge to its production, the increased functionality of mobile devices is hastening the convergence of mobile technologies with the wider user-generated content movement associated with web2.0 rhetoric and technologies. This is the movement promoting the web as a medium for writing and participation not just for reading and passivity. It uses technologies such as wikis, mashups, blogs, newsfeeds and podcasts to move the web from a centralised broadcast medium to one where everyone has a voice. Mobile devices extend and enhance this voice because they allow users to capture content, for example images, sounds,


data and voices themselves, from the real world, from events as they happen, specific to when and where they happen. The rise of citizen journalism (for an account and analysis, see Ananny & Strohecker, 2002) is a very specific example of the power of mobile phones and ‘user-generated’ content.

Meanwhile, previously unknown musicians and disenfranchised political groups use the same technologies to propagate their material and their views, and in doing so they create a more fragmented and complex world where the received wisdom and the accepted tastes no longer have the hegemony or the authority that they had in more static, stable times.

Mobile students are now able to create, access and publish not only facts about the outside world but the inside world too, information about themselves, their friends and affiliations, their feelings, their days and their doings. Every mobile phone has personal information management software, that is calendars, tasks, notes, contacts etc, that can be made visible to the chosen few or the unchosen many but now social network software, such as Facebook, Jaiku or Twitter, on mobile phones can capture and distribute content that is less purely functional and much more intimate. The wider visibility of this personal information is part of the transformation of identity and student’s sense of themselves and their communities, no longer based in the purely physical and the face-to-face.

Whilst much of this account of the consumption of knowledge sounds benign, for example the dramatically increased levels of individual choice, control and convenience, there are drawbacks. The first is that these developments reinforce a tendency to view knowledge and other forms of content merely as commodities or assets. The second is that this choice and control are exercised at a purely personal level, allowing individuals to each pursue their own curiosity, constructing their own private libraries and inhabiting their own worlds of knowledge. This erodes the idea of a commonly accepted canon, a common curriculum, of things we all need to know and are assumed to know and replaces it with what some people have referred to a neo-liberal nightmare — not dream but nightmare.

This will have consequences for the perceptions that students have of their universities. Historically these granted the less well-off access to learning, knowledge and technology but this access has always been constrained by lecturers, teachers, employers, librarians and caretakers, by exam boards and by opening hours, by preferred suppliers and by acceptable white-listed URLs.

Student devices change all this and challenge the role of the education professions and the educational institutions, progressively demystifying their roles as gatekeepers, custodians and arbiters of technology and knowledge.

This is not to ignore their role as guides or intermediaries, nor is it to ignore their work in nurturing intrinsic motivation and providing extrinsic motivation, merely to place them all in a more complex context.

Disruption — nuisance, threat and student devices

Disruption is often used about mobile devices in educational settings (for a typical example, see Sharples, 2001). The exact meanings of the word are not usually unpacked but they have considerably greater significance and force when we think about student devices rather than institutional devices. There is a weak version of disruption that amounts to nuisance; phone calls in class, texting in exams, photographs that should not be taken, inappropriate ring-tones and so on. There is however also a strong version of disruption. These devices allow students to access and store images and information of their own choosing and perhaps create and distribute new images and information independently of the lecturers and of the university.

The long-term consequence must be to challenge the authority of the curriculum and the institutions of formal learning. At the moment, education is still delivered primarily and knowledge is accessed primarily through



1 formal institutions on institutional premises. The technology to enable this is accessed on institutional premises. This gives institutions enormous power and control over the nature and style of learning that can be accessed, especially by less affluent students with few alternatives.

The institutions of formal learning regulate and control access to knowledge, technology and learning for less privileged parts of students: the universities are also the agents of equity and inclusion. Our point here though is that student devices confront this stranglehold on learning, the universities and the lecturers are no longer the gatekeepers.

Interestingly, Selwyn (2003) uses similar but different sources and analysis to draw a similar picture of the UK schools sector.

Infrastructure, blending, procurement and sustainability

Student devices present a major challenge to many of the institutional practices and procedures associated with ICT and ‘conventional’ desktop e-learning. It is easy to say that education should embrace student devices but not easy to say how. This is part of the paradox. Historically, institutions rather than individuals have taken the responsibility for the provision of the IT needed to deliver and administer learning. This can be explained as the benign industrialisation and electrification of learning, necessary to deliver modern mass learning, ensuring quality and uniformity, and mapping standardised curricula onto standardised technologies. All too often, the institutional provision of IT led to a very narrow prescription about the hardware, peripherals, connectivity, operating systems, applications and privileges that could be accessed by students and lecturers. In the era when the dominant technology was networked desktop PCs this made sense, at least in terms of procurement, installation, support, staff development and user training, and was usually managed through a centralised IT unit.

As more mobile technologies proliferated, this has become a less tenable approach and has been seen as a constraint on personal and professional choice amongst lecturers, and amongst students, rapidly acquiring their own personal technologies and wanting to access institutional learning resources.

In technical terms, the diversity and transience of mobile devices are orders of magnitude greater than with desktop technologies; in financial terms, this transience and diversity are insupportable and increasingly seen as unsustainable (UCISA, 2009) Experience in early pilots (for example, Traxler

& Riordan, 2004) suggested that students were not likely to value a second device, a university-provided device, that did not express their taste or aspirations and that it would inevitably be the one left at home.

On the other hand, wholeheartedly adapting an approach centred on student devices is challenging and radical for institutional IT units. Their roles would change drastically, depending on the institution and its mission, and on its finances.

Furthermore, university IT units would take the lead in implementing whatever policies are considered necessary for uniformity and equity. This might include issuing vouchers for purchase or hire of devices, for airtime and connectivity (voice, messages, data) as appropriate. It might also include standards and minimum specifications within which student choice and purchase could be managed. Standards and specifications are attractive and it might be possible to promulgate national standards but even in stable areas of IT, standards do not have a good record.

Blending, the term used for the integration of different and appropriate technologies in order to deliver and support optimal learning, is another key concern in the acceptance of student devices. How can educational quality be assured when one of the components of delivery is so diverse and


volatile? Can student devices only be used for optional or enriching material, or perhaps only with specified categories of students?

The ethics of student devices

There are various ethical aspects to the increasing prevalence of mobile devices in our society and these have an immediate bearing on any consideration of student devices. Ethics covers everything from the legal and regulatory aspects of our actions, utterances and behaviour to informal expectations about etiquette, expectations, protocols and norms. Ethics are a major constituent of culture and identity (because our sense of right and wrong is part of who we are and who we feel comfortable with and so differs across sub-cultures, generations, social classes and ethnic communities.)

Many of what we described as the social consequences of mobility have ethical aspects, even something as simple as texting in class or answering a call whilst eating.

Student devices mean that we are moving away from the simple dichotomies of regulating acceptable use. At the risk of over-simplifying, we used to make a distinction between formal learning activities in our universities on our equipment and self-motivated learning activities outside our institutions not on our equipment. We had a duty to regulate the former and had no mandate to regulate the latter. If we are to embrace student devices, this simple dichotomy breaks down and the boundary becomes blurred. Guaranteeing e-safety becomes more problematic when on the one hand we encourage the use of student devices for learning but on the other hand have no ability or authority to control how, when or where they are used, nor any control over the applications, data or networks they support. At the very least, policies of acceptable use must evolve rapidly to address the affordances of student devices.

There are other issues. With increasingly sustainable and sensitive contextual information, student devices necessarily can give institutions far greater insights into the locations and behaviour of students. Enriching the educational experience must involve engaging as fully as possible with this contextual information and perhaps linking it to other education systems such as learning platforms or attendance registers. With this comes the potential for greater surveillance and oversight of students. Concerns about privacy and surveillance may stop some students volunteering their devices.

Some students are already saying, ‘not on my phone’ because they feel educational material on a personal, social and recreational phone is intrusive

(eg informal analysis of data from MELaS project data by author).

Other issues of student devices are merely the issues of any mobile devices used educationally not just those owned by students. The problems are increased however when the boundary between personal and educational becomes blurred.

Inclusion and student devices

Many of the previous remarks about student devices, for example those about ownership, identity and personalisation, seem to make the case for student devices as an expression of consumer choice and student preference and thus put student devices in a positive or benign light. There are several areas however where an unqualified acceptance of student devices, an acceptance that would imply that universities unreservedly support whatever devices are preferred and owned by students, is problematic. One of these areas is equity or fairness, ensuring equality of opportunity and access.

If institutions are to embrace student devices there must be provision for everyone to have the same kind of provision. This means not just devices for everyone, but everyone owning the device they choose. Anything less than this creates divisions and hierarchies but needs complex resourcing



1 since student devices are not merely hardware devices but also involve connectivity and airtime and, by definition, cross the border between personal and educational use.

Quality, training and content for student devices

Other areas where the unconstrained operation of student choice is problematic include quality assurance and staff training. In both these areas, we have to recognise that the problem does not just lie between students with their devices on the one hand, and technology support and infrastructure on the other. There is also the educational component, mediated by teachers and lecturers. Currently there are many small-scale pilots and projects using mobile devices to deliver or support learning. These are taking place in every sector (and in many countries) (see for example,

Traxler et al, 2008). With the exception of those using SMS, Flash, Bluetooth, podcasts or perhaps Java, they all depend on students being provided with devices. Many of these pilots and projects are looking to explore mobile learning and to learn lessons and from a methodological perspective, this is easier with a homogeneous and predictable technology platform. It is also easier from a staffing and infrastructure perspective since planning and training are comparatively straightforward. It does however mean that most mobile learning pilots and projects are unsustainable because they are predicated on finance in order to provide subsequent cohorts of students with devices. Working with student devices solves this problem but faces staff developers with the enormous challenge of preparing teachers and lecturers to work with a range of devices that cannot be predicted, of preparing content and lessons for a range of devices that cannot be predicted and of ensuring the ongoing quality of courses across this unpredictable range of platforms. This is a considerable challenge, a major paradigm shift and another part of the paradox.

Which dreams and responsibilities?

To return to our starting point of dreams and responsibilities, we see a paradox approaching. Student devices unlock the dreams of agency, control, ownership and choice amongst students but put the dreams of equity, access and participation at risk. Universities cannot afford, procure, provide nor control these devices but they cannot ignore them either. Clearly such a stark choice is an over-simplification; there is no simple question and no simple answer.


Ananny, M. and Strohecker, C. (2002) Sustained, Open Dialogue with Citizen

Photojournalism. Proceedings of Development by Design Conference. Bangalore, India.

December 1–2, 2002

Bar, F., Francis, P., and Weber, M. (2007) Mobile Technology Appropriation in a the Seminario

sobre Desarrollo Económico. Desarrollo Social y Comunicaciones Móviles en América Latina,

Buenos Aires, April 20th

Bull, M. (2005) No Dead Air! The iPod and the Culture of Mobile Listening. Leisure Studies 24(4):


Cooper, G. (2002) The Mutable World: Social Theory. In B. Brown, N. Green and R. Harper

(eds) Wireless World: Social and Interactional Aspects of the Mobile World. London:


Des Bordes, A. and Ferdi, S. (2008) Do Knowledge and New Technologies Need a New

Epistemology? Proceedings of BOBCATSSS 2008 Providing Access to Information for

Everyone, Zadar, Croatia. available online at

Donner, J. (2008) The Rules of Beeping: Exchanging Messages Via Intentional ‘‘Missed Calls’’ on

Mobile Phones. Journal of Computer-Mediated Communication, 13 (2008) 1–22

Du Gay, P., Hall, S., Janes, L., and Nequs, K. (1997) Doing cultural studies: the story of the

Sony Walkman, London: Sage

Fortunati, L. (2002) The mobile phone: Towards new categories and social relations.

Information, Communication & Society, 5:4,513 — 528

Gergen, K.J. (1996) Technology and the self: From the essential to the sublime. In D. Grodin and


T. Lindlof (eds.) Constructing the self in a mediated age. Beverly Hills: Sage. pp. 127–140

Goffman, E. (1971) Relations in Public, Harmondsworth: Allen Lane

Green, N., Harper, R. H. R., Murtagh, G. and Cooper, G. (2001) Configuring the Mobile

User: Sociological and Industry Views. Personal and Ubiquitous Computing, 5:146–156

Hemmi, A., Bayne, S. and Land, R. (2009) The Appropriation and Repurposing of Social

Technologies in Higher Education. Journal of Computer Assisted Learning, 25, 19–30

International Telecommunications Union (2004) Social and Human Considerations for

a More Mobile World. Report from ITU/MIC Workshop on Shaping the Future Mobile

Information Society, 26 February 2004, Seoul

Jones, M. and Marsden, G. (2006) Mobile Interaction Design. John Wiley & Sons: West

Sussex, England.

Kumar, S. (2004) Mobile communications: global trends in the 21st century. International

Journal of Mobile Communications, Volume 2, Number 1, 67–86

Ling, R. (2004) The Mobile Connection — the cell phone’s impact on society. San Francisco, CA:

Morgan Kaufmann Publishers

Nyíri, K. (2007) Mobile Studies: Paradigms and Perspectives. Vienna: Passagen Verlag:

Hungarian Academy of Sciences

Pertierra, R. (2005) Mobile phones, identity and discursive intimacy. Human Technology 1


Plant, S. (2002) On the Mobile — the effects of mobile telephones on individual and social life.


Prensky, M. (2001) On the Horizon. NCB University Press, Vol. 9 No. 5, October 2001

Rettie, R. (2005) Presence and Embodiment in Mobile Phone Communication. Psychology

Journal 3(1): 16–34

Rheingold, H. (2002) Smart mobs — the next social revolution, Cambridge, MA: Persius

Rolt, L. T. C. (1947) High Horse Riderless, Bideford: Green Books

Satyanarayanan, M. (2001) Pervasive computing: vision and challenges, Personal

Communications of IEEE, Vol: 8, Issue: 4, pp 10–17

Selwyn, N. (2003) Schooling the Mobile Generation: the future for schools in the mobile-networked

society. British Journal of Sociology of Education, 24:2,131–144

Sharples, M. (2001) Disruptive Devices: Mobile Technology for Conversational Learning.

International Journal of Continuing Engineering Education and Lifelong Learning, 12(5/6),


Sheller, M. and Urry, J. (2003) Mobile Transformations of ‘Public’ and ‘Private’ Life’. Theory,

Culture & Society, Vol. 20, pp. 107–125

Shortis, T. (2009) Revoicing TXT — spelling, vernacular orthography, and “unregimented

writing”. In S. Wheeler (ed) Connected Minds, Emerging Cultures — Cybercultures in Online

Learning, Greenwich, CT: Information Age Publishers

Sørensen, C., Mathiassen, L. and Kakihara, M. (2002) Mobile Services: Functional Diversity

and Overload. Presented at New Perspectives On 21st-Century Communications, May 24–25,

2002, Budapest, Hungary

Thurlow, C. (2003) Generation Txt? The sociolinguistics of young people’s text-messaging.

Discourse Analysis Online, available online thurlow2002003-paper.html [accessed on May 19th 2009]

Traxler, J. and Riordan, B. (2004) Using PDAs to Support Computing Students. In Proceedings

of LTSN Annual Conference September 2004, Belfast: ICS-LTSN.

Traxler, J., Riordan, B. and Dennett, C. (2008) Proceedings of the mLearn2008 Conference:

The Bridge From Text To Context, Wolverhampton: University of Wolverhampton

UCISA (2009) UCISA’s Top Concerns 2008–9 available online at surveys/tc/tc2008-9/top_ten.aspx [accessed 14 May 2009]

Wiess, A. (2007) Cloud computing: PC functions move onto the web. netWorker, Volume 11,

Issue 4 (December 2007), pp 16–25





Section 2: Other Papers

0291 Next Generation User



To ensure the relevance of and to influence the ongoing enhancement of user ICT provision and the associated awards, Digital 2020 (Digital 2020 is the digital skills partnership for Yorkshire & Humber — www.digital2020. and the Scottish Qualifications Authority ( jointly commissioned Sero Consulting to undertake research in ICT User skills.

The focus was the vision for ICT user skills in 2013 — referenced as ‘Next

Generation User Skills’ — taking account of:

Skills that all employers will need

, which they may not currently recognise — including web presence, information productivity, market research, infrastructure management.

Skills that people (especially young people) will already have

, but which may not be recognised or accredited.

Generic occupational skills

that people will need — such as remote working, online communication, information research, lifelong learning and, not least, management of their digital environment.

Essential skills for living and learning

in a digital age — including communication, accessing public services and underpinning personal e-confidence.

Working closely with e-skills UK (e-skills UK is the Sector Skills Council for

Business & Information Technology —, the team sought to


scenarios for the use of web, digital media, communications,

■ business and social applications in 2013.

Take account of

emerging needs across industrial sectors, in SMEs and micro-businesses, in public sector employment, in the community and at home.

Identify skills requirements mapped within a high level cross-cutting

framework, complementary to the definition of National Occupational



opportunities & barriers relating to definition, delivery and achievement of awards.

This paper is drawn from the resulting public report, ‘Next Generation User

Skills (NGUS) — Working, Learning & Living Online in 2013’ (September

2008), which highlighted opportunities and barriers, mapped to national credit and qualification frameworks and to available awards. This paper provides:

An overview

of the current ICT user skills landscape.

A model

representing the digital activities and competencies that might constitute the ‘Next Generation User Skillscape’.

A mapping

of that activity space on to tools and awards, with a


analysis identifying weaknesses in provision.

An overview of the recommendations to the report sponsors.

Appendices to this paper covering Recommendations, Stakeholder

Consultation, the Awards Mapped, and the Frameworks / Standards


David Kay (ed.), Bob McGonigle, Walter

Patterson, Barbara Tabbiner

Sero Consulting Ltd, Shefield Technology

Park, Arundel Street, Shefield S1 2NS [email protected]


Possibilities for the digital literacies required in everyday living, learning and working in the United Kingdom in 2013.

The world is awash with statistics on the impact of the web on 21st century living, learning and working. They are accompanied by the pronouncements and predictions of experts from every camp, from those heralding a new brave new world of co-creation and choice, to warnings from the dark side in terms of ethics and educational standards, tempered by increasing recognition that

‘we’ may no longer be in control of such matters.

Meanwhile, surveys report that around

80% of businesses have invested in

IT and 60% have websites. Whilst the extent and value of adoption differs signiicantly across businesses and sectors, the foundations for new ways of working and doing business are broadly in place, with older and static businesses typifying the laggards. It would not be unsafe to project that, by

2013, even more people will be required to use ICT in the workplace and in their everyday lives, increasingly involving online communication and web-based applications.

This represents a scenario to which those responsible for developing curricula and awards must respond — in the primary and secondary phases, vocational and applied learning, work based and adult community provision and higher education.

To ensure the relevance of and to inluence the ongoing enhancement of user ICT provision and the associated awards, Digital 2020 and the Scottish Qualiications Authority jointly commissioned Sero Consulting to develop a vision for ICT user skills in 2013 — ‘Next Generation User

Skills’ — taking account of:

Skills that all employers will need, which they may not currently recognise.

Skills that people (especially young

■ people) will already have, but which may not be accredited.

Essential skills for living and learning in a digital age.

This paper is drawn from the resulting public report, ‘Next Generation User

Skills — Working, Learning & Living

Online in 2013’ (September 2008), which provides:

An overview of the current ICT user skills landscape.

A model representing digital activities and competencies that might constitute the ‘Next Generation User


A mapping of that activity space onto tools and awards, with a gap analysis identifying weaknesses in provision.

An overview of the recommendations to the report sponsors.

Landscape are available at, along with the original report.

Why Next Generation User Skills?

Evidence of change

The world is awash with statistics on the impact of the web on 21st century living, learning and working. These are accompanied by the pronouncements and predictions of experts from every camp, ranging from the denial of real educational or economic significance through to the heralding of a new brave new world of co-creation and choice (learner led education, consumer as contributor, etc). All this is accompanied by persistent warnings of the dark side in terms of ethics, educational standards and employer requirements — tempered by increasing recognition that ‘we’ may no longer be capable of controlling of such matters.

In the Spring of 2008, several major research pieces were launched:

An in-depth summary of access to the internet and its uses across the UK population in the Oxford Internet Institute’s Internet Surveys (www.oii.

Ofcom’s Social Networking Research ( literacy/medlitpub/medlitpubrss/socialnetworking) informed the research carried out by Dr Tanya Byron for her Review of the risks (

uk/byronreview) faced by children in their use of the internet and video games.

Ofcom’s Media Literacy Audit ( literacy/medlitpub/medlitpubrss/children) of the UK, with separate analysis for Scotland.

■ e-skills UK’s IT & Telecoms Insights research (, covering industry trends & skills implications.

The University College London ‘Google Generation’ ( whatwedo/programmes/resourcediscovery/googlegen.aspx) higher education study, jointly commissioned by JISC and the British Library.

Meanwhile, from a business perspective, the third annual Yorkshire and

Humber survey of business ICT adoption (Yorkshire Forward, 2008 — www. indicated that 82% of the region’s businesses have invested in IT, of whom three quarters (60% if all businesses) have websites. Whilst the extent and value of adoption differs significantly across the range of businesses and sectors, the foundations for new ways of working and doing business are broadly in place, with older and static businesses making up large numbers of the laggards.

It would not be unsafe to project that by 2013 even more people will be required to use ICT in the workplace, increasingly involving online communication and web-based applications.

What we do and how we do it

It is widely recognised that these changes are, for increasing numbers of people, impacting on what we do with ICT (especially online) and also how we do it.

At the start of the NGUS investigation, the team considered the wide range of existing online services as well as business and education practices, to draw up an indicative map of ‘what we do’ with ICT. In Figure 1 they are loosely clustered in the diagram to represent thematic associations

(e.g. record keeping, shopping, selling, paying bills), with the vertical axis




Figure 1: Activities and capabilities ladder


Video editing

Being known



Making returns

Finding jobs



CPD updating



Revision Gambling

Voicing opinion


Health check

Accessing services

Appoint ments

Reporting crime

Commu nities

Finding people

Paying bills

Personal inance


Record keeping




Getting around

Entertain ment



Keeping in touch


Text msging


Figure 2: ICT mediation model

Working indepently and conidently online to acquire and reference information










Working accurately and eficiently with data and media to create meaningful outputs




Working safely and responsibly with others online towards a shared outcome








Sharing output with others


Figure 3: Potential shift by education sector

IN 2008





Silver surfer

Next Generation User






BY 2013

Ofice worker

Silver surfer




Ofice worker

Traditional Application


indicating levels of capability and confidence (e.g. finding out about the weather is lower level than finding jobs).

There is also recognition that the web is transforming the way we go about many activities. The ICT mediation model (Figure 2), originated by Genny

Dixon of e-Skills, illustrates the interplay between the online / offline and the collaborative dimensions. For example, what was once much more private

(such as buying books or insurance) has been enhanced by the involvement of strangers in the form of ratings and reviews. This significantly changes the everyday requirements for ‘digital literacy’.

System response to change

It is increasingly recognised the second generation of web-based applications (defined loosely as ‘web 2.0’) has significant implications for harnessing information technology across the education system.

The impact of learner expectations and the implications for styles and modes of learning, teaching and research are already recognised to be crucial considerations within the education system, as evidenced by the

Becta commissioned research into Web2.0 technologies for learning at Key

Stages 3 & 4 ( rp_02&rid=14543) and the JISC / BL commissioned ‘Google Generation’ higher education study ( resourcediscovery/googlegen.aspx).



It is however possible that the implications of what appear to be largely

‘social’ or ‘personal’ applications (ranging from casual communication in Twitter to platforms such as Personal Learning Environments) are underestimated in terms of economic value to UK plc.

Will these methods of working (of communicating, collaborating and contributing) become core skills and attributes in the world of employment by the time, for example, current 11 year olds leave school — the essential

Personal Learning & Thinking Skills for 2013?

Can the education system and its qualification frameworks play a vital role in harnessing the native ICT capabilities of young learners not only for teaching and learning but also to catalyse the workplace skills of the future?

Assuming that young people will accept the incorporation of ‘their’ skills in to education, this will depend on a number of complex and often systemic factors:

Do the

behaviours of digital natives fit the purposes of education and

■ employment?


teachers and lecturers across subject areas capable of supporting

■ and adding value to such ways of working?

Are they compatible with

curriculum design and assessment methods?

Will the

risks be surmountable in terms of safety, quality and other ethical issues?

Assuming that is the case, designers of curricula and qualifications for young people must look ahead to develop the ’next generation’ technology based competencies that will be required in the workplace of 2013. As illustrated in Fig.3, that will involve shifts of emphasis in all education and training sectors, and not least in secondary school and college provision.

However, this NGUS investigation has underscored the complexity of the operating environment in which curriculum and qualifications are developed, not only in the UK but in the wider European context (www.ecompetences.

eu/site/objects/download/3871_071011eCompParis.pdf). The identification and implementation of change in the IT User skills space is perhaps uniquely complex on account of the speed and uncertainty of technological change, coupled with differing rates of adoption and therefore levels of perceived need across stakeholders in industry and education.

This is illustrated when we consider the potentially conflicting and sometimes complementary influences on this ‘skillscape’:


— software tools and online services are developed ahead of need or demand and are therefore not widely recognised in industry consultation.


— as in any education system, the awards have a critical mass and lifecycle of their own. They may furthermore be addressing the same space from different educational perspectives.


— the range of agenda and levels of urgency cannot be easily reconciled. Consider, for example, the tension between volume demand for entry level ICT skills in such as the NHS with the Leitch (www. ryID=21&ContentID=37) focus on Level 2 and beyond.

Real world activity

— Meanwhile, everyday people from silver surfers to primary learners are inventing their own digitally enabled workflows and learnflows.

A future for IT awards

These challenges raise the issue of the relationship between the development of these ‘next generation’ ICT and digital skills and the ICT curriculum itself.

Curriculum & Awards

Awards based speciically on ‘ofice’ applications are unhelpful

Low levels of real information literacy are a core concern

ICT Functional Skills are the new user skills benchmark

The ITQ can incorporate any new user skills as they evolve

The ITQ is for occupational skills, so it should steer clear of Web 2.0

User skills are not an issue for the

‘Google Generation’, so awards are unnecessary

User skills should be embedded in other curricula

Character & Scope

User skills awards need regularly updating

User skills need to be industry or role speciic

User skills are generic and independent of purpose

User Skills are not about competence in isolated applications

The Internet and Web 2.0 is transforming how IT is used at work

User skills should encompass content creation and social software

Instruction & Pedagogy

User skills should be treated as an everyday language

User skills should be taught remedially not generally

User skills delivery should be scenario and project based

User skills are best developed through informal learning

User skills beyond applications cannot be reliably assessed

Instruction and Web2.0 skills are incompatible

Teacher and instructor capability is the critical success factor


1. Acquire — download data, media, software

2. Buy — goods, services

3. Calculate — cost, business plan

4. Capture — sound, image

5. Collaborate — with a group for work or leisure

6. Communicate — with one or more people (asynchronous), report something

7. Compose — text (e.g. a message, a document)

8. Create — edit, combine digital media (e.g. a ilm, a podcast)

9. Disclose — my identity, personal details

10. Explore — a simulation, scenario, projection or role

11. File — store information, records

12. Illustrate — a document with an illustration or layout

13. Learn — school, CPD, personal interest

14. Meet — conference or other synchronous activity

15. Navigate — ind and travel to places (i.e. using maps, GPS)

16. Organise — an appointment, meeting, project

17. Present — information

18. Publish — a digital artefact, a website, a podcast

19. Reference — something for the future (e.g. bookmark)

20. Register — for a public, commercial or open service

21. Search — for information

22. Sell — goods or services

23. Share — information, recommendations, media, other social networking

24. Survey — gather information, elicit votes

In England, the widely reported disinterest in ICT as a subject amongst digitally adept learners from KS3 onwards is variously attributed to a combination of the content of curriculum and the capabilities of the teachers. Whilst the 14–19 Diploma in IT suggests new curriculum possibilities, the supply of teachers may be worsening. Graduate Teacher

Training Registry statistics for 2008 ( applicantstatistics/2008) show a significant decline in IT applicant numbers for England, Scotland and Wales.

In the light of these trends, are there grounds to challenge the role of ICT as a discrete subject area with its own ‘user’ qualifications if schools and colleges are to harness information technology for new modes of working and living, as well as for learning and teaching?

During the Next Generation User Skills research, a number of challenges were raised regarding the nature of ICT curriculum and awards and the associated pedagogical challenges. The following statements (drawn from the consultation Topic Guide) represent issues to be taken in to account by those developing educational policy or designing curriculum and qualifications relating to ICT user skills.

When consulted, delivery organisations and practitioners almost universally recognised the continuing tendency to structure awards in silos which do not represent emerging user activity or its likely trajectory. As illustrated in

Figure 4, traditional qualification silos are symptomatic of



Isolation of activity workflows — for example, office v. networking or media v. office.

Outdated levelling — for example of infrastructure and even digital media skills in higher level professional awards.

Awards such as DiDA (Edexcel) and NC Digital Media Computing (Scotland) suggest new approaches, but still fall short of the near total convergence of the ‘participation-media-admin-infrastructure’ skillscape required of the 21st century digital citizen. Too much is left to the imagination and experience of the tutor in respect of NGUS- related emphases. Furthermore the size of awards and the combinations necessary to cover the NGUS space (e.g.

Digital Citzen plus Digital Cre8or plus ECDL from the BCS portfolio) are unattractive in terms of both funding and learning models.

Figure 4: Qualiications silos


Digital media


Book keeping



Social networking

Living in a digital world

Participation Media

Ofice user skills


Computers for the terriied





Figure 5: NGUS ecosystem





Real world things that people need to do


Capability to choose and use tools to perform activities


Software applications that enable or generate activities

Figure 6: NGUS competency model


Personal e-conidence









A. Enquiry

A1. Formulate questions as online enquiries

A2. Find, gather and collate information

A3. Research & evaluate on-line content and services

A4. Manage references (e.g. bookmarks) in context

A5. Explore a virtual scenario or simulation

A6. Use information to support decision making

B. Digital Literacy

B1. Understand on-line safety, security and privacy

B2. Recognise social responsibility


B3. Understand and respect digital property rights

B4. Compose communications to suit target recipients

B5. Learn critically from reviews of published work

B6. Organise, format and enter data

C. Participation

C1. Communicate and share information

C2. Create and maintain an online identity

C3. Submit ratings, reviews and recommendations

C4. Contribute appropriately to networked community activities

C5. Use shared applications

C6. Work collaboratively online towards a goal

C7. Moderate and manage the activities of an online group

D. Production

D1. Create digital artefacts

(diagrams, designs)

D2. Capture digital media (visual, audio)

D3. Edit digital media (visual, audio)

D4. Integrate (mash-up) applications and content

D5. Publish digital content (web,

PDF, e-book)

D6. Enable content to be discovered online

D7. Control versions of digital assets

NGUS ‘Skillscape’

Skillscape components

Section 2 recognised that the operating environment for IT skills and associated qualifications is made especially complex on account of the dynamic roles played by

Tools — emerging and sometimes disruptive technologies, manifested in software and services which can generate as well as facilitate activity.

Activities — evolving user behaviours, evidenced in personal and shared workflows and learnflows.

Tools and Activities are therefore not simply industry mandated, as top down responses to business requirements. The interplay is that of a dynamic and chaotic ecosystem, which drives the definition of and demand for

‘Competencies’ — the capabilities required to deploy the best combination of tools in desired activities.

The following sections describe the emerging landscape of activities, competencies and tools, which are indicative of the requirements for curriculum and qualifications on the 2013 horizon.


A core set of 24 ‘real world’ ICT enabled activities are proposed, listed in alphabetical order. They represent the activity landscape introduced in Section

2.2. Whilst some of the activities can be undertaken at several levels (e.g.

Communicate, Illustrate), others are more easily ‘levelled (e.g. Acquire, File).

Whilst there are many alternatives and potential additions, it is suggested that a group such as this provides a sound indicative framework for assessing requirements and evaluating awards.


The NGUS investigation distilled competencies into 34 cases divided into 5 groups:

As illustrated in Figure 6, two competency groups which represent the underpinning foundations of personal e-confidence are assumed to be required by all users:

Digital Literacy — including safe and social conduct.

Digital Independence — including management of the IT environment.

These support three broad and complementary areas of competence:

Enquiry — including the ability to investigate resources.

Participation — including the ability to collaborate.

Production — including ability to create media.

E. Digital Independence

E1. Understand technology operations and concepts

In 2008, each NGUS competency group may be

E2. Install, link and network hardware required to a greater or lesser degree by different user

E3. Install and update software types. However, each group is becoming increasingly integral

E4. Manage personal infrastructure and data to personal effectiveness in living, learning and work. The

E5. Use a range of digital and interactive devices

34 competencies are are shown opposite.

E6. Make appropriate ICT tool selection

E7. Explore and self-learn digital technologies

E8. Synchronise devices and data




A mapping to broader frameworks of life skills offers an important reflection on the coverage and significance of the 34 NGUS competencies. The following table maps the five NGUS groups on the Scottish Curriculum for

Excellence (, the Personal

Learning & Thinking Skills (PLTS, and US

NETS framework (

Independence Enquiry NGUS

Competency Group

Curriculum for Excellence

(LTS, Scotland)

Successful Learners

Effective Contributors

Responsible Citizens

Conident Individuals

PLTS — Personal Learning & Thinking

Skills (QCA, England)

Independent Enquirers

Effective Participators

Team Workers

Creative Thinkers

Relective Learners


NETS — National educational

Technology Standards (ISTE, US)

Creativity and Innovation

Communication and Collaboration

Research and Information Fluency

Critical Thinking, Problem Solving,

Decision Making

Digital Citizenship

Technology Operations and Concepts



Like the categorisation of activity and competency above, any listing or grouping of software tools will always be incomplete and contentious. Again, however, an indicative list is helpful in determining the shapes and hues of a constantly changing landscape (or ‘skillscape’). It deliberately contains just one example in each case. Depending on the way a user interacts, some tools are classified as ‘applied’ (e.g. Google or e-Bay).

Participation Production


Mapping of activities and tools

Each of the three elements of the skillscape can be mapped against the others. For those involved in the development of IT user curricula, it is perhaps most thought provoking to consider the mapping of tools against activities.

Production Enquiry

Media Capture and Manipulation





Presentation and Publishing








Website Development

Word Processor

Publishing Design

Data Entry and Manipulation

Bulletin Board



Personal Information Manager

Collaborative Environment

Social Network

Media Library

File Sharing










Information Research and Knowledge Management




Information Gathering

Information Discovery

Learning Space

Scenario Simulation

Idea Recording



Internet Explorer

Survey Monkey

Instant Messenger



Bit Torrent



Sim City


Weight Watchers







(Alphabetic list)

Media Presentation Data Knowledge Collaboration


























OCCURENCES 5 4 6 5 3 3 3 6 5 6 7 8 14 18 6 8 8 3 6 7 6 7 6 16 15 6 2

























This example, developed by the project team, illustrates

The prevalence of some tools — pervading a large number of activities; compare, for example, browsers or social networking applications with the media tools.

The complexity of most activities — requiring mastery of multiple tools; compare learning or acquiring with calculating (whether with a spreadsheet or a simple calculator).

The activity / tool matrix represents, therefore, a powerful illustration to conclude this examination of the evolving IT user or digital citizen skillscape.


Regardless of the names of tools and activities with which it is populated, the mapping emphasises the rich competencies and the type of approach that will be required if awards are to support the learning and teaching of Next Generation User Skills. It underscores the recognition that the teaching of tools in isolation (regardless of how many are covered in the overall award) with a focus on ‘menu mastery’ does not address the NGUS requirement.




English NQF

Scottish SCQF

Mapping of competencies and awards

This section examines existing awards in the context of the proposed NGUS competencies.


A total of 101 awards (The 101 awards are listed in an Appendix to the original commissioned report, available at were identified in the ‘Next Generation User Skills’ space across the English and

Scottish qualification frameworks, ranging from entry level user awards to higher level professional awards covering topics of NGUS interest. Whilst omissions may be identified, the team is confident that these awards are representative of the current coverage.

The team reviewed the awards in order to identify the key awards which could be seen as defining the current and future shape of IT User curriculum and qualifications, based on the assumption that the broader mass of awards are strongly influenced by such ‘flagship’ and high volume exemplars.


Key award in shaping the future of user skills up to NQF Level 2 / SCQF

Level 5 within the current paradigm of ICT skills qualiications

Strongly relevant to the NGUS space in terms of level and / or content

Not strongly relevant in terms of level and / or content

Total Awards
















Total = 18

Total = 8

This exercise identified 26 ‘key’ awards (The 26 awards are listed in an

Appendix to this paper, available at, which became the focus of the subsequent mapping exercises:

Entry3 = 4

L3 = 1

L1 = 6

L4 = 4

L2 = 8

L5 = 4

The commissioned report maps and reviews these awards as follows:

Maps each of the 18 English awards on to the 34 NGUS Competencies (in

5 groups in Section 3.3); the newly developed e-Skills unit in Collaborative

Technologies is included as an indication of direction of travel.

Maps each of the 8 Scottish awards on to the 34 NGUS Competencies (in

5 groups in Section 3.3).

Gap analysis

This section focuses attention on the Next Generation User Skills gap by examining the subset of 19 NGUS competencies which are covered by 50% or less of the 26 awards. The table summarises those mappings in terms of the number of NGUS competencies covered by each award and highlights perceived gaps in coverage.

In addition, as detailed below, C4 (Networked Community) & E7 (Selflearning) might also be considered as gaps.




Score shaded indicates 25% or less coverage (4 England, 2 Scotland, 6 in total)

Competency bold italic indicates likelihood of poorer match than assessed here

NGUS Group Competency Statement





A3. Research and evaluate on-line content and services

A4. Manage references (bookmarks) in context


Explore a virtual scenario or simulation

A6 Use information to support decision making

B5. Learn critically from reviews of published work Digital Literacy






C2. Create and maintain an online identity

C3. Submit ratings, reviews and recommendations

C5. Use shared applications


Work collaboratively online towards a goal

C7. Moderate and manage activities of an online group







England out of 18









Scotland out of 8

Total out of 26






















Integrate (mash-up) applications and content

D5. Publish digital content (Web, PDF, e-book)

D6. Enable content to be discovered online

D7. Control versions of digital assets

Digital Independence

Digital Independence

Digital Independence

Digital Independence

Digital Independence

E1. Understand technology operations and concepts

E2. Install, link and network hardware

E3. Install and update software

E5. Use a range of digital and interactive devices

E8. Synchronise devices and data




























For the first four competencies (

enquiry, digital literacy, participation and

production), the same gap skills exist in Scotland and in England. In the final competency (

digital independence), there is variation between the countries with two gap skills in common and two in difference.

In the

enquiry competency, awards typically only require information to be found and used, and do not progress to evaluation of that information or the associated services (A3). The common research skill of managing references or bookmarks is absent (A4). Using information to support decision making, is also absent (A6).

In the

digital literacy competency, the skill of learning critically from the review of published work is weak (B5), being only present through teacher or peer review. Student and other course related work is widely published on the internet and thus this is an important omission. The gaps here and in

enquiry suggest that awards need to strengthen building of critical information literacy skills.

The present generation have online lifestyles — it is therefore a concern that the largest number of skills gaps appear in

participation. Managing an online identity (C2), rating and recommendation (C3) and contributing to social networks and communities (C4) are commonly used and essential skills, but are not systematically addressed in any of the qualifications on offer. Collaborative skills in using shared applications (C5), working


collaboratively online (C6) and managing an online community (C7) are almost universally missing. It is notable that the core skill of working together is present in the majority awards but the context is still in the real world as opposed to online.

Increasingly, online life (whether leisure, study or work) is not simply lived as a recipient or consumer; producing and sharing differing types of digital assets will be a key activity in the virtual world just as it is in the real world.

In the

production, gaps are generally present in the skills of integration

(D4) and more surprisingly of publishing (D5) and being able to control version of digital assets (D7). Having your content and services readily and consistently discovered in the virtual world will be essential but is also missing (D6). These skills gaps can be found in awards in specialised units; however this approach does not address a world in which publishing and integration are increasingly in the hands of users.


digital independence, Scotland and England share the gap skills of installing & updating software in general user units (E3) — surely required by every computer user — and of learning about digital technologies (E7).

In England, gaps exist in understanding technology operations & concepts

(E1), which is weakly covered, and dealing with network hardware (E2). In

Scotland, gaps are to be found in using digital devices such as mobile phones

(E5) and synchronising such devices (E8). The gaps here are significant as they relate to commonplace tasks.

In summary, these gaps indicate that award construction needs to take more account of maturing information literacies and of the common practice and demands involved in ‘living online’. It is also apparent that many of these competencies are not derived from teaching and testing in applications (such as an office toolset). The practice of these ‘new’ competencies (which may be superficially associated with web 2.0, but are of wider import) is typically integrated in practice, lending itself to project based curricula.

Incorporating these competencies will present significant challenges to those who develop and deliver qualifications. Not only is the rate of change in online technology and capabilities is high (requiring a move towards annual updating of portions of teaching guidance), but also instructors may feel challenged by aspects of the associated practice (not simply technique) and assessment organisations may find it hard to address economically something less suited to hitherto successful online testing models.


Addressing the NGUS challenges within the context of national curriculum and qualifications strategies requires a systematic approach, which covers:

A definition of the


The development of delivery


The design and introduction of new

awards and assessment models.

The recommendations arising from the Next Generation User Skills investigation are therefore framed in the form of an end-to-end ‘action plan’, such as might be adopted by an agency responsible for both curriculum and awards development. In this respect the SQA is in a special position, as illustrated in its development of the PC Passport suite. Individual stakeholders (such as a Sector Skills Council, a toolset vendor, an economic development agency or those responsible for teaching standards) may play their parts in such developments through national, regional and local partnerships.

The seven recommendations are structured over an imagined four year timeline (Figure.7), further detailed in the commissioned report.



Figure 7: Recommendations overview







7. Embed NGUS in other subjects

2. Make vendor linkages

3. Develop competencies and scenarios

1. Offer tutor guidance/CPD

Skillscape Capability

6. Validate Next Generation

Digital User Awards

4. Introduce project / portfolio assessment

5. Validate units to ill key gaps




Sandra Romenska

Beyond Distance Research Alliance,

University of Leicester, UK [email protected]


Higher Education Institutions (HEIs) operate in a borderless and complex environment, abundant in potentially useful information. The Creating

Academic Learning Futures (CALF) research project, carried out in partnership by the University of Leicester and University College Falmouth in the UK, involves the development of approaches and tools for structuring and iltering information, in order to facilitate institutional decision-making in participative and creative ways.

One of the aims of the CALF project is to involve students in creating and exploring a variety of plausible

‘alternative futures’ for learning and teaching technologies in higher education. This paper discusses some of the issues that are emerging in the course of the research process and presents ideas for the future, grounded in and emergent from ‘student voices’ from the CALF research project. Students expected the technologies of the near future to enable them to become cocreators in their own education processes.

The future scenarios imagined the rise of learning technologies which instead of becoming outdated with use, become more valuable as more user-generated content is invested, technologies which are truly learning in that they learn about their users and constantly morph / adapt to their users’ needs. Finally, increasing virtualisation was a recurrent theme across most student-generated scenarios.

The paper concludes with a discussion of some of the strengths and limitations of using technologies for involving students in creative activities for generating future scenarios for higher education. The technologies used by the project enabled collaborative creative thinking across a broader spectrum of possibilities about the relationship between the present and the future of higher education.

0164 The learning technologies of the future: technologies that learn?


The first universities were institutional innovations which emerged in 12th to 14th century Europe as a result of the need to consolidate and expand intellectual resources in response to increasing demands for knowledge and skills in the economy and society (David, 2006). Despite debates as to whether universities have remained these “medieval organisations,” unchanged over the 700–800 years of their existence (Clarke, 2003; Kerr, p.152, 1982) or have been transformed by major changes (Clarke, 1996;

Kyvik, 2004), consensus seems to prevail about intensifying pressures for reform in HEIs today (EC, 2003; Aghion, 2007; LERU, 2006). Technological, financial, political, regulatory, demographical, cultural and psychological factors bring major challenges to twenty-first century higher education and its governance systems, curriculum, mission focus, external relations, research and financing. While these challenges can be viewed as both threats and opportunities, it is important that planning and management are not dominated by short-term thinking about immediate problems and maintaining established practices. Neglect of the long term is increasingly problematic in meeting the challenges of complexity and change in higher education. In order to be able to look beyond the constraints of the present, especially when the investment of significant resources is concerned, HEIs need to sharpen their capacity to systematically explore and connect together various driving forces, trends, and conditioning factors so as to envisage alternative futures (Lancrin, 2004, OECD, Notten,

2006). Involving today’s learners in a dialogue about the future of learning is essential for ensuring that strategies for the future of HEIs take into account changes in learners’ expectations and cultures. Engaging the learners of today in imagining the future can provide platforms for strategic conversation even between those who may sometimes be considered to be worlds apart (Goudet et al. 1996). This dialogue may enable the future to be created collaboratively rather than predicted (Lancrin, 2004, OECD,

Notten, 2006) — evidence from cognitive psychology suggests that mentally simulating alternative future visions can influence future behaviours (Parks et al., 2003; Sanna et al. 2003; Roese et al. 1995).

This paper is structured in two parts. The first discusses the conceptual and methodological challenges of researching student perceptions of the future of learning. The second presents some initial ideas about the future of learning, grounded and emergent from a research project which aims to uncover, model and represent student ideas about the future. The CALF project is led by University College Falmouth, and the research is supported by the University of Leicester. CALF involves staff and students from the two institutions, creative partnerships with other HEIs, international organisations, corporate, research and technology partners.

Conceptual framework

CALF provides learners with opportunities to surface and articulate views about the future of learning in HEIs. It uses creative events in real and virtual environments, social networking and web tools to encourage learner engagement in creating alternative futures through the acts of authoring content, collaboration and participation. CALF uses the paradigm of future studies to explore questions about student ideas for the future of



2 learning — about who is going to learn what, how and where in the future

30 years from now.

Futures studies as a strategic management framework have developed over many years, originating from the writings on alternative futures of Herman

Khan for the RAND organization in the 60s (Burt, 2007). Futures studies work seeks systematically to explore and connect together various driving forces, trends, and conditioning factors so as to envisage alternative futures

(rather than predict the future). As a result, long and short-term policies and strategies can be produced, which then can in turn enable people to create a desired future (Edwards, 2007).


There are multiple approaches to futures studies, and one of the most widely employed, although contested, is scenario development. Defined as “plausible, challenging and relevant stories about how the future might unfold,” scenarios incorporate quantitative models with qualitative assessments of social and political trends (Raskin, 2005). On the continuum of analytical tools, they come between deterministic quantitative models of the future and purely narrative descriptions (Nakicenovic et al. 2000).They can refer to both descriptions of possible future states and descriptions of developments.

From a cognitive perspective, creating futures is not only a problem of a discrepancy between a present state and an imagined state. Choosing how to describe the discrepancy at a particular time will determine what future will be created, therefore to understand young people’s visions of the future requires uncovering the underlying logic and assumptions of present realities and policies and presenting them in a format open for questioning and challenging (Edwards, 2007). It is necessary to provide learners with opportunities to develop a “foresight language” for what is in essence critical discourse analysis for exploration of future states. In selecting the future envisioning methods some key requirements need to be met to ensure that a range of perspectives is captured so that there is potential for different discourses to emerge. In this way, truly divergent alternative scenarios can be developed, in line with the definition of the scenario method:

“Scenarios are consistent and coherent descriptions of alternative hypothetical futures that relect different perspectives on past, present, and future developments, which can serve as a basis for action.” (OECD, Notten, 2006)

The CALF project partners believe that there are better ways of engaging with students than by seeing them as customers or consumers, and that creation of future scenarios in collaborative activities is one of the richer, more accessible and useful approaches to futures studies compared to conventional questionnaires or interviews (Wildman, lnayatullah, 1996;

Salmon, 2008 ). The emphasis on group work and collaboration in the choice of futures workshops as one of the research instruments in CALF is based on a number of assumptions. It is hoped that group work will help establish a shared sense of ownership of the created scenarios for the future of learning, as to what is feasible and desirable (Cunha et al., 2007). Also, collaborative creation of scenarios involves the establishment of networks among the participating students, allowing them to share awareness of each other’s knowledge resources, ideas, and visions of the future.

The present paper reports on data collected during three creative events, held between November 2008 and April 2009, involving 29 students. Every event is structured in a way aimed at introducing students to a variety of ways of thinking about the future of learning and helping students build a vocabulary which would support discussions about the future. The first two events, involving 26 students, used a specially developed wiki, integrated into the event. The third event, involving 3 students, was set up in the virtual world Second Life.


The wiki event format

The objective of the CALF project wiki was the production of series of generic scenarios, created by inviting students to form a narrative from a series of statements about how they saw the future of learning in higher education. Wikis “enable rapid and easy authoring direct to the Web. Wiki pages can be used by all to publish new content direct to the Web, including text, images and hyperlinks; and to edit existing content (Wheeler et al.,

2008). Students can develop their own knowledge content with alacrity using a wiki and seldom need to study alone because of participation in a technologically mediated social space conducive to the formation of communities of practice” (Wheeler et al. 2008). Every page on a wiki is created and editable through the web using a web browser and therefore wikis express a high point in the attention to the connection between community and content, thus offering a way to implement in research practice the core principle of the CALF project that learners are not mere

“receptacles” of ideas but participants in the dynamic creation and discovery of what is to be learned. This lead to the choice of wikis as an approach to developing future scenarios, in the process making learning outcomes contextualized and relevant.

The format of the events involved a discussion of digital and Web technologies existing at present and ways in which they could change the future of learning. The participating students were encouraged to think about the likelihood of future scenarios and searched the web for images, videos or applications that they associated with a particular scenario. As a result of the activities the students learnt how to use wikis and created wiki scenarios for future learning in higher education which have since been made available on the CALF project wiki:

The use of the wiki enabled collaborative creative thinking across a broader spectrum of possibilities about the relationship between the present and the future of higher education. New ideas emerged in a way that would not have been possible if conventional scenario planning methods were used.

The use of the wiki allowed the replacement of the traditional snapshot and chain portrayals of scenarios by a network, which allowed the seamless integration of multiple views of the present and the past, occurring in multiple systems (e.g. global and local). A fractal “leaf of goals” metaphor best represents the functioning of the wiki as a scenario tool, where a fractal is the whole which when split into parts, each part is (at least approximately) a reduced-size copy of the whole. This property of the CALF future scenario wiki illustrates the continuum where activities, events and objectives lie and incorporates the assumption that any one event is itself a composite of an indefinite number of component events that would have been very difficult to capture without the use of the wiki. The collaborative scenarios created through the wiki emphasized technological change without overlooking social change, thus escaping a common criticism of conventional ways of scenario development.

The collaborative creative character of the wiki tool addressed another shortcoming of traditional scenarios — the time they usually take to develop.

The combination of web-based and face-to-face activities allowed students to collaboratively generate, mix, edit and synthesise scenarios within a shared and openly accessible digital space.

An important advantage of the use of a wiki for developing future scenarios was that it allowed storing of the narratives, comparing them and deriving generic scenarios by combining common elements for possible, probable and desirable futures. It allowed interventions in fluid and informal creative ways.

Data about the ideas of the participating students of the future of learning were collected through observation and note-taking during the student



2 discussions at the wiki events and analysis of the text of the scenarios created on the CALF project wiki by the students at the two events.

Second Life event

The third CALF event took place in Second Life, the 3D virtual world where users can socialise, connect and create using voice and text chat. The choice of setting the event in Second Life was determined by one particular challenge of future studies — “the cost of thinking”.

This challenge is one which those working in the area of consumer research call “the finite or quantal choice problem.” It refers to the difficulty people have in comparing diverse alternatives. According to theories about the cost of thinking in choice problems, when having to make choice between alternatives people form perceptions by acquiring information about each alternative and then processing this information to arrive at an expected utility (Shugan, 1980). The comparison between the characteristics of the alternatives will be associated with a cognitive effort — the characteristics are evaluated and their differences assessed. Therefore, the more comparisons are required to make a choice, the more difficult the choice — the cost of thinking. Determination has costs — ubiquitous information, numerous alternatives, time pressure, limited information processing capabilities, and the general effort exerted to solve the problem. Choice theorists say that generally, the net utility of finding the best product from one set of products may be different from the net utility of finding it as best from another set of products. That is, there may be a cost associated with the act of making a decision—the “cost of thinking” (Murrey et al. 2007)

In the case of the CALF project, the cost of thinking that participants have to pay is significant — they have to imagine possible futures of learning, to compare them and to make a choice of their preferred future. The comparison is between entities or concepts — futures — which do not yet exist—either complete new systems or new states of existing systems. This represents a relevance gap.

A number of properties of Second Life offer a way of addressing this challenge. Second Life provides a “sandbox” (Salmon, 2009) where participating students can compare alternatives and characteristic which are not that distant and abstract any more. By providing interactivity within the environment and a ‘feeling’ of presence and immersion, dialogue and encounter, Second Life allowed the participants in the CALF project to visit and immerse themselves in learning locations and cultures in a way that is not possible in real life. It was hoped that in this way it could give a very real sense of a possible future for learning technologies. The experiences in

Second Life can provide a platform for the creativity, imagination and viable innovation in engaging with the technologies and pedagogies of the future that can reduce “the thinking cost” of having to compare alternative futures.

During the Second Life CALF event, the three students entered Second

Life for the first time and explored different sites — the Beyond Distance

Research Alliance Media Zoo, the replica of the Sistine Chapel recreated by Vassar College and NASA’s moon probe launch site. After the event the students were interviewed individually about their experiences in Second

Life and their ideas about the future of learning.

The next section of this paper presents a summary of the visions of the future of learning of the students who participated in the CALF project events.

Visions of the future of the students

The issues which emerged from the interviews were quite varied, and ranged from the particular to the general. The initial analysis of issues did not


contain strong evidence of a consensus or of common themes. The issues raised in the interviews, however, could be grouped into several areas, each of which included a number of perceived trends.

The initial scenarios proposed by the students were centred on ideas about increased flexibility in the provision of education, increased accessibility and participation in higher education in the future which would lead to an increase in the diversity of the available educational content and the student demographic profile.

“80% of the population today is enrolled in a programme of study and since all learning content became free, producers receive their income from advertising and donations.”

“80% of the population today is enrolled in a programme of study, this means our society is more educated. I am studying full-time at the University of Leicester and today the vice-chancellor announced that the university would invest 3 million pounds in renovation of the student accommodation blocks which means I can study in a safe and comfortable environment with my friends.”

Quotes from the student scenarios

The students envisaged that the stake of non-traditional providers in higher education would grow and that the competition between HEIs would increase, leading to a fall in the cost of education:

“Prof. Lindsey returned our assignments today — I have done well. I think this is because I used external resources in addition to my traditional degree structure which has enabled me to learn when it’s convenient for me.”

Quotes from the student scenarios

At the same time, a recurrent theme across the student scenarios was the expectation that education will be a continuous process, with the concept of “completing education” disappearing. Interestingly, the driver for the disappearance of “an end to education” was not conceived to be the pressure of ever-increasing amounts of information that will come in the future. The desire “to always learn new things” was also identified as a driving force, coupled with the expected low-cost of learning and the enabling power of technology to deliver learning conveniently to the individual needs of the students:

“I found my grandmother’s graduation photographs today. I keep thinking what a funny thing this “graduation” must have been. How could they have assumed they could “graduate” and inish “education”? If I want to be employable, I need to spend at least ¼ of my week in learning new things, otherwise I will fall behind.”

“It is fun, learning new things. People like learning something new, always, so if you can learn anything, I mean with technology, why stop learning? People will learn more in the future.”

“All learning content became free. Producers receive their income from advertising and donations.”

Quotes from the student scenarios

The participants in the event expected that the use and importance of technologies for education would increase and that the role of usergenerated content, social-networking, peer assessment and referencing, and the use of interactive and participative approaches to teaching would also grow. As part of this process students expected that the rise of learning technologies, which instead of becoming outdated with use, become more valuable as more user-generated content is invested into them, and that the technologies will become “truly learning” in that they learn about their users and constantly morph / adapt to their users’ needs — the way that Amazon, iTunes or Youtube recommendations work today. One student gave as an



2 example of “a technology that learns” his iPod because he had invested time, effort and resources to personalise it and now it “knows” about his preferences and style, thus becoming more valuable with use.

“One of my Facenote contacts also has an interest in Shakespeare; we got in touch after we discovered we had tagged the same course components on Youtube.”

“Anybody can add to and change educational resources. You can check their quality by the number of times they have been favourite, tagged and recommended.”

“I worked with Abel from Argentina and a Katya from Russia on a task that Glaxo

Welcome had posted on the examination discussion board on Facenote. We used translation software for the online discussions so all three of us could speak our native languages and still understand each other.”

Quotes from the student scenarios

An interesting projection was that HEIs of the future would need to be more involved in socially responsible projects and activities as part of their strategies for competition for students. Environmental and social considerations were the few areas of the students’ future narratives where the envisaged futures were not entirely optimistic. Expectations of HEIs addressing issues of environmental and social responsibility were present across all of the student-generated narratives about the future:

“All learning is now done [partly] on campus since computers were banned after the

UN Commission on Climate Change discovered that computers contribute greatly to global warming. “

“I transferred my studies from Kyoto University to the Sorbonne after Kyoto failed their recycling targets for 3 consecutive years.”

“I decided I will study Automated Chemical Synthesis with the University of Bath because they are supporting so many of the social causes I support — they are donating funds for HIV treatment in Lesotho, rural community development in France and literacy projects in Bulgaria.”

Quotes from the student scenarios


In this paper some of the methodological, practical and conceptual issues of developing future scenarios for learning with students were discussed. A few clarifications are in order to point out the limitations of this paper. As it is aimed at a futures scenarios study, it does not propose an exhaustive academic analysis of the current situation of HEIs. Rather, it offers a description of trends and ideas of possible futures for learning from the perspective of the students who took part in the CALF project. A challenge that the chosen approach presented was one common for scenarios — that generic scenarios are of little interest and of limited use to organisations precisely because they are too general. Also, a relatively small number of the students made a large number of contributions, while the majority of the students made only few. Despite these issues, the student experience revealed that future thinking events can be seen as vehicles for the empowerment of students, opening up new possibilities for thinking about the future. Some of the futures envisaged by the students focussed on technology, others on society, the economy and the environment. A common thread across all of them is the emphasis on the enabling role of democratic and participatory debate about the future of learning and the importance of the ability to think creatively and imaginatively in the construction of scenarios.



Aghion (2007) Why Reform Europe’s Universities? Bruegel Policy Brief, Issue 2007/04, available at:

Burt, G. (2007) Why Are We Surprised at Surprises? Integrating Disruption Theory and System

Analysis with the Scenario Methodology to Help Identify Disruptions and Discontinuities. In

Technological Forecasting and Social Change pp. 731–749

Clarke, B. (1996) Substantive growth and innovative organisation: New categories for higher

education research. Higher Education, 32, pp. 417–430.

David, P. (2006) Europe’s Universities and Innovation — Past, Present and Future. SIEPR

Discussion Paper No. 06–10

EC (2003) The Role of Universities in the Europe of Knowledge, Communication from the

Commission. COM (2003) 58 final, Brussels, 5 February 2003.

Edwards, M. (2007) “Every Today Was a Tomorrow”: An Integral Method for Indexing the

Social Mediation of Preferred Futures. In Futures, Volume 40, Issue 2, pp. 173–189

Godet, M. and Roubelat, F. (1996) Creating the future: The Use and Misuse of Scenarios. In

Long Range Planning, Vol. 29, No. 2. pp. 164–171.

Havas, A. (2008) Devising Futures for Universities in a Multi-level Structure: A Methodological

Experiment. Technological Forecasting & Social Change 75 (2008) 558–582.

Kerr, C. (1982) The Uses of the University. Cambridge, MA., Harvard University Press

Kyvik, S. (2004) ‘Structural changes in higher education systems in Western Europe’. In Higher

Education in Europe, Vol. 29, N. 3, pp. 393–409

Lancrin, S. (2004) Building Futures Scenarios for Universities and Higher Education: an

International Approach. Policy Futures in Education, Volume 2, Number 2, 2004

Lancrin, S. (2006) What is Changing in Academic Research? Trends and Futures Scenarios.

European Journal of Education, Vol. 41, No. 2, p..169–202

LERU (2006) Universities and Innovation: The Challenge for Europe. League of European

Research Universities, November.

Murrey, K., Häubl (2007) Explaining Cognitive Lock-In: The Role of Skill-Based Habits of Use in

Consumer Choice. In Journal of Consumer Research Vol. 34

Nakicenovic et al, (2000) Special Report on Emissions Scenarios: A Special report of Working

Group III on the Intergovernmental Panel on Climate Change. Cambridge: Cambridge University


Notten, P. (2006) Scenario Development: a Typology of Approaches, in OECD Think Scenarios.

Rethink Education

Parks, C., Sanna, L. and Posey, D. (2003) Retrospection in Social Dilemmas: How Thinking About

the Past Affects Future Cooperation. In Journal of Personality and Social Psychology, Vol. 84,

No. 5,pp. 988–996

Pina, M. Cunha, E. and Chia, R. (2007) Using Teams to Avoid Peripheral Blindness. Long

Range Planning 40 (6) pp. 559–573.

Raskin, P. (2005) Global Scenarios in Historical Perspective. In Ecosystems and Human

Well-being: Scenarios — Findings of the Scenarios Working Group Millennium Ecosystem

Assessment Series, edited by Carpenter, S. et al. pp. 35–44. Washington, DC: Island Press

Roese, J. and Olson, J. (1995) Outcome Controllability and Counterfactual Thinking.

Personality and Social Psychology Bulletin, N.21,pp. 620– 628.

Salmon, G. (2008) Future Learning: Desire or Fate? Paper presented at JISC Innovating e-Learning online conference 2008. Available at content2008/salmon/salmon.pdf [accessed on May 19th 2009]

Salmon, G. (2009) The future for (second) life and learning (forthcoming). In British Journal for

Educational Technology, Vol. 40, No. 3., pp. 526–538

Sanna, L. J., Stocker, S. L. and Clarke, J. A. (2003) Rumination, imagination, and

personality: Spectres of the past and future in the present. In E. C. Chang & L. J. Sanna (Eds.),

Virtue, vice, and personality: The complexity of behaviour (pp. 105–124). Washington, DC:

American Psychological Association.

Shugan, S. (1980) The Cost of Thinking. In Consumer Research Journal, Vol. 7, No. 2, p. 99–111

Smithers, R. and Woodward, W. (2003) “Clarke dismisses medieval historians”. The Guardian,

9 May 2003,, [accessed on

May 20th 2009]

Wheeler, S., Yeomans, P. and Wheeler, D. (2008) The Good, the Bad and the Wiki:

Evaluating Student-generated Content for Collaborative Learning. British Journal of Educational

Technology Vol. 39 (6), pp. 987–995

Wildman, P. and lnayatullah, S. (1996) Ways of Knowing, Culture, Communication and the

Pedagogies of the Future. In Futures, Vol. 28, No. 8, pp. 723–740




0123 When the shoe doesn’t it: supporting students who are challenged by online educational technologies


Building on a body of current research, the study reported in this paper explored the experiences of graduate level students who found a technologyenabled classroom challenging. At a small university in Western Canada that specializes in graduate and upper level undergraduate programs, MBA students worked in a blended learning environment. The MBA program began with a four-week online introductory session, followed by one of three on-campus residency sessions. After completing the residency component, students returned home and began the first distance learning session. The task of balancing career, family, and learning was challenging, and as a result, by the end of an eight month period, 15% of the students had left the program.

The aim of the study was to determine what tools could be incorporated in the online component of the graduate program that would support students in their learning. The specific objectives were:

To explore the challenges facing students as they worked in the online environment.

To discover which types of tools had enhanced the students’ work in the online classroom.

In order to achieve these aims and objectives, a two phase action research plan was developed and students were invited to attend a focus group to discuss their experiences in the online classroom.


Some individuals experience challenges making the transition into higher education from secondary schools and from the workplace . As a result, students may need assistance from the institutions if they are to succeed with their educational pursuits. In our study we examined the perceptions of students who had started an MBA program at a university in Western Canada in May 2008. These students had an average of 17 years’ work experience, and were, on average, 40 years in age. They came from a number of different geographic locations, both in Canada and worldwide, which meant the group of students had a wide variety of experiences and skill sets.

The graduate program operated in a blended learning environment, with the first classroom experience being an on-campus residency of three weeks.

Prior to the start of the residency, students were directed to online, notfor-credit, activities which included course readings and an orientation to the learning environment. This online session, ‘Getting Down to Business’

(GDB), was four weeks in length, and was primarily intended to prepare the students for their upcoming on-campus classroom experiences and for their studies in general. The activities included instruction and navigation of several university-based sites, readings related to the upcoming residency, assignment submissions via online drop boxes, and an opportunity to interact with one another in an informal discussion area. It was hoped that


Jo Axe, Donna March

2005 Sooke Road, Victoria, BC,

V9B 5Y2, Canada [email protected], [email protected]


Taking an action research approach, this paper explores the experiences of graduate level students who found a technology enabled classroom challenging. After taking part in an online orientation activity, the students began their studies in a three week face-to-face residency, followed by two distance learning sessions. At the end of the second session, 15% of the students had either taken a leave of absence, or left the program permanently.

Current literature focuses on several issues that have a negative inluence on students’ ability to achieve their educational goals in an online environment. These issues include isolation, lack of preparedness, and feeling overwhelmed. Building on this, we explored the following question: How can educational technology be used to increase retention for students enrolled in an online course? Aimed at identifying approaches to overcoming some of the problems associated with studying online, we sought to examine the perceptions of MBA students by asking the participants to take part in a focus group discussion.

This paper will discuss the challenges found by the graduate level students who worked in a computer-mediated educational environment, as well as highlight some of the solutions aimed at increasing student retention by enhancing the online classroom. Findings revealed that technologies aimed at encouraging student presence in the online classroom, as well as those that allow them to interact socially online could positively increase student retention. In addition, employing tools that support the students’ desire to repeat and relect on material serves to improve the experience for those studying online. Finally, instructors who display comfort with the use of technology also have a positive impact on student learning.

By providing a heightened awareness of the issues faced by graduate students working online, as well as spotlighting some key solutions, this study underscores the importance of targeting appropriate technologies when designing the online classroom. Further studies in this area could explore undergraduate engagement, generational differences, as well as learning preferences in the context of online learning.


this ability to chat online with other students would aid in the creation of a learning community which would support the students throughout the program. Once in the residency, students had face-to-face classes in four subjects. While the classes were held in a physical location at the university, students were expected to make use of online tools and use a Moodle-based educational platform to locate a variety of resources.

After completing the on-campus residency, students engaged in their first distance learning session (DL1), taking two online courses simultaneously.

This was followed closely by another distance learning session (DL2), again with two online courses, and then students returned to campus for their second residency. Figure 1, below shows the flow of the program.

Figure 1: Flow of program

GDB (online) Res # 1






Res # 2


Programme continues . . .


15 % AT T R IT I O N

Between the start of the MBA program and the end of DL2, 15% of the students left the program. While it was understood that students left for a multitude of reasons, we were interested in finding out how the students that remained perceived the role of technology in their choice to continue with their studies.

Current literature speaks to the power and effectiveness of the online learning environment. Palloff and Pratt discuss the mutually empowering act — a means by which people share with each other, work, and live collaboratively. However, there are also several potential concerns for students who work in the online environment. For example, Bender (2008) noted that the feeling of being overwhelmed can contribute negatively to students’ experiences in the online classroom . In addition, McInnerney and Roberts (2004), as well as McConnell (2006, 2000) focussed on the isolation experienced by some students in the online classroom; isolation that can cause students to feel dissatisfied with their choice of educational environment. While isolation may be counteracted by the development of a learning community, it could serve to exasperate existing problems , if not built and supported effectively.

Palloff and Pratt (2007) also discussed some of the key threats to success in the online classroom. They noted one of the main issues that had a negative influence on students’ ability to achieve their educational goals was conflict with classmates, conflict which may be compounded by asynchronous online communication tools . In an earlier work (1999), they also discussed students’ resentfulness at being asked to work with others. This can result in a variety of team-related issues such as mistrust, reduced participation, and even attrition .

Building on these observations, we explored the following question:

How can educational technology be used to increase retention for students enrolled in an online course?

Aimed at identifying approaches to overcoming some of the problems associated with studying online, we sought to examine the perceptions of MBA students by asking the participants to take part in a focus group discussion.




Using action research to frame the study, we developed an approach that included the four steps shown in Figure II: plan, act, observe, and reflect . As participants in the research we were able to not only observe the subjects, but also to enter into conversations that extended the information available.

Our plan involved two phases, the first of which was exploratory in nature, directed towards finding out the issues that concerned the students . The second phase has not yet been completed, but, based on the feedback provided in the first phase of this study, we expect to implement and examine the use of some additional online tools for students.

In Phase I, we planned our approach to data collection and analysis, completed an ethical review for the project, and identified potential participants. In addition, we gathered background information on program attrition rates and student demographics. With regards to the analysis, we wanted to allow themes to emerge from the data, rather than concentrating on predetermined categories . This desire for an emergent approach led us to pursue qualitative methods. We determined that a focus group, with a small number of students, would allow us to use semi-structured, open ended questions that would enable participants to explore the concepts, and allow us to probe more deeply to obtain thick, rich data. In addition, the focus group setting would allow us to study the interactions between the group members, and put the participants on a an equal basis with each other.

This was a key consideration as we wanted to insure that the participants felt comfortable discussing some of the problems they had encountered and bringing forward potentially contentious issues.

The focus group consisted of two males and three females, aged from 33 to 54; this represented 8.5% of students who started the MBA program in May 2008. The participants came from a variety of locations across

Canada, from British Columbia to Newfoundland. The focus group lasted approximately one and a half hours, with questions covering three domains:

(1) students’ expectations of online learning; (2) the challenges faced in the online environment; (3) tools that were or could be effective for students studying online. The first domain contextualized the discussion by allowing the participants to reflect on their perceptions of what online learning meant. In the second domain, students explored their experiences in the online courses. While some of the issues they exposed were content related, others provided insight on course design and online technologies. Questions asked under the third domain allowed students to elaborate on the answers provided earlier, focusing on the tools that they had found effective as well as their overall impressions of what is important to an online learner.

In addition to audio taping the focus group, we engaged a scribe to capture the key points raised in the discussion. After the scribe transcribed the handwritten notes, we independently reviewed both the tape and the transcripts, coming to independent conclusions on emergent themes. We then met several times to discuss the themes and how they were situated in current literature.


There are many reasons to explore the appropriate use of technology in the online educational environment. Traditional teaching pedagogies are making way for newer constructivist methods, where students are encouraged to work together in different learning environments. The use of computer mediated communication in distance learning continues to increase and, as

Dawson noted, there has been a change in student demographics — from those who have traditionally populated the more conventional, brick and mortar institutions towards a greater number of older students desiring

Figure 2: Phase 1 approach




Ac t

Observ e


a more flexible approach to their learning . Competition is increasing for distance learning programs and, therefore, the need for institutions to explore educational technology and its impact on retention in the online learning environment is critical.

By studying students’ experiences with online technology, educational institutions can be better informed when selecting online educational platforms. Understanding students’ perceptions of the online learning experience, as well as the tools that best serve the needs of these students, will be invaluable to universities and schools as they move to expand their methods of delivery. Furthermore, students considering online learning can feel confident that educational providers have done their due diligence by exploring research that validates and informs their decisions on the use of technology. Therefore, research such as this study can not only benefit members of society who are seeking higher education, but it can also benefit those who work institutions seeking to provide newer ways of delivering their programs.


At the time of the Phase I focus group, participants had completed the GDB activity, five on-campus courses taught face-to-face in Residency I, and four online courses. This allowed the participants to share their insights based on a solid understanding of both online and face-to-face delivery models.

Accordingly, the comments gathered for this research were based on the participants’ lived experiences . Several themes emerged from our analysis of the data: (1) presence, (2) review / reflect, (3) social context, and (4) skill sets. In this section, we will explore these themes in light of both literature and the perspectives of the research participants.

Twenty years ago, Feenberg (1990) wrote of personal presence in the online environment when he discussed the then “new phenomenon of computer mediated communication (CMC)” . Feenberg noted that in the online environment, there is an emphasis on active participation, and a lack of presence may be met with anxiety . Still relevant two decades later, this observation was affirmed in comments from one of the focus group participants. This student echoed Feenberg’s observations, noting that while student presence was important, it seemed to ebb and flow throughout the course and at times “there was a panic of people posting things”

(S1). Nonetheless, focus group participants agreed that one of the key requirements for successful online learning was the need to have a strong connection with fellow classmates. Unfortunately there was the potential for a negative impact when new students arrived in the online classroom. As the following participant stated, “reading posts from people I didn’t know was difficult. I didn’t have any context. Once I met them, it was different. The context was better because I knew them” (S2).

Linked to personal presence, the benefits of learning in a social context have also been widely noted. However, establishing an inclusive social context can be challenging in an online class with over 40 students. The ability to read, reflect, and respond to multiple discussion threads can be overwhelming and, as Bender found, feelings of being overwhelmed contribute negatively to students’ experiences of online learning . One student found a way to reduce the impact of high post volume by prioritizing the posts, reading only those from classmates s / he already had an existing relationship with, “in looking at the posts from people I didn’t know, I just disregarded them and read the ones from the people that I knew” (S2). Unfortunately, this meant that (s)he was only engaging with a selection of classmates, so the social context was limited.




Creating an environment where students can engage socially can be challenging for an instructor new to the online teaching environment as it requires a different skill set. One focus group member identified the need to have “instructors who are comfortable with the technology” (S5).

Other participants expressed surprise at the lack of interaction from some instructors, stating that “I expected it to be more ‘back & forth’ with the faculty” (S1) and “if I’m paying for an education, I’m looking to your experts to learn. If it wasn’t for my cohort, I probably would have quit” (S2). This frustration not only speaks to the strength of the students’ relationships, but also emphasizes the value of an instructor who maintains an active and engaging presence on the course site. Clearly then, instructors need to be comfortable and at ease when using online learning tools, as well as contribute to insightful discussions, if the students’ experience is to be both positive and meaningful. The result of working with a skilled instructor can be seen in the following participant’s remark “she was phenomenal; she answered your question, but also posted another, and she would post a link or to YouTube to check out” (S5). It is apparent then that this instructor was adept in making use of the technology in a purposeful way.

In an asynchronous environment, where personal presence and social context are not always easy to promote, the use of video or audio recordings can be of some consolation and may play a significant role in students’ success. As this participant commented, video clips that accompany textbased course notes can help to reassure the student, “seeing the face of the

instructor on the screen gave me comfort” (S4). Other students confirmed the value of video clips and the Moodle platform when they stated that, “the

only way I got through Finance was those videos!” (S3) and “the advantage to

having Moodle over straight lecture was that you always had Moodle to go back

to” (S4). Here the students were referring to their ability, in an asynchronous environment, to review material multiple times and reflect on it without the pressures found in the face-to-face classroom, where an instant reaction and response are often necessary. However, as the following student found, reliance on Moodle was not without problems:

“It’s funny how quickly Moodle becomes part of your everyday world, so there was this funny shift when Moodle went ‘down’. There was this little voice in my head that went “oh my, I’m going to be so screwed because there are all these assignments due.” (S2)

In this case, a platform upgrade was scheduled that caused disruption in the availability of the Moodle-based course, leaving students without access to their online classroom.

When asked to recommend online learning tools that could increase their chances of success, the focus group members concluded that the use of visual tools such as webinars, video conferencing, and videos clips would serve their needs well. As the following participant stated, an ideal course would “have a webinar, and have it recorded so that you can play it back if you can’t be there for the presentation” (S4). It is therefore clear that seeing the instructor, in digital format, or hearing his / her voice, was perceived as a key to success in the online learning environment.

Regarding the use of technology in the online environment, the participants’ identified the need to address four critical areas. First, they expressed the necessity of having technology that supports both student and instructor presence. Secondly, tools that allow them to review and reflect on the course material are necessary. Third, the social context of learning was highlighted, and finally, it was important to have instructors who were able to use technology well. Students did not express the need for a multitude of complex tools, but quite simply expressed the need for technology that supported these identified needs.


As a consequence, our recommendations addressing the identified needs are shown in Table 1, below.

Table 1: Recommendations

Identiied need Recommendation


Review / relect

Social context

Skill set

Video conferencing, webinars, audio clips

Video clips, PowerPoint slides

Discussion areas, asynchronous and synchronous tools

Training for faculty on the use of online technology and pedagogy

From our analysis of the data, we believe that if these relatively straightforward technologies were available in all online courses, and faculty were to receive training to support their use, students could achieve greater success, which would lead to a decrease in attrition of online students.

Based on this study, perfecting existing tools, making them easy for both the instructor and students to use, as well as ensuring they are readily available, is a key to successfully retaining students in the online classroom. Also, when developing new tools, the emphasis should be on technologies that support students’ ability to create a presence, review and reflect on course material, and engage socially, as this would have a positive impact on students’ experiences in the online classroom.

While this study is still in its early stages, it is evident that careful use of educational technology is imperative if Higher Education Institutions are to provide appropriate online learning environments. Incorrect choice or application of technology can negatively impact the student’s ability to learn.

Identifying obstacles to learning, as well as gaining a clearer understanding of strategies that can be employed to mitigate challenges faced by students studying online, will lead to an enhanced learning environment, which in turn could positively influence student retention.


The participants in this research project clearly identified the need for synchronous tools that could enhance the asynchronous environment. These tools would not only provide the opportunity for students and faculty to be present in the courses, but they would also allow members of the online classroom to engage socially. In order to achieve this goal, participants recommended the use of visual tools, such as webinars and video lectures.

Students also recommended tools that would be available for viewing on a repeat basis, as well as those that would facilitate reflection. With this in mind, education providers must seek out versatile tools that can mimic some of the positive attributes of the face-to-face classroom, using modern technology that allows repeated access.

In addition to providing tools, it is critical to have instructors who can comfortably navigate and use technology; merely translating lecture notes into a digital format is not sufficient. Instructors who are well-versed and comfortable with visual and audio aides, who can bring other resources to the virtual classroom like YouTube, and who can express themselves with creativity and energy in discussion threads will most likely be received with enthusiasm and excitement from students. By engaging students online in this way, it is expected that they will be motivated to continue with their studies and not just become another attrition statistic.

In conclusion, we were reminded of Feenberg’s comment that “we must remember that CMC is a technology in process. Designers and users should involve themselves in the invention of the systems they require, rather than passively accepting what they are offered as a final product”. We



2 must therefore continually strive to adapt and change technology to fit our requirements, finding new ways to accommodate our need to connect with others as we learn.


Axe, J. (2009) “Developing a Learning Community: Exploring the Impact of Online Activities

on the Building of Campus-Based Communities.” In School of Lifelong Education and

Development. Bradford: University of Bradford

Bender, T. (2003) Discussion Based Online Teaching to Enhance Student Learning: Theory,

Practice and Assessment. Sterling, VA: Stylus Publishing

Berg, B. (2004) Qualitative Research Methods for the Social Sciences. Boston: Pearson

Educational Publishers

Bransford, J.D., Brown, A.L. and Cocking, R.R (1999) How People Learn: Brain, Mind,

Experience and School. US: National Research Council

Dawson, S., Burnett, B. and O’Donohue, M. (2006) “Learning Communities: An Untapped

Sustainable Competitive Advantage for Higher Education.” International Journal of Educational

Management 20, no. 2 (2006): 127–39.

Feenberg, A. (1990) “The Written World: On the Theory and Practice of Computer

Conferencing.” In Mindweave: Communication, Computers and Distance Education, edited by Robin Mason and Anthony Kaye, 22–39. New York: Pergamon Press

McConnell, D. (2006) E-Learning Groups and Communities. First ed. Maidenhead, England:

Open University Press

McConnell, D. (2000) Implementing Computer Supported Cooperative Learning. Second ed.

London: Kogan Page Limited

McInnerney, J.M. and Roberts, T.S. (2004) “Online Learning: Social Interaction and the

Creation of a Sense of Community.” Educational Technology & Society 7, no. 3 (2004): 73–81

Palloff, R.M. and Pratt, K. (1999) Building Leaning Communities in Cyberspace: Effective

Strategies for the Online Classroom. San Francisco: Jossey-Bass

Palloff, R.M. and Pratt, K. (2007) “Building Virtual Communities: Techniques That Work!”

Paper presented at the Annual Conference on Distance Teaching & Learning, Madison,

Wisconsin, August 2007

Palloff, R.M. and Pratt, K. (2005) Collaborating Online: Learning Together in Community. San

Francisco: John Wiley & Sons

Palys, T. (1997) Research Decisions: Quantitative and Qualitative Perspectives. Second ed.

Scarborough, Ontario: Thomson Canada Limited

Reason, P. and McArdle, K.L. (2004) “Brief Notes on the Theory and Practice of Action

Research.” In Understanding Research Methods for Social Policy and Practice, edited by Saul

Becker and Alan Bryman. Bristol: The Polity Press

Robson, C. (2002) Real World Research. Second ed. Oxford: Blackwell Publishing

Schutt, R.K. (2008) Investigating the Social World: The Process and Practice of Research. 6th ed.

Thousand Oaks, California: Pine Forge Press

Stringer, E.T. (1999) Action Research. Second ed. Thousand Oaks, CA: SAGE Publications


Yorke, M. (2000) “Smoothing the Transition into Higher Education: What Can Be Learned from

Student Non-Completion?” Journal of Institutional Research



Shane Sutherland

Pebble Learning, eInnovation Centre,

University of Wolverhampton, Telford

Campus TF2 9FT [email protected]

Alan Paull

58 Norton Wood, Forest Green,

Nailsworth, Stroud GL6 0HG [email protected]


The PortisHEad project developed tools to support applications to UK higher education through learnerowned e-portfolios; including the ability to target unique e-portfolios to different institutions. The original demonstration tool helped address the recommendations of the Schwartz report for fairer admissions to higher education.

However, despite good learner feedback and a strong sectoral imperative, the tool was not implemented by UCAS, the application service. Despite the withdrawal of UCAS from the project the remaining partners developed a generic application toolkit which allows any e-portfolio user to auto-complete educational or employment-related

‘application-type’ forms using learnerowned data from their e-portfolio.

The toolkit is consistent with the ‘thin e-portfolio model’ propounded by the

JISC-funded e-Portfolio Reference Model project. It uses an ‘open standard’ webservice which is easily implementable by ‘form-owners’; access to data is managed by the learners and remains secure. The toolkit is easy to deploy and has already generated signiicant interest not only from admissions tutors but also for its utility to teachers and staff developers. This paper points to how learner-controlled technologies, and learner-owned data, can be meaningfully utilized to engage with intra- and extra-institutional systems using open standards and web services. It also illustrates that technological dificulties are less critical than organisational ones.

0081 PortisHEad: portfolios in successful Higher Education admissions

Project overview

Applicants to higher education (HE) in the UK apply through an application service called the University and Colleges Admission System (UCAS).

Typically applicants will apply to multiple institutions but can only submit a single application ‘form’ within which they are allowed to present a generic

500-word ‘personal statement’. The process does not allow applicants the opportunity to provide extended or differentiated ‘profiles’ or to augment their applications using external resources.

This paper reports on the PortisHEad project (JISC, 2008) which built on the work of the UK’s Joint Information Systems Committee (JISC) funded e-Portfolio for Lifelong Learning Reference Model project (eP4LL, 2006), itself part of the wider eFramework initiative supported by DEST et al

(eFramework, 2008). The project sought to address the Schwartz (2004) report recommendations:

‘to produce a more integrated service for applicants and speciically to facilitate …

Transfer of information from applicants … Structuring the personal statement and reference, in particular through the insertion of course-speciic prompts … Providing feedback to applicants’

In the context of the ePortfolio For Lifelong Learning (eP4LL) work the next logical step was to implement a practical application of the model using an open-source tool (developed by the project), and an existing e-Portfolio system; with a view to learning lessons from a state-of-the-art implementation. The PortisHEad project carried out this work using the

PebblePad system to implement a ‘real life’ pilot version of the web services developed for the Reference Model project, including the use of structured

Entry Profiles and structured Personal Statements for admissions to UK HE.

PortisHEad aims and objectives

PortisHEad’s aims included implementing the integration of the PebblePad e-portfolio system throughout the admissions process reviewing and amending existing admissions processes, so that the learner’s e-portfolio can be placed at the centre. It was vital to the project that the learner remained in control of the construction and submission of the application, and that

Higher Education Institution (HEI) staff could choose how much of the extra information from the e-portfolio to use in selection processes.

In particular the project sought to:

■ revise current Information Advice and Guidance (IAG) and application management practices in feeder establishments and the University of

Wolverhampton for the selected student groups;

■ enable electronic data transfer from the students’ e-Portfolio to the

University via UCAS, linking into current UCAS centralised procedures;

■ provide feedback to applicants, to include assistance to those who are not successful;

■ improve induction mechanisms via further data transfer to populate the university e-portfolios of successful students.




Project methodology

Using a case study approach the PortisHEad project sought to implement an extension of the e-Portfolio Reference Model to enable groups of students from feeder establishments and from the University of Wolverhampton to use their e-portfolios to research, prepare and submit applications via

UCAS, to receive IAG about their applications from their school or college, to receive feedback from HE admissions staff and to form the starting point for enrolment and induction into an HEI. Using the ‘thin e-portfolio model’, based on Web Services and a service oriented approach (SOA), the project aimed to put the learner, via the e-portfolio, at the centre of the HE admissions process.

The project supplied support to the students involved at the application stage, in order to ease their involvement with the technology and the new parts of the application process.

The anticipated impact of the project was to enable a close coupling between the IAG and admissions processes on the one hand, and on the other, the applicants’ experience of learning through the IAG events and personal reflections that occur during the preparation, submission and assessment of their applications and onward through enrolment and induction into university life. The project sought to demonstrate the efficacy of fully electronic admissions, including faster processing and better integration of admissions processes than current systems.

Iterative feedback and project evaluation was gathered through interviews with the students and careers / application advisors. Additional data was gained through close alignment and joint project meetings with colleagues on the ADoM and Delia projects (University of Nottingham) working in the same domain. External evaluation was conducted by the

University of Nottingham.

Outcomes and outputs

The Application Interface

In pursuance of the project aims the University of Wolverhampton;

Pebble Learning and APS developed a demonstrator tool allowing e-portfolio users to:

■ access the UCAS application services and register themselves;

■ conduct course searches from within their e-portfolio environment;

■ utilize course information to write Personal Statements against learner Entry Profiles;

■ submit their Personal Statement using a web-service to their application on the UCAS system;

■ finally, and perhaps most significantly, the tool allows users to publish specific

‘presentational’ or ‘application’ e-portfolios to any of their 5 named institutions.

Figure 1: Application interface and personal e-portfolio data store used to

auto-ill forms


As the current Personal Statement on the UCAS application is common to each of the institutions applied to, the ability to create and publish unique e-portfolios to each institution is seen as an important contributor to enhancing the relevant information about the learner that is available to selectors, thereby supporting holistic assessment of the individual learner

(Schwartz, 2004).

Implementation issues

Unfortunately delays in the appointment of a new liaison member of staff at

UCAS led to very limited involvement of UCAS in all project developments.

This severely affected the project’s ability to continue with its case study approach, which was abandoned for the final project stages. The interim report to JISC (Paull, 2007) stated:

The project has been forced to re-focus its work on demonstration activity, rather than a case study approach, which means that outcomes with respect to the learner and tutor-centred aspects of the project, rather than the technical, cannot be investigated in a realistic fashion.

The inability of UCAS to implement the link between the e-portfolio system and the UCAS system ‘Apply’ brought about an end to the project’s work with students who had created application e-portfolios. Notwithstanding the foreshortened experience feedback from the mentor was encouraging:

The webfolio’s that have been created are shaping up to be fantastic and the students themselves are very excited about the prospect of using them…

Although not quite inished, you can clearly see how valuable they have found this process and all have added the URL link into their personal statement… Two of the courses applied for, social care and law, are always popular [the students] are really pleased they have another way of showing themselves to the admissions tutor, which may help in the selection process. (Student Mentor)

The URL link referred to above represent what was perhaps the final ‘nail in the coffin’ for student involvement when UCAS decided to remove URLs from the Personal Statements of all applications. It is interesting that UCAS took this action as it unnecessarily prevents any student from including web links in their Personal Statement; precluding art and design students from referencing online folios, IT students from referencing websites or programs, and so on.

PortisHEad Interoperable Form Fill

With 5 months of the project left to run, APS and Pebble Learning agreed to develop an alternative means of supporting the generic aims of the project, choosing a method which firmly situates the control of learner-owned data with the e-portfolio user. PortisHEad Interoperable Form Fill (PIFF) allows students to import their personal data safely from a HR-XML web-service enabled e-portfolio system or data source allowing them to fill in web forms on the internet automatically (PIFF, 2008).

Webmasters of sites containing educational or employment related forms can very easily implement this tool kit on their web-based forms. It is simple to use and secure. None of the student’s personal information is stored anywhere and the information can only be obtained by the student after they log on to their e-portfolio system using their user name and password.

The system has been designed to be compatible with all major browsers and has been tested with Internet Explorer 7, Firefox 3, Opera 9 and Safari 3.1.1.

The system is not dependent on any libraries and makes use of Javascript and Ajax to perform its activities. The system is relatively light on resources.




In simple terms the web master includes a button on the web page which allows the student to log on to his or her e-portfolio account; single version data (for example a single surname) automatically populates the mapped field on the form. Where data has multiple values (for example address or email) the toolkit allows the user to choose the ones they wish to use for this current application. As part of the demonstration, toolkit examples have been created which allow users to use their e-portfolio data to populate forms for:

A popular online job-search company;

An undergraduate programme for overseas students;

A post-graduate programme;

An application for a health-service card;

The core fields of the UCAS registration form.


The original aims of the project were predicated on the willingness and capability of a major educational stakeholder to extend their core systems to allow individual e-portfolio users to engage with them to manage complex application processes. Limited resources and development lead-in times extending over more than 2 years prevented its meaningful engagement. As a result the project failed to deliver the reusable web services it had hoped to develop which would have allowed remote systems to:

Conduct course searches;

■ submit application data;

■ aggregate ‘student entry profiles’; and

■ present targeted e-portfolios as part of the wider application.

For the e-portfolio developer it was apparent that the complex application forms could not be easily replicated within the e-portfolio environment.

For example, certain fields are pre-populated according to previous actions and the options for one field may be dependent upon the content of two or more other fields. The complexity of replicating the forms would also lead to significant maintenance overheads year-on-year.

So, from both the perspectives of application form provider and e-portfolio developer, it became apparent that a lighter weight, more generalisable solution would be more desirable. The method eventually developed is consistent with the view of the e-portfolio system as a ‘personal learning system’ where learner-owned data is able to be repurposed and reused multiple times for myriad purposes: and consistent with the ‘thin e-portfolio model’. Using the open standard HR-XML means that the method could feasibly be deployed by HR systems and Management Information Systems as well as e-portfolios. From the perspective of the form owner, the toolkit is easily deployed and does not require any rewriting of the target form.

This new readily available functionality has the potential for significant impact on the transmission of data between systems, because it is relatively simple for developers to implement, easy to use for learners, and does not require complex security measures, because the learner controls access to the personal information. Within an institution where e-portfolios are widely used, it ought to be possible for all internal ‘application-type’ forms to include the ability for users to auto-fill common data fields directly from their personal e-portfolio information. From the perspective of PortisHEad as an admissions demonstrator project, we believe that the toolkit will be a suitable vehicle for universities, colleges and others to deploy in support of their student relationship management systems in the future, particularly for enquiry management, application, admissions and enrolment functions.

The project ended in October 2008 by which time PIFF was to have been


deployed for piloting in institutional contexts. Demonstrations of the toolkit to-date have been well-received and have generated significant interest from other projects thinking of utilizing the tool. Of particular interest is the application of the tool within the University of Cumbria who are using e-portfolios for continuing personal and professional development as part of the Flourish project.


ADoM (2008) Admissions Domain Map elearningcapital/admissions/adom.aspx

Delia (2008) Delivering Enhanced Learner Information for Admissions whatwedo/programmes/elearningcapital/admissions/delia.aspx eFramework (2008) eFramework for Education and Research

eP4LL (2007) ePortfolio for Life-long Learning elearning_framework/elfref_notts.aspx

JISC (2008) PortisHEad capital/admissions/portishead.aspx

Flourish (2008) Flourish: the eCPD project programme_users_and_innovation/flourish.aspx

Paull, A. (2007) Project Interim Report

PIFF (2008)

Schwartz, S (2004) Fair admissions to higher education: recommendations for good practice





0213 In the eye of the storm: preliminary evidence on the use of online learning diaries


The student experience in higher education has been analysed at length, but a full understanding of it eludes researchers and Higher Education

Institutions alike (Tinto 2006). Students’ experiences can be stormy and unsettling — with students’ previous beliefs and assumptions about learning, the world and themselves colliding with the epistemologies and ontologies inherent in higher education, in general and in discipline contexts. Barnett

(1996) calls this colliding of worlds — ‘displacement’ and Brah (1996) describes the space where different identities meet — a ‘diaspora space’.

Edwards and Usher (2001) talk about education as (dis)location in which various educational practices cause location as well as dislocation.

Online learning as part of the higher education landscape is often presented as a safe space where gender, power and class are not as apparent as in faceto-face settings. Online learning is heralded as ‘taking the distance out of distance education’, ‘borderless education’ and the great ‘equaliser’ where anyone can study anytime. This apparently removes constraints usually associated with face-to-face education such as the need to be in a particular location and to attend lectures at specified times (Edwards & Usher, 2001).

Students may also wish for a freer environment where they can continue with their personal and professional lives, and study when they choose; therefore for them online learning can be a dream come true. However, online learning regularly surprises many of these students with its own impacts on personal and professional lives. While it may be borderless, other new barriers are introduced, such as time-zones, computer literacy, and the need to balance personal and professional commitments, and so on.

Whether face-to-face or in an online learning environment, the student experience can regularly evoke images of a brewing storm as pressures mount, personal and professional life worlds clash with study commitments and students trying to make sense of their experiences. In this paper we explore the impact of online learning diaries as the eye of the storm — a quiet and safe space where learners can reflect on the chaos around them.

Online learning diaries provide glimpses as to how students experience the storm but also how they experience the centre of the vortex. While it is often assumed that learning diaries do, in fact, create a safe and quiet space for students to reflect, evidence seems to indicate that some students experience the act of reflection as discomforting and / or even useless.

In the literature review we will specifically explore the impact of learning diaries in the context of online learning as displacement or (dis)location.

The methodological considerations provide insight into the research design and research choices made. We then continue to analyse and discuss the findings, and conclude by proposing several considerations which may further increase the function of learning diaries as an essential element of online learning.

Literature review

The function of learning diaries in the design of learning experiences has been well documented and researched (e.g. Biggs 1999, Bisman 2007,


Dr Paul Prinsloo

TVW 4–69, Directorate for Curriculum and Learning Development (ICLD),

PO Box 392, University of South Africa,

Republic of South Africa, 0003 [email protected]

Dr Sharon Slade

Open University Business School, Foxcombe Hall, Boars Hill, Oxford OX1 5HR [email protected]

Ms Fenella Galpin

Open University Business School, Foxcombe Hall, Boars Hill, Oxford OX1 5HR [email protected]


The surprising lack of pressure and speed in the centre of the vortex of a storm are in stark contrast to the force and destruction often experienced at its periphery.

Many spectators watching a developing storm will be caught between fear and a desire to escape. The metaphor of a storm has been applied here to the emotions experienced by many students enrolling in online learning courses. Not only do the requirements of studying online collide with personal and professional commitments, the experience of learning online

(often in groups) results in many students feeling displaced, scared or out of control.

Learning diaries, especially in an online environment, present students with an opportunity to reach the centre of the vortex, though this may not be as quiet and safe as we may have presumed.

This paper reports on students’ relections in their learning diaries as a prescriptive part of the Professional Certiicate in Management offered by the Open

University. The research focused on the unstructured learning diary entries of 12 students from one tutor group over an 18 day period of a short compulsory online course. This phenomenographic study used grounded theory as methodology to analyse and describe students’ use of their learning diaries. The research found ample evidence that online learning diaries provide students with a safe space to relect on the vortex around them. Without a quiet and relective centre, students may be overwhelmed by the wider forces impacting on them. Students’ postings provided rich descriptions of the vortex of studying online and the function of having a centre to which to withdraw. There is, however, also evidence that posting relections in learning diaries can itself be a dislocating and uncomfortable experience for some learners, while others question its usefulness.

The work provides practical and useful information for managers of online learning experiences, instructional designers and curriculum developers.

Boud 2001, Cunliffe 2002, Moon 1999, and Salmon 2002). For the purpose of this research we were curious to evaluate how students experienced and evaluated learning diaries. Previous research by the authors explored online learning as a (dis)locating experience. In this study, we focus on how students evaluated the use of online learning diaries in the context of (dis) location. The question we explored was: Do learning diaries provide students with a safe space, a quiet centre in the midst of the storm?

In this literature review we will revisit (dis)location as metaphor (originally proposed by Edwards & Usher, 2001) and enrich the metaphor by exploring online learning as displacement (as proposed by Barnett, 1996) and diaspora

(as proposed by Brah,1996).

Edwards and Usher (2001) highlight the fact that spatial metaphors are increasingly used to explore the impacts of globalisation which bring to the fore issues of border, location and boundaries. Wiseman (1998 in Edwards and Usher, 2001) describes a world “in which relationships are becoming less two dimensional and hierarchical and more like networks, rhizomes and Internet links”. Edwards and Usher (2001) agrees with Brah (1996) who speaks of current times as a ‘diaspora space’. The issue in this diaspora space is not only relocation but also (dis)location (Edwards & Usher, 2001).

Computer-mediated communication has “created a situation where both clock time and physical space can be transcended” (Edwards & Usher, 2001) and where all inhabitants in this global village “are likely to be strangers”

(Turner 1994 in Edwards and Usher, 2001). Cyber space is then proposed by a number of authors as diaspora space where we all are nomads who are not necessarily ‘homeless’ but “capable of recreating our home everywhere”

(Braidotti 1994 in Edwards & Usher, 2001). Cyberspace, having no ‘centre’ and ‘limited hierarchy’, demands that we speak of ICT no longer “simply in an instrumental sense as an efficient tool of communication, but more aptly as a socially and culturally produced space that stimulates new forms of interaction, helps restructuring and forging creolised identities and produces new relations of power, for example, between teachers and learners”

(Edwards & Usher, 2001).

Research undertaken by Steel and Hudson (2001), found that the “most prominent drawback, unsurprisingly, was the fragility of technology and its negative impact on the learning and teaching process” (2001). The fragility of technology not only refers to the many possible technological hiccups that teachers, designers and students face, but also to the “robustness of the technology” (Steel & Hudson, 2001) with continuous changes and innovation, resulting in students and teachers constantly feeling behind with the latest developments. Interestingly, the fear of technological failure was found (Steel & Hudson, 2001) to be the most feared scenario. “Even if the technology has never failed for example, the fear is that it could” (Steel

& Hudson, 2001). In her research on online learning, Fleckenstein (2005) found unreliable technologies to have “played a central role in disrupting community building” and “online participation was subject to seemingly random forces that disrupted and prevented the growth of fellowship”

(Fleckenstein, 2005).

While online learning is often celebrated as a truly democratic space,

Edwards and Usher (2001) caution that cyberspace “produces new formations of social and economic power and it is against these that its democratic actuality must be judged”. They refer to Tabbi (1997 in Edwards

& Usher, 2001) who “argues that” it is precisely the disembodiment, disembeddedness and decontextualisation (no bodies, no history, no place), or dislocation, of electronic discussion that will always limit the democratic, and hence educational, potential of cyberspace.” Online environments can dislodge students’ and teachers’ “monochromatic worldviews that are often racist, sexist, and homophobic” (Luke 1996 in Edwards & Usher, 2001).




“Virtualisation does not imply disembodiment, but relies on disembodiment”

(Fleckenstein 2005, emphasis added). Students therefore frequently devise strategies to cope with this disembodiment and with feelings of disequilibrium. Fleckenstein reports how students often insert information about their physical environments into their virtual ones (Fleckenstein 2005).

The disembodiment inherent in online learning can also result in students sending pictures of themselves or exchanging contact numbers, or arranging to meet face-to-face (Fleckenstein 2005).

This disembodiment and disembeddedness also impact on the configurations of teachers and students alike. Often, neither teachers nor students are prepared for these reconfigurations and find them (at least in the beginning)

(dis)locating and causing friction (Crawford, 1999). In online education, the role of teachers changes to becoming guides and facilitators of learning

(Steel & Hudson, 2001), often resulting in educators experiencing a sense of (dis)location from traditional perceptions about their authority and subject expertise. The change from being “gods of knowledge to directors of or leaders in the pursuit of knowledge” often result in professional disorientation (Crawford, 1999).

The different (dis)locations Edwards & Usher (2001) discuss in relation to online learning, can be summarised as follows:

■ Dislocations of identity — the impact of anonymity, new identities, changed autobiographies (disembedded, disembodied and decontextualised). The different markers of identity (dress, facial expressions, body types, accessories and labels) which play a major role in face-to-face communication are absent in online environments. Often

■ students and teachers assume different personalities and identities online.

Dislocations of

relations — the relationship between teacher and students is changed; relationships between students are affected, and the

■ relationship between students and texts is altered.

Dislocation of

authority — the traditional authority of teachers and texts are questioned; notions of authority range from number of postings,

■ proficiency of expression; number of links / followers, etc.

Dislocation of

trust — hierarchical and horizontal observation and

■ resulting in mistrust and loss of certainty.

Dislocation of

roles and competencies — competent and professional educators and students may find themselves dislocated in online contexts where their previous roles and competencies become superfluous, and new roles and competencies are required.

Online learning as (dis)locating practice intensifies the need for reflexivity in which participants constantly make meaning of the fluid and uncertain environment (Edwards & Usher, 2001). The authors refer to the point made by Giddens (1991 in Edwards & Usher, 2001) that the proliferation of information and personal decision-making are “existentially troubling” and that participants are forced to increased levels of reflexivity as ambiguity, insecurity and existential anxiety becomes “unstoppable”. Reflexivity creates opportunities for participants to map their own location (however temporarily) and those of others (Edwards & Usher, 2001).

Prinsloo, Slade and Galpin (2008), using the notion of “location” as heuristic framework analysed student learning diaries and found evidence of different types of dislocation and also, evidence of how students attempted to locate themselves. The following dislocations were identified by Prinsloo et al, 2008)

■ The (dis)location of being and learning online.

The (dis)location of becoming part of the management discourse.

The (dis)location of learning in a team.

The (dis)location of working against time constraints.

The (dis)location of not being online.


Their research also found evidence in several student postings of how students located themselves to cope within the general context of (dis) location.

The context, programme and participants

The focus of this project is the learning diary as a structured and compulsory part of an online course held over eighteen days. It comprises activities organised in four timed and sequential stages of between three and six days.

Students are allocated to a group of about twelve participants supported by a tutor, and are asked to be online every day.

Throughout the course, students are required to keep a private


online learning diary in which they are asked to review their learning and experiences of working online. The learning diaries are therefore an integral part of the structured learning experience with students required to have at least four entries in their diaries (one for each stage). It is important to note that the content of the learning diaries is not assessed


. Students are given no further advice regarding the content of their diaries, nor how reflexivity can potentially enrich their learning and empower them in becoming managers.

The content and structure of students’ entries into their learning diaries is left entirely to them


. They are, however, also required to formally review each stage in the public domain of their tutor group. In these stage reviews, they are requested to reflect and evaluate both their own learning and their experiences in groups.


Despite the minimum requirement for posting, nine students from our study group posted entries into their learning diaries on ten of the 18 days of the course. Six students posted reflections on more than 15 days while the minimum number of actual postings was six (Prinsloo, Slade & Galpin, 2008).


The research scope and methodologies were clarified with staff from the University Research Ethics Committee. Care was taken to ensure the anonymity of the students and tutor involved.

The approach adopted was phenomenographic as explored by Richards

(1999) and Cope (2004). Phenomenographic research is interested in the qualitative differences among the perceptions and experiences of individual students. In an attempt to ensure validity and reliability in phenomenographic research, Cope (2004) proposes a number of verification strategies which we incorporated in the research design. The strategies adopted include acknowledging the researchers’ background and prior experiences; reporting the means by which an unbiased sample was chosen; describing the data collection and data analysis processes; declaring the processes involved to control and check interpretations; presenting results in a manner which permits informed scrutiny; and the clear description of categories adequately illustrated with quotes.

At the end of the course, learners were required to complete an evaluation form in which they appraised different elements of the course, including the learning diary. The 18 comments regarding the learning diary were

1 Although they are told that tutors have read access

2 This addresses to some extent the caution by Boud (2001) of the impact of assessment on the authenticity of the content of learning diaries

3 In the schema proposed by Varner and Peck (2003) the unstructured nature of these online learning diaries contribute to greater relection and more authenticity.

4 There are several differences between the format and content of students’ relections in their learning diaries and their reviews of stages in the public domain. This is the focus of further research not reported here.



2 extracted anonymously and clustered into broad categories by two of the three researchers. These categories were compared and agreement reached

(and differences noted) following a verification by the third researcher. Some statements were allocated to more than one category.

The 18 comments resulted in three broad themes:

Theme 1: Learning diaries as locating experience

Theme 2: Learning diaries as dislocating act

Theme 3: Pointers for improvement of the function of the learning diaries

Analysis and discussion of indings

The three identified themes are explored in detail, analysing students’ postings and reflecting on the implications for the use of online learning diaries.

Theme 1: Learning diaries as locating experience

Nine of the 18 statements were clustered under this theme. Statements were clustered here if they showed appreciation for the opportunity to complete entries in a learning journal, commented about the positive impact of having a place to show their emotions and / or tracked the progression of their learning and development as managers.

The following statements


are examples of comments under this theme:

“I wasn’t just satisfied with the learning diary — I thought it fantastic! All tutor groups/ courses with a conferencing facility should have this. All conferences & learning diary’s should not carry marks, but like here — you can’t get an overall pass unless you contribute to both!!!!!!!”

“Liked the diary — opportunity to let off steam and reflect from the beginning”

“Loved the learning diary it was an excellent tool to capture students feelings.”

I think I got better with the diary as the challenge progressed, I lost some of my self consciousness and understood better the effectiveness that could be gained from it.

“the Learning Diaries technique would be a great tool for me to take into the workplace for staff inductions and project management. I found the diary a great tool to refer back to for assessing progress and analysing problems.”

These statements confirm the findings of Prinsloo, Slade and Galpin (2008) that students experience learning diaries as a safe space to vent their frustrations, hopes, and fears.

Theme 2: Learning diaries as dislocating act

A less researched issue is the sense of dislocation experienced by some students when required or asked to post entries to a learning diary. Six of the

18 statements were linked to this theme, and included

“It was difficult to keep up entering in the learning diary as the <course> was fairly time demanding as since I was also travelling on business it was also very difficult for me to keep p with time zone changes.”

“Although the learning diary is a good tool, I am desperate for noting my thoughts, I have a great memory at work so never write anything down, this didn’t change, even with the constant nudges to do so. I am sure I would have got much more out of the diary had I used it more.”

1 The statements are presented anonymously and unedited. Grammatical, syntax and typing errors have not been changed.


“I found the learning diary hard to fill in, but I find things hard to express anyway, one of my weaknesses.”

“the learnind diary felt like a stone”

In considering these statements, we concluded that:

■ the act of completing the learning diary sometimes added to the pressures of finding enough time to do the other online activities amidst balancing personal and professional lives.

■ there were feelings of discomfort with recording personal thoughts, either from students who felt self-conscious, or those experiencing difficulty expressing their thoughts.

■ not all understood the potential role of the learning diary as reflection-onaction (as proposed by Schön 1983, 1987).

■ the requirement to complete a learning diary was experienced as very negative.

These findings offer potentially useful insights for course designers considering the future use of learning diaries. A basic assumption is that learning diaries are intended to enhance and deepen learning rather than frustrate it. With regard to this research, it is important to note the remark that “The aim of the work is not to develop reflective skills in these students, but to improve their learning. The quality of their reflection is incidental”

(Moon, 1999). The amount of time required by students to post reflections should not negatively impact on time available for study activities. Although some students may perceive reflection time as a ‘waste’ and a distraction from the real purpose, reflexivity and a reflective mindset has been proven to deepen learning (O’Donnell, Reeve & Smith 2006; Ryan 2005 online). It is understandable that some learners experience discomfort and are selfconscious. Many have never been exposed to the act of conscious reflection.

As prior experiences regarding reflectivity may differ, it is quite possible that students will not know how to reflect nor what to record.

Theme 3: Pointers for improvement of the function of the learning diaries

In theme 3, students have proposed specific improvements to the use of online learning diaries. Six comments dealt with suggestions on the improvement of learning diaries in the context of the course, and included:

“Learning diary a bit primitive. Think this area could be improved. Maybe some prompt questions already typed in there.”

“Also I think more guidence of what to record in the learning diary would help as sometimes I just made entries without any real point or purpose to them other than I knew I had to make one.”

“It was nice to reflect some thoughts on the learning diary. I just hope we get some feedback that will include the work on the learning diary, because I will find it interesting to see what the tutor thinks about my thoughts.”

The first element in this theme is the need for more structure. One student proposed ‘prompt questions’ and another ‘more guidance’. Varner and

Peck (2003) propose that learning diaries can be described as varying along two primary continua, namely a vertical axis indicating varying degrees of structure, with the other axis signifying whether the focus of the learning outcomes are inwardly or outwardly focused. The more inwardly a learning diary is envisaged to be, the less structure, while the more outwardly a learning diary is designed to function, for example, being assessed, the more structured it should be. Research by Prinsloo, Slade and Galpin (2008) also suggests that more structure may actually impede spontaneity and result in a loss of some authenticity. As each of the different stage reviews is followed by a formal structured public review , the learning diary in its current unstructured format reveals rich and thick descriptions of students’



2 experiences of the vortex. As Prinsloo et al (2008) suggest, more guidance in the orientation to the course and the possible inclusion of some of the postings of previous learning diaries, may actually allow students to find their feet quicker in posting their reflections. The ‘primitive’ nature of the learning diaries is intended.

Another supported element is the need for feedback from the tutor or the course management team. Prinsloo et al (2008) found that many students actually address the online tutor in their diaries in the form of a dialogue.

Although tutors are allowed to contact a student should they become aware of some serious impeding factors that the student may deal or struggle with, s / he will routinely only read the postings without responding. Boud (2001) warns that there may be several factors inhibiting or frustrating reflexivity in a learning experience. He explores two inhibiting factors, namely the impact of who will read these reflections, and whether and how these reflections will be assessed (2001). To engage reflexively with a space where one deliberately decides to be (dis)located for the sake of growing while knowing that someone else will watch your ideas, may cause emotions to be so dislocated that the purpose of reflexivity as dislocating practice loses its impetus. “The expectation of writing for an external audience can profoundly shape what we write and even what we allow ourselves to consider. The range of consequences of being read by others can stretch from mild embarrassment to loss of a job or even worse” (Boud, 2001). Boud further explores the effect of assessment on diary writing and concludes that knowing that the diaries will be read and considered as part of formal assessment, may adversely affect the quality and honesty of writing.

The learning diary in the context of the course can then actually become much more. Students can become aware that learning diaries are one way to nurture a ‘reflective mindset’ which enhances better decisions and strategic thinking (Schön 1983, 1987).

One posting refers to the fact that the student ‘confesses’ that “the only reason that I wasn’t very satisfied with this is due to my own lack of time to fill it in.” Although the student accepts responsibility for not gaining any benefit from the learning diary, their confession points to a need to make the benefits of posting reflections in the learning diaries more explicit.

Two postings pointing to the need for more clarity on the function of the learning diary in the context of the course are:

■ “bit unsure at first about the learning diary — actually completely missed it for the first three days !!”

“Still not sure what the learning diary contributed to the course, but it was interesting to note how my attitudes and approach had changed during the course”.

Both indicate that the purpose of the learning diaries was not always clear to students (except as a requirement for the course). Although the second posting shares uncertainty regarding its purpose, it also suggests an appreciation for the fact that the learning diary did provide an indication of changing “attitudes and approach”. What is interesting here is the implication that the student actually contemplated previous postings in his or her learning diary and became aware of a change in attitudes and approach during the course. In the midst of the uncertainty, this posting shows the benefit of having a trail to retrace and reflect upon.

Should students be encouraged to read their previous postings and write one final reflection, then the richness of their experiences and growth as learners and as managers may become more obvious to them and the managers of the course. Such reflections can then become part of the orientation to the course to show how the learning diary as an essential element of the course contributes to a reflective mindset.



Online learning, like the broader student experience in higher education, can be described as a ‘diaspora space’, and a (dis)location. In online learning space, place, identity, roles and competencies are swept away to create a vortex in which students personal, professional and student agendas clash and morph. This study found ample evidence that online learning diaries provide most students with a safe space to reflect on the vortex around them. Without a quiet and reflective centre, students are occasionally overwhelmed by the forces around them. Students’ postings provide rich descriptions of the vortex of studying online and the advantage of having a centre to withdraw to. There is however also evidence that posting reflections in learning diaries itself can be a dislocating and uncomfortable experience for some learners, while others question its usefulness.

Although this small study can not be considered as representative of all students taking part in online learning, they do, however, provide some important pointers for the designers and managers of online learning environments.

Without proper guidance on the function of online learning diaries, the act of posting reflection may contradict its very purpose. This research confirms previous studies that reflection does enrich and deepen learning, but only when properly introduced and explained. While the act of reflection remains an uncomfortable experience for many learners, this should not distract from its usefulness. As learners develop a reflective mindset and praxis, learning diaries can become a normal (and celebrated) part of the daily lives of managers.


Barnett, R. (1996) Being and becoming: a student trajectory. International Journal of Lifelong

Education 15(2): 72–84

Bisman, J.E. (2007) The learning journal as critical reflective assessment tool in accounting

education. Paper delivered at the Annual Conference of the Accounting and Finance

Association of Australia and New Zealand (AFAANZ) at Conrad Jupiters, Gold Coast,

Queensland, Australia, 1–3 July

Brah, A. (1996) Cartographies of diaspora. London: Routledge

Crawford, R. (1999) Teaching and learning IT in secondary schools: towards a new pedagogy?

Education and Information Technologies 4: 49–63.

Cunliffe, A.L. (2002) Reflexive dialogical practice in management learning. Management

Learning 33:35–60

Edwards, R. and Usher, R. (2001) Globalisation and pedagogy. Space, place and identity.

London: Routledge Falmer.

Fleckenstein, K. (2005) Faceless students, virtual places: emergence and communal

accountability in online classrooms. [online] Computers and Composition 22(2): 149–176

Mintzberg, H. (2004) Third-generation management development. Training & Development

(TD) March: 28–37

Moon, J.A. (1999) Learning journals: a handbook for academics, students and professional

development. London: Kogan Page

O’Donnell, A.M., Reeve, J. and Smith, J.K. (2007) Educational psychology: reflection for

action. Hoboken, N.J.: J. Wiley

Prinsloo, P., Slade, S. and Galpin, F.A.V. (2008) Online Learning: Narratives of (Dis)location

Ryan, T. (2005) When you reflect are you also being reflexive? The Ontario Action Researcher,

8(1). Available at [accessed on February 23rd


Salmon, G. (2002) Mirror, mirror, on my screen… Exploring online reflections. British Journal of

Educational Technology, 33(4): 379–391

Schön, D. (1983) The reflective practitioner: how professionals think in action. London: Temple Smith.

Schön, D. (1987) Educating the reflective practitioner. Address to the 1987 meeting of the

American Educational Research Association. Available at howteach/schon87.htm [accessed on February 23rd 2009]

Steel, J. and Hudson, A. (2001) Educational technology in learning and teaching: the

perceptions and experiences of teaching staff. Innovations in Education and Teaching

International, 38(2): 103–111

Tinto, V. (2006) Research and practice of student retention: What next? Journal of College

Student Retention, 8(1):1–19

Varner, D., and Peck, S.R. (2003) Learning from learning journals: the benefits and challenges

of using learning journal assignments. Journal of Management Education, 27:52–77




0315 To what extent could

Business Process Management

Suite (BPMS) contribute positively to e-learning?


The purpose of this paper is to evaluate the effectiveness of Business

Process Management Suite (BPMS) as a teaching-learning technology with the lens of the conversational framework (Laurillard 2002). Commercially,

BPMS is “a more comprehensive approach to Business Process Management

(BPM), it provides all of the process management capabilities of BPM software, plus the following functionality: knowledge management, document management, collaboration tools, business analytics, and a work portal”. ( From the view of analysts, BPMS is

“…an integrated collection of critical software technologies that enables the control and management of business processes. As compared with other model-oriented development tools, such as integrated service environments (ISEs) and integrated development environments (IDEs), a BPMS emphasizes business user involvement

in the entire process improvement life cycle, from design through implementation, deployment, monitoring and ongoing optimization. Rather than reducing reliance on people through automation, a BPMS emphasizes the value of coordinating people and information, in addition to systems, as central resources.”

(Hill, et al. 2007, italics: author)

It is such wide-ranging aspects of BPMS that lead the author to embark on an initial evaluation BPMS as a teaching-learning tool. The inspiration for this paper came from Professor Laurillard inaugural lecture (2008a) and her paper on learning technologies (Laurillard, 2008b). In her lecture and papers she suggested the possibility that the conversational framework may be used to evaluate teaching-learning technology and identify what such new technology can offer to make the teaching-learning process more effective for the learner and the teaching process more proactive for teachers. One of the challenges that struck the author, was the need to find a technology that could transform education from a bottom-up approach by equipping the teaching community with a tool that is simple to use, like the invention of “chalk and blackboard” or more recently, Microsoft PowerPoint. In other words, can BPMS be an education technology for teachers which can be easily and quickly deployed for online learning? Just as the commercial enterprises look to BPMS for continuous process improvement (CPI), could teachers and lecturers also look to BPMS to continuously improve their teaching and their learners’ learning processes.

In addition, the author hopes to link commercial technological development with research in teaching-learning technologies. At this stage, this theoretical evaluation aims to address the preliminary question — could the educational communities adopt BPMS, a tool that has evolved from the commercial world to further enhance teaching-learning process?

The literature and scope

Since BPMS is a relatively new topic, initial search of current academic


Mr. Thean Keat Khoo

King’s College London, Information

Services and Systems, IT

Systems — Enterprise Applications, 26–29

Drury Lane, London WC2B 5RL [email protected]


The purpose of this paper is to evaluate the effectiveness of Business Process

Management Suite (BPMS) as a teaching-learning technology with the lens of the conversational framework

(Laurillard 2002). The paper hopes to link commercial technological development with research in teachinglearning technologies and bring about better collaboration between the two.

This theoretical evaluation aims to address the preliminary question — could educational communities adopt BPMS, a tool that has evolved from the commercial world to further enhance teaching-learning process? The scope of this paper and its evaluative study will be limited to using the conversational framework. The paper will briely discuss

BPMS and its relation to business process and business process management to provide a brief introduction. The main section of this paper will be a detailed analysis of key BPMS components against the conversational framework.

The conclusion will provide a summary of the effectiveness of BPMS as a teachinglearning tool based on the requirements set out by the conversational framework.

The results of the conclusion could lead to further empirical research on BPMS as a teaching-learning technology tool and may be the opportunity to request funding to carry out a proof of concept.


literature did not return significant and relevant findings to support evolution and analysis of BPMS as potential teaching-learning technology for e-learning and e-teaching. The most relevant academic work was a study by

Helic et al. (2005) at the University of Technology Graz, Austria. The study focused on the technologies behind Business Process Management (BPM) and how such technologies could be used to enhance and deliver e-learning.

Some of the technologies discussed are in the context of Enterprise

Application Integration (EAI), Workflow Management Coalition (WfMC),

Extensible Markup Language (XML), Service-Oriented Architecture (SOA),

Business Process Modelling Notation (BPMN) and Business Process

Execution Language for Web Services (BPEL4WS). These technologies are fundamental technical components in the construction and running of BPMS as it is known today. Without them, BPMS will not have evolved or matured to a point where business users with little or no programming knowledge could model, execute and monitor business processes online.

Instead of evaluating these BPM technologies in isolation, this paper will focus on the application of BPMS and will use the conversational framework as a benchmark for effectiveness. It should be noted that in their study,

Helic et al (2005) did conclude that general learning processes and business processes have strong similarities in both user aspects as well as technical aspects. They concluded that applying BPM technologies to manage e-learning process can improve a wide range of common learning situations in e-learning systems

The scope of this paper and its evaluative study will be limited to using the conversational framework. The paper will briefly discuss BPMS and its relation to business process and business process management to provide a brief introduction to its origin. The main section of this paper will be a detailed analysis of key BPMS components against the conversational framework. The conclusion will provide a summary of the effectiveness of BPMS as a teaching-learning tool base on the requirements set out by the conversational framework. The results of the conclusion could lead to further empirical research on BPMS as a teaching-learning technology tool and may create opportunities to request funding to carry out a proof of concept.

A brief history

To best describe BPMS, the author starts with examples that are related to the commercial world because this is where BPMS originated. An example of a business process is online purchase, which many readers could relate to, but who may not be fully aware of the complexities of the processes which need to take place behind the scenes for an online purchase to be fulfilled. The business process of buying something online could consist of a coordinated chain of activities intended to produce business results. In the case of buying a book online, the fundamental outcomes are the delivery of the book to the right address and that the right price has been charged to the purchaser’s credit card. The component steps of this process are carried out by both systems and people. To make the process work, an exchange of the goods (the book), money and data has to take place across systems and people. For example, when the right book has been selected and paid for by purchaser, the system has to trigger a process to find the book from the right location in the warehouse, pack the book and place the packed item in the next process — the distribution process. The distribution process has to coordinate deliveries with a third party system and notify the purchaser on the e-commerce site that the book ordered has been dispatched. This simple example shows that business processes often run across different systems and people. In this case, the online book store, the information systems required may consist of an e-commerce site, a third party payment system,



2 an inventory system for stock control and a logistics system to manage the distribution. From a human aspect, they are packers, couriers and managers — to mention but a few — to make this process work.

In recent times, the struggles that the commercial world faces are not centred on the efficiency of each of specific business information systems, but on complexities of enabling cross-functional end-to-end processes across multiple systems (Butler 2007). According to a report on BPMS (Silver 2006), the demand for cross-functional thinking instead of the traditional discrete business functional mindset brought about the management discipline of business process management (BPM) that began twenty years ago as a different way to think about business. Another commercial report published

(Butler 2007), concluded that the origin of BPMS came about as the result of a mishmash of re-engineering, enterprise integration and workflow solutions.

From the two reports by Silver and Butler and the lectures presented by

Professor Laurillard, the author sees a number of similarities between the commercial enterprise and ICT in education. Like the commercial enterprise, many teaching-learning technologies such as Blackboard (www.blackboard.

com) and eCollege ( have matured in isolation and as a standalone component. Like many commercial applications they are lacking in ability to make connections to other education ICT systems, and more importantly to other teaching-learning technologies to ultimately provide teachers and students with a seamless teaching-learning environment. A typical student enrolling on an online course might have to interact with different systems — student portal, its Virtual Learning Environment (VLE) such as Blackboard, and other learning technologies in order to complete the required learning process. The same could be said for an average teacher who might be required to use various ICT technologies and student information systems to develop and run an effective course. What could be lacking are the technical capabilities and supporting infrastructures that could integrate all these teaching-learning technologies into a single environment which is process focused instead of system or functional focused that a commercial BPMS could possibly provide.

Figure 1: Conversational framework (Laurillard 2002)





Teacher’s ideas Learner’s ideas





Other learner(s)

Adapt task practice environment


Task goal


Practice environment



Adapt actions


Learner’s practice

Draft outputs

Draft outputs

Adapt actions


Other learner(s)


General Components of BPMS

The following section describes the general functional areas or components of BPMS. In Helic et al (2005) study, these functional areas or components were referred to as phases. The three phases that were evaluated in their study were model and design phase, deployment and execution phase and analysis and improvement phase. They mapped these to a learning process developed from their experience, which also consisted of three phases -modelling, learning, and observation and improvement phase.

Since 2005, commercially, BPMS has incorporated many BPM components and functional areas (Butler 2007). Selected components will be discussed individually in relation to related core structures within the conversational framework (see Figure 1).

Process modelling

This component covers the ability to quickly and rigorously define processes that cut across systems and people. This is frequently delivered by using drag-and-drop modelling environment that often includes process wizard and templates. Since BPMS is a commercial tool, the wizards and templates are based on best practice for business processes. However, if BPMS is adapted to teaching-learning, then similar wizards and templates could be used to design learning and teaching processes. This could be based around the conversation framework or other teaching-learning framework. But unlike a stand-alone modelling tool like Microsoft Visio, the modelling capabilities are seamlessly linked to the design and development environment.

The process modeller will allow teachers of a given subject to create a learning process diagram that shows interactions between teacher, learners and practice environments as outlined by the conversational framework. In the case of practice environments, this could be other learning systems such as simulation games for business students, online self-assessment, online video and discussion for example. The integration component discussion will further show how the conversational framework core structures can be brought together to complete the learning process.

Design and development

This is also referred by some as the authoring environment for “what you see is what you get” (WYSIWYG) development platform, where what the screen or interface that the users see is the same as during the development stage. This component of BPMS is tightly linked to the process modeller, turning the process map into user interfaces and web forms that could include features specified in a learning process. For example, to be able to explain Opportunity Cost (Laurillard 2008a) as a learning outcome, could be followed by further instruction to form groups of three and assign specific roles such as teacher, student and observer. This is then followed by a user interface to rank or vote for the best explanation. To the student, BPMS offers a single interactive environment for learning and for the teachers and learning technologists, it could be the tool that facilitates change without the need for extensive programming or “codeless” application development.

This is made possible with web 2.0 technologies like AJAX (Asynchronous

Java and XML) that allow drag-and-dropped control from the design palette to quickly design forms and user interfaces.

The design component is not only suitable for developing interfaces for students’ interaction and integration with other learning systems, the design function coupled with the rules function (see rules management and execution) could be used to develop a questionnaire for feedback and self assessment. For example, an interface can be designed to track a number of questions answered correctly and based on a set rules defined by the teacher, increase or decrease the level of difficulty in the next set of



2 questions presented in a subsequent user interface. In many commercial enterprises, the ability to build and deploy fully-functional web application with little or no programming has empowered many knowledge workers to focus on creating processes that could better address customers’ needs, leaving much valuable time for IT specialists to focus on the value added task such as integration with other business systems (Butler 2007).

Deployment and execution

This is the engine that orchestrates or runs a particular process or multiple processes. This is where the process model and the designed interfaces are executed and orchestrated. Based on the previous example of learning outcome, this is where each of the specific interfaces could be deployed according to the prescribed learning pattern (process). For example, the student could not progress to the discussion interface if they have not yet completed a set of prescribed tasks or students could not proceed to rank the best explanation until they contributed to an online discussion on the concept of opportunity cost.

In advanced BPMS, such engines come with the ability for process tracking.

The ability to have a built in process tracking capability will allow teachers to interactively monitor the learning process that has been designed. For example, which students that have not participated in posting questions or how many answers have been given and the number of iterations a learner has given and how their ideas have evolved as a result of the other learners’ comments. In the commercial enterprise, this feature is used to collect and process data about the transaction connected to a process. In the case of online purchase of a book, it is used to track the number of books purchased and the number of orders that were processed on-time. It is also used to handle errors — such as when an order has become stuck in a process sequence.

In the conversational framework, this component of the BPMS could be used to link the teaching-learning exchanges of answers, feedback, and actions taken by both teacher and learners on reflection of their contributions, answers and outputs. It could track answers given by the learners and the ideas or comments from other learners. The tracking could trigger an alert such as an email or a mobile text message notifying a student that he or she has not been participating in posting comments or when feedback a has been received from fellow learners.

Integration facilities

This component focuses on the ability to support and deliver interaction with other information systems via various technologies that allow bidirectional connections. This is not only limited to data level integration but to reusable technology such Service-Oriented Architecture (SOA). The ability to integrate with other existing systems allows BPMS to work with pre-built application interfaces — such as an existing student portal for managing student access to various e-learning environments such as Blackboard and online e-journal databases. In some BPMS this component is known as Web

Service Adapters Development because it provides connection to existing functionality and tools for creating new services. Using this function, a proven teaching-learning process could be packaged as a web service and be deployed for consumption by other departments or courses that have the need to use the same model in many different modules. Although the process will be the same, the concept discussed could be adapted according to the need the specified learning outcome. This could encourage reuse of a proven model and sharing of teaching process as a web service.

Integration in BPMS is not only limited to system-to-system (S2S) integration.

It can also support human-to-human (H2H) and human-2-system (H2S) interaction and integration (Butler 2007). Over the years, many integration


vendors came to realise that not all processes can be automated and many commercial processes involve a high level of interaction among individuals, where human intervention cannot be replaced. Returning to our example of learning outcome, the lecturer’s comments to a group discussion is an example of H2S interaction. The practice environment and the learners’ practice may require a form of H2H interaction where a social science student may be required to undertake some field work to carry out an interview or observation. The recording (voice or video) can then be posted as the learner’s idea on to the learning process as a case study for comments by other learners.

The collection of cases recorded can form a social learning environment for the next stage of the course or it can be linked to the learner’s online portfolio so that other teachers may review and assess how the student has progressed through his or her individual learning process. However, a detailed discussion on student learning life cycle management and BPMS is outside of the scope of this paper.

Business rule management (BRE — Business Rule Engine)

This is the “logic” behind the business process. Before the days of programming, “conditions” were coded into applications to automate the flow of decision. For example, if a consumer purchased more than five books, a hard coded business rule could be used to allow free delivery.

Each time the business wants to make a business decision to change the rule, the business person responsible for the rule couldn’t make the change without intervention from a programmer. In theory, BPMS the rule engine is delivered as an independent but integrated component to the entire suite.

This allows rule change without impact to the underlining process and can be carried out through simple web interface, on the fly. But not all analysts share the same view, Silver (2006) warns that change to some complex rule engines may not require programming, but it may still require a programmer to implement. The author’s own experience with some BPMS rule engines can be said to be user friendly — some taking a Microsoft Excel approach and other more graphical in nature.

The rule engine might not have direct application to the conversational framework, but indirectly this function offers teachers the ability to design

“logical” online programmes with capabilities that could be applied to developing self-assessment and delivering some form of personalised learning — for example, to start a new process if a student continuously answers a set of questions wrongly. The rule engine could be used to release a pending task, to post a group’s questions / answers for all other groups to view when all groups have submitted their posting online. The rule engine can be used to update process data and send notification.

The following example might not provide a direct link to teaching-learning but what the author hopes to show is how other educational business systems or processes could be integrated into online teaching-learning process using BPMS: A student who has just completed a particular online module and its assessment outcome updated automatically to the student record system and a notification alert to register for the next module after the university exam board has approved the posted grade.

Simulation and testing

There are tools for process simulation and optimization. When a process has been mapped out using design function, a business user could run a “what-if ” simulation using estimated data to see how the process will behave in various conditions. Again, using the online book order example, a process designer might simulate demand for online orders to understand the resource require to pack and deliver the books on-time. In BPMS, the simulation is tightly integrated to the modelling function and in most cases it is the same environment used for modelling. The optimization component



2 further allows data to be collected at run-time and stored for historical analysis. Hence simulation can be run against historical data to help improve a process that has been running over-time. Teachers might find this function useful to test drive their course to see if the resources allocated in relation to time for a selected type of teaching method are sufficient. However, without further empirical tests, it is not possible to determine the application of simulation to the teaching-learning process.

Analysis and improvement

The analytic function has been briefly mentioned in the process execution section as process tracking. But in BPMS, process analysis is an important component that deserves further discussion. One way to see the analytical component of BPMS is in its operational value. It may be likened to the heart beat and blood pressure of a live business process. In many BPMS, this often takes the form of a dashboard with indicators, meters and graphs to show the performance of a given process. Hence, some BPMS analysts refer to this function as performance management (Silver, 2006) as this is often linked to operational key performance indicators (KPI). To others, this component is known as BAM — short for Business Activity Monitoring (Butler, 2007).

Business users of BPMS use BAM / KPI to perform their own analysis and determine the root cause of process problems. The high level of visualisation provided by BAM provides business users an easier way to identify critical path flows and process bottlenecks. For example, when a late delivery is often experienced on a given day, the process owner can drill down and explore in detail to find the root course. It could be that a parttime courier is employed to deliver on Thursdays and that he or she is not familiar with packing order. BAM tools provide further analytical capabilities, such as statistical correlation to different KPIs and explore interactions.

For a number of orders delivered on-time with a given level of staffing over a given period may provide an insight into resource allocation. At this stage, it is not possible to find a relevant application for the conversational framework. However, in the wider context of course management, indicators such as attendance — online or class room, and exam results may provide insights into general “wellbeing” of a course. The author recognises that further work is necessary to better explore how the BAM function can be applied to teaching-learning process, but it is beyond the scope of this paper.

Extended functionalities

Most commercial BPMS tools also come with built in features for online discussion and other user and group collaborative functions. However, if a university is using the Blackboard feature for posting online discussion, the design environment along with its integration capability will allow seamless integration to the relevant components of Blackboard and other e-learning tools. Some other collaborative functions include instant messaging, blogs and community bulletin boards. Other extended services may include document and content management that is seamlessly incorporated into the process flow. The document and record technology is capable of storing, archiving, indexing, picking and tracking all types of content — structure and unstructured data. This capability could be used to manage learners’ assessments, student projects and mine online discussions for the unstructured knowledge. In the commercial world, this capability is known as case management and function is deployed for processes that have a high level of focus on a particular type of content e.g. applying for a mortgage online. In the teaching-learning process, this function could be used to facilitate a collaborative effort to develop a new course, where creation of a new course is managed as a case that is subjected to review, feedback, adaptation, reflection and finally approval.



The theoretical analysis of the above eight components of BPMS show that at least six of those components or functions meet the requirement of the conversational framework. Of the two components — Simulation & Testing and Analysis & Improvement did not entirely fit into the conversational framework. For both of these functions, empirical research may be required to determine their role in supporting teachers and learners. Returning to the questions that this analysis hopes to answer, (1) Can BPMS be the education technology for teachers to easily and quickly deploy learning online? Almost certainly — this ability is supported by the process modelling, design & development, deploy & execution, integration and business rules engine.

(2) Just as the commercial enterprises look to BPMS for continuous process improvement (CPI), could teachers and lecturers also look to BPMS to continuously improve their teaching and their learners’ learning processes, and in the longer term increase value creation and productivity? This question cannot be fully answered in this theoretical analysis based on the literature reviewed and limited teaching-learning experience of the author, but BPMS may still offer a degree of possibility through its simulation and analytic functions.

The general outlook of this paper is that BPMS as a collective technology and software has the potential make a contribution to improve the teachinglearning process. Although the commercial world and education may share similar needs for ICT in process management, it is not certain from this theoretical analysis of BPMS that it would bring the same benefit to education as it has to commercial enterprises. The benefits experienced by businesses might not be repeatable in all aspects of education, more specifically in the teaching-learning process. Unlike the commercial process, teaching-learning may not be as clear-cut as “the interaction between individual and provider is a very personal contract and learners are not customers, and they are not always right” (Laurillard, 2008b) — the imbalanced, uncertain and delicate relationships that exist throughout the teaching-learning process between teachers and learners make adapting a commercial technology such as BPMS for education a challenging task, but not an impossible one.


Business Process Management (2007) Building End-to-end Process Solutions for Agile

Business. Butler Group.

Hill, J. B., Cantara, M., Deitert, E. and Kerremans, M. (2007) Magic Quadrant for Business

Process Management Suites. Gartner RAS Core Research Note G00152906

Helic, D., Hrastnik, J., and Maurer (2005) An Analysis of Application of Business Process

Management Technology in E-Learning Systems. Available at pubs/elearn2005a.pdf [accessed on March 1st 2009]

Laurillard, D. (2002) Rethinking University Teaching — A conversational framework for the

effective use of learning technologies (2nd ed.). London: Routledge Falmer.

Laurillard, D. (2008a) Open Teaching: The Key to sustainable and effective open education. In T.

Iiyoshi, & M. V. Kumar, Opening Up Education: The Collective Advancement of Education through Open Technology, Open Content, and Open Knowledge. Boston: MIT Press

Laurillard, D. (2008b) Digital technologies and their role in achieving our ambitions for

education. Inaugural Lecture at the Institute of Education. Available at ioewebserver.ioe. [accessed on March 1st 2009]

Silver, B. (2006) Understanding and Evaluating BPM Suites. Bruse Silver Associates. Available at [accessed on March 1st 2009]




0161 A taxonomy of podcasts and its application to higher education


Podcasts are being well accepted by the general public and by higher education institutions. Podcasting combines the advantages of radio and cassettes, such as flexibility, learner control and personalization afforded by recorded audio (McLoughlin & Lee, 2007). As Campbell (2005, p. 34) points out, what´s new about podcasting is the “ease of publication, ease of subscription, and ease of use across multiple environments’’.

Most podcasts are audio files and listening is instinctive while reading has to be taught. Durbridge (1984) emphasizes the advantages of audio over printed media: comprehension is enhanced by the spoken word, adding clarity and meaning, and improving cognition. Podcasts offer new opportunities for creativity, independent learning, and collaboration (Clothey

& Schmidt, 2008). Among other advantages, they focus on issues such as illiteracy and dyslexia, free eyes and hands for other purposes (Clark &

Walsh, 2006 in Rosell-Aguiar, 2007). However, they have implications for the different types of learners: visual learners (as opposed to aural) may not find suitable materials or be able to engage with them. Podcasting started with audio but three varieties can be considered, audio-only podcast, enhanced podcast, and video podcast also known as vidcast or vodcast. Enhanced podcasts combine still images with audio files (Liu & McCombs, 2008;

Salmon et al., 2008).

There are three perspectives in educational podcasting (Harris & Park,

2008): (i) the perspective of lecturers — they facilitate to emphasize the information which lecturers feel to be critical for their students. It enables direct communication and interaction with students which goes beyond temporal and spatial limitations of conventional face-to-face education.

(ii) The perspective of students — it enables repeated learning and offers an opportunity for the effective use of time. (iii) The University’s perspective — podcasting is a communication enabler, reaching out to a wider community.

Podcasts may be used to deliver course materials or provide additional resources for students, providing the potential to allow lecturers to focus on interaction. Functionalities such as pause, forward or skip mean that the user is in control of the pace. Students may be attracted to the new format.

However, as Dixon & Greeson (2006 in McLoughlin & Lee, 2007) reported, more than 80% of podcasts were never downloaded to a portable player or another device, being simply consumed on the PC. Podcasts have been used in higher education with different purposes. In many cases, lectures are being recorded and podcasted, so that students can listen to them later at their convenience; for example to increase their understanding of material covered in specific lectures (Bongey et al., 2008). However, recording a podcast in the class is not a good option because there is background noise and the podcast is usually long and boring (Carvalho et al., 2008).

Podcasting also allows faculty members to offer advanced extra content to highly motivated learners or for remediation to slower learners, although representing an increase in teacher workload for those who create the content (Rosell-Aguiar, 2007), which is often not institutionally recognized

(Carvalho et al, 2009).


Ana Amélia Carvalho

University of Minho, Instituto de Educação e Psicologia, Campus de Gualtar, 4710-057

Braga, Portugal [email protected]

Cristina Aguiar

University of Minho, Escola de Ciências,

Campus de Gualtar, 4710-057 Braga,

Portugal [email protected]

Romana Maciel

University of Minho, Instituto de Educação e Psicologia, Campus de Gualtar, 4710-057

Braga, Portugal [email protected]


In this paper we address the uses of podcasts in higher education and we propose a taxonomy for podcasts. We describe results obtained within a study that is being conducted at the University of Minho, in Portugal, focusing on the use of podcasts and their implications towards learning in higher education. The project involves 6 lecturers from different scientiic domains — Education, Humanities, Social

Sciences, Engineering and Biology. These lecturers created 84 podcasts in order to support their undergraduate and master courses during the 1st and 2nd semesters of 2007/ 2008 and the 1st semester of

2008/ 2009. A total of 479 students — 372 undergraduate and 107 master students — were enrolled in 20 courses.

Some students were not only podcasts listeners but they also had the challenge and the opportunity to create their own podcasts (34 episodes). Podcasts were classiied in different types (Informative,

Feedback, Guidelines and Authentic materials), styles (formal or informal), length (short, moderate or long), purpose and medium (audio or video), according to a taxonomy proposed by the authors. The majority of podcasts was Informative (76), followed by podcasts with Feedback (30),

Guidelines (9) and Authentic materials (3).

Most podcasts were short (102), mainly in informal style and only 21 were vodcasts.

Students´ reactions about podcasts implementation in higher education revealed their acceptance of this new tool and their receptiveness to podcasting in other courses. The majority of students found podcasts a positive resource in learning, although they did not explore one of the main advantages of this technology — portability. Lecturers also found podcasting a useful resource for learning and recognized its great potential as a pedagogical tool but stressed that it is too time consuming.


Most podcasts are created by lecturers, but some lecturers invite their students to create their own (Frydenberg, 2006, Lee et al., 2007, Lee &

Tynan, 2008, Carvalho & Aguiar, 2009). At the beginning it was limited to one-way communication from teachers to students, but recently podcasts have been used to provide a two-way communication between both interlocutors (Harris & Park, 2008). McLoughlin & Lee (2007) also argue that podcasts are moving away from didactic methods of teaching and transmission of content to enable learner agency in the learning process. Lee

& Tynan (2008) believe that the true potential of podcasting technology lays in its knowledge creation value, and its use as a vehicle for disseminating learner-generated content.

Clothey & Schmitt (2008) synthesized three categories of authorship: faculty can direct their students to primary sources found on the Internet, they can create their own presentations for students or ask students to create their own materials to share with their classmates. Rosell-Aguiar (2007) distinguishes two categories of authorship in language learning: authentic content provided by native speakers of the target language and to be used by native speakers, such as news feeds or radio programming, and teaching materials specifically designed for language learning.

A taxonomy of podcasts

A taxonomy of podcast may help teachers create and use podcasts in their courses. There are evaluation criteria for podcasts (Austria, 2008, Schrock,

Table 1: Students enrolled in the study and respective courses (n=479)

Cycle Program Courses

Applied Languages


Applied Biology

Biology and Geology

Computer Science

Conversational Analyzes (CA)

Descriptive Linguistics (DL)

Multimedia Educational Materials (MEM 2007/8)

Multimedia Educational Materials (MEM 2008/9)

Technology and Educational Communication (TEC)

Heredity and Evolution (HE-AB 2007/8)

Heredity and Evolution (HE-AB 2008/9)

Genes and Genomes (GG)

Heredity and Evolution (HE-BG 2008/9)

Operational Systems (OS)

Research Methods (RM 2007/8)

Communciation Sciences Research Methods (RM 2008/9)

Social Psychology (SP)

Hypertext (HY)

Multimedia Systems (MS 2007/8) Educational Technology

Multimedia Systems (MS 2008/9)

Pedagogical Supervision Education and Multimedia Technologies (EMT)

Technologies and Digital


Usability Assessment (UA)

Adults Education and

Communitarian Interv

Educatoinal Mediation and Supervision

Learning and Social Network (LSN)

Leadership and Groups’ Dynamics (LGD)




































































2009), podcast rubric (Bell, 2007), guidelines for podcast production and use (Avgerinou et al., 2007, Hendron, 2008, Ross et al., 2008), and a podcast development model (Edirisingha et al., 2008), but not a taxonomy.

We developed a taxonomy of podcasts, based on literature review

(Geoghegan & Klass, 2005, Cebeci & Tekdal, 2006, Lee et al., 2007, Rosell-

Aguiar, 2007, Clothey & Schmidt, 2008, Edirisingha et al., 2008, Hendron,

2008, Lee & Tynan, 2008) and from our own experience as podcasts producers (Carvalho et al., 2008, 2009, Carvalho & Aguiar, 2009). It has the following assumptions: (i) podcasts are not for use in the classroom; (ii) podcasts are not lectures recorded in the class during face-to-face sessions; and (iii) podcasts should be reusable. Note however that, although reusability is important for any learning object, it depends on its purpose: if the podcast gives feedback to students, it cannot be reused with other students.

The taxonomy we propose has six dimensions: type, medium, length, author, style and purpose.

1. Type: we consider four types of podcasts. Informative (it presents concepts, analysis, synthesis, description of tools or equipments, reading of excerpts / poems, etc.); Feedback / Comments (to students assignments and group work); Guidelines (to field work and to practical work; recommendations about studying, group dynamics, reflective learning etc.), and Authentic materials, this means, materials created for the public and not for a specific course or students, such as interviews, news, radio programming, etc.

2. Medium: audio or video (audiocast, enhanced podcast, vodcast and screencast). Audio podcast is the most common, and enhanced podcast is gaining popularity, which combines images and audio. Video podcast is also mentioned as vodcast, and if it is a screen captured with audio, it is called screencast (Edirisingha et al., 2008).

3. Length: Short (1–5 minutes), Moderate (6–15 minutes) or Long (>15 minutes). Podcasts should not take more than 30 minutes if conveying detail and facts, as suggested the Scottish Council for Educational

Technology (1994). Cebeci and Tekdal (2006) proposed podcasts nolonger than 15 minutes, because long podcasts generally cause a loss of attention and a subsequent decrease in comprehension. Lee and Chang

(2007) created podcasts of 3–5 minutes in a radio style version and recommended short, lively and entertaining podcasts. The rule should also be that podcasts’ purpose and content determine podcast length.

4. Author: Lecturer, Student, and other (experts, local community, and representatives). Lecturers can create their own podcasts for students, they may use authentic materials found on the Internet, such as interviews, or they can also ask students to create their own podcasts to share with their classmates.

5. Style: Formal or informal. Style is related to the degree of formality adopted. Edirisingha et al. (2008) mentioned that to make podcasts more interesting, they may incorporate informal learning content such as people´s experiences, opinions, and so on. “A friendly tone invites students to learn and helps to build intimacy with the speaker”

(Edirisingha et al., 2008, p. 165). A podcast should have a beginning, middle and an end, three important parts in keeping students’ attention.

Audiences like structure applied in a new and surprising way (Geoghegan

& Klass, 2005). It is important to engage students and is better to keep content short and simple, clear and concise (Hendron, 2008).

6. Purpose: described as an action verb (inform, analyze, develop, motivate, mediate for reflective learning, etc.).



This paper reports the use of podcasts in higher education and describes results obtained in the context of a project that is being conducted in Portugal, at the University of Minho. The project aims to implement podcasts in learning contexts and to evaluate its implications for learning while assessing students’ and lecturers´ reactions to this new pedagogical tool.This study involved 20 courses and 6 lecturers, who produced and used 118 podcasts during 3 semesters (1st and 2nd semesters of 2007/ 2008 and 1st semester of 2008/2009).

Table 2: Characteristics of informative podcasts (n=76)

Cycle Author Courses Number Length Purpose








Apply a speciic knowledge acquired in the classroom

Complete and develop a subject discussed in class















Short Synthesis of a subject matter





















MS 2007/8
























Information about how to use the forum in the blackboard

Give learning outcomes and information about study resources



Give extra content by reading a text

Give course content

Moderate Give course content


Give learning outcomes and information about study resources



Give extra content by reading a text

Explain the resolution of an heredity exercise

Short Give course content

Moderate Give course content

Moderate Describe concepts and technology

Clariication on the project and about voluntary participation

Clarify evaluation rules

Motivate to read a book






Develop extra course contents

Indicate aspects to be focused during next session

Clariication on the project and about voluntary participation

Clarify evaluation rules





HY 21 Short

Motivate to read a book

Present oneself to a friend, to the family or to an employer










Develop extra course contents

Style Medium
































Data collection instruments

Data was collected by two previously developed questionnaires. A Digital

Literacy Questionnaire (DLQ) was filled in by students at the beginning of each course and was set to characterize students’ knowledge and uses of web 2.0 tools. The second questionnaire — an Opinion Questionnaire

(OQ) — was filled in at the end of each course to inquire students’ reactions to the use of podcasts. Interviews were also conducted with some students and lecturers.

Sample characterisation

Students and courses

A total of 479 students — 372 undergraduate and 107 master students — participated in this research, being enrolled in 20 courses. The majority of students was female (67%) and this was the case in almost every course (Table 1) except in Operational Systems (OS), where males were overrepresented (77%).

The undergraduate students were enrolled in 13 courses belonging to different programmes: Biology (4), Engineering (1), Communication Sciences

(3), Portuguese Studies (2) and Education Sciences (3). Almost all masters students were teachers enrolled in masters courses in Education (6) and

Digital Art (1) (Table 1).

A total of 118 podcasts (56 in undergraduate and 62 within master courses) of varied types, lengths and with several purposes were created by different authors in the podcasts project. Most students (57%), either undergraduate

(58%) or graduates (51%), did not know what a podcast was, though they were used to downloading music and files (64%) or software (55%).

Data analysis

Podcasts uses

Podcasts were recorded and delivered with different types, lengths, purposes, authorships, media and style. Podcasts characteristics are summarised in

Tables 2 to 5, where information about the cycle of studies, courses´ names

Table 3: Characteristics of podcasts with feedback / comments (n=30)

Cycle Author Courses Number Length Purpose





1 Short

Comment the corrections done about the analysis of educational multimedia software or videogames











Personalized feedback to group assignments

Personalized feedback to group assignments







MS 2007/8

MS 2008/9


















Comment about students posts in the forum

Comment students presentation about the learning theories

Comment contributions to the forum (about 3 grids of software analysis)

Comment about students’ answers to DLQ

Comment podcasts created by students

Comment students’ answers to some questions about module 1

Comment students analysis of Vercial Project

Personalized feedbacks

Comment peer work on educational software analysis

Style Medium

I Audio
















and the podcasts used in each course is also available.

Informative podcasts were created by all the lecturers either for undergraduate or masters students (Table 2). This is actually the most used podcast category, with 76 episodes. Most of them were only audio files (55) and 21 were vodcasts. The majority was of an informal style, but 4 were of a formal style. Informative podcasts were mainly short (63), some moderate

(9) and 4 were long. Their purposes were very diverse: to give several kinds of instructions / information, to motivate students, to present and further develop course contents. Students were also challenged to create their own episodes producing 25 informative podcasts. Such podcasts were either short audiocasts (4) recorded informally to make a synthesis of a particular course subject, or short videos (21) in a free style (depending on owns choice) and with a common purpose: to present students to someone they wish to

(friends, family, employer).

Podcasts designed to give feedback to students were the second most used podcast type in this study, with a total of 21 audiocasts delivered by lecturers and 9 episodes recorded by students. All were short and mainly informal

(Table 3).

Only 2 lecturers adopted podcasting to give feedback to their students, mainly to comment assignments of several kinds. This podcast type, which could be seen as a powerful one, as it can be applied by every teacher in every course, does not seem to be very popular, as inferred by the lecturers´ reaction to the implementation of podcasts in University of

Minho (Carvalho et al., 2008). Actually, the possibility of reusing podcasts is a concern shared by such lecturers, who wish to “make our efforts and work with this tool worthwhile”. Podcasts used to give personalised feedback or comments — to a particular student or a certain group — are not attractive because they can not be reused in other courses or in another year. Students also had to produce podcasts of this type to comment their classmates´ assignments on educational software analysis.

Such episodes were all short and formal audio files.

Podcasts to guide students´ work or to give recommendations were also created: a total of 9 short audio files of both styles, all produced by lecturers

(Table 4).

The longer podcasts used in this project were: 3 interviews with experts in an informal style (Table 5). They were delivered by Lecturer F, who also created 3 long informative podcasts (Table 2). The purpose here was to motivate students and inform them about the potential of the interview method or of the non-intrusive methods of research, again in an audio format of informal style.

Table 4: Characteristics of podcasts with guidelines (n=9)

Cycle Author Courses Number Length Purpose













Guidelines to assignment

Recommendation to the next session

Orientation to website analysis, group work and about the upload of group work in the






2 Short Give orientations to study











Guide the group work

Guide the WebQuest report

Guides the inal assignment

Guidelines to assignment (a critical review of a paper about usability evaluation)

Style Medium

I Audio












Students’ reactions to podcasts

Podcast listening

The majority of students, either from undergraduate courses (87%) or from masters courses (96%), listened to the podcasts delivered by their lecturers, mainly through the Blackboard e-learning platform (50%) or on personal computers (49%) –owned by 96% of all the students — and very rarely a

MP3 player (4%) or other mobile devices. Such results are in agreement with the ones achieved by other authors (Dixon & Greeson 2006 in McLoughlin

& Lee, 2007; Young, 2007). Students had to listen to podcasts again (73% undergraduate and 81% master students), and they did so mainly to revise its contents (80%) or to take notes (23%). Besides listening to podcasts, several students also felt the necessity to write down their contents, either totally

(7%) or partially (31%).

Students think that the best and more useful podcasts are the ones which give summaries (51% of the students’ answers), guidelines (46%) or those which deliver contents (43%) and news (40%). These categories can be classified according to our taxonomy in the informative (summaries, news, contents) and guidelines types. Students also showed their preference towards short (29%) or moderate (38%) podcasts. The great majority of them regarded the integration of podcasts in learning as an advantage

(90%), independently of being undergraduate or graduate students, and were receptive to having podcasts in other courses (81% undergraduate and 92% master students).

Podcast quality

When asked about podcast quality parameters, students who listened to the podcasts mainly pointed out its audibility (95% of undergraduate and

93% of master students) and noted almost without exception that they were clear (89% of undergraduate and 93% of graduate students) and that the lecturer voice was a friendly one (89% of undergraduate and 93% of master students).

Students mentioned that they listened to podcasts carefully and attentively

(44%). Some of them also stressed the sensation of proximity they felt with their lecturers while listening to podcasts (28%). However, they would still like to have podcast content in a written format, a preference more frequent in undergraduate (48%) than in masters (36%) students. Generally, students did not consider podcasts long except the ones enrolled in Research

Methods (RM) courses. These podcasts were interviews with experts and were the longest episodes. In students’ opinions not only were they “difficult to pay attention and keep listening the recorded file, but we had to take notes and stop/ restart the record often in order to achieve a full comprehension or to write down the necessary notes”.

Students as podcast creators

As creators, students (n=34) seem to have appreciated the experience of producing a podcast. In fact, all of them referred “it has been an

interesting, new, innovative, useful and also funny experience”. Some of them

(33%) reported some technical problems in recording and a few stressed

Table 5: Characteristics of podcasts of authentic materials (n=3)

Cycle Author Courses Number Length Purpose

RM 2007/8 1




Motivate students and inform about the potentialities of the interview method





RM 2008/9

1 Long

Motivate students and inform about the potentialities of the non-intrusive methods of research

Style Medium

I Audio


the strangeness felt when they heard their voices (22%). However, the pedagogical potential offered by this new tool was undoubtedly recognized by 56% of the students who produced episodes.

Lecturers’ opinions about podcasts

The lecturers enrolled in this project concluded that creating podcasts is difficult and can be a very time-consuming task: besides “the need to get familiar with the adequate software”, it is necessary “to write, rehearse and record what one wants to say”. Most of them also referred that “the time spent and the effort made are not recognized by the institution” and the possibility of reusing podcasts in other teaching/ learning contexts is a major concern for those who want to continue using this tool. Nevertheless, all the lecturers considered their participation in the project “a very positive experience” and they all recognized a great potential in this pedagogical resource. Podcasting can actually be a very useful and powerful strategy for improving classes and motivating students: they still are a pedagogical innovation and they may allow time for the development of other activities in class. Also, podcasts are permanently available, allowing students to listen to their content at any time, whenever they need or want it.

In conclusion, and in spite of the drawbacks identified, lecturers considered the introduction of podcasts in their courses a very positive experience and they plan to continue using podcasts, with the goals of minimizing the required production time, reusing podcasts in other pedagogical experiences and enlarging its use to other contents and other podcast dimensions.


Podcasts were adopted by learning institutions and are being increasingly used to support pedagogical environments in higher education, making a taxonomy of podcasts in teaching and learning useful and necessary. We proposed a taxonomy based on a literature review and on our own research with podcasting implementation in higher education.

Within our project we created 118 podcasts spanning all the possibilities in every dimension proposed in the taxonomy. Thus the episodes had variable type, style, length and medium, being created by different authors and with different purposes. The taxonomy proved to be simple, easy to use and allowed to classify all the podcasts created or reused.

Students and lectures were receptive to the use and creation of podcasts.

However, students did not take full advantage of this technology as they do not use mobile devices to listen to podcasts. Further research is needed to understand this limitation.


Austria, J. (2007) Developing Evaluation Criteria for Podcasts. In Libri, 57, no. 4: 179–272. [accessed February 2009]

Avgerinou, M., Salwach, J. and Tarkowski, D. (2007) Information Design for Podcasts. In

C. Montgomerie & J. Seale (Eds.), Proceedings of ED-MEDIA, 754–756. Chesapeake, VA:


Bell, A. (2007) A+ Podcast Rubric. University of Wisconsin, Stout. Available at www.uwstout.

edu/soe/profdev/podcastrubric.html [accessed January 2009]

Bongey, S., Cizadlo, G. and Kalnbach, L. (2008) Explorations in course-casting: podcasts in

higher education. Campus-Wide Information Systems 23, no. 5: 350–367.

Campbell, G. (2005) There’s Something in the Air: Podcasting in Education. Educause, 33–46.

Carvalho, A.A. and Aguiar, C. (2009) Impact of Podcasts in teacher Education: from

Consumers to Producers. Proceedings of SITE, 2473–2480. Chesapeake: AACE.

Carvalho, A.A., Aguiar, C., Santos, H., Oliveira, L. and Marques, A. (2009) Podcasts in

Higher Education: Students’ and Lectures’ Perspectives. IFIP — 9th WCCE (accepted).

Carvalho, A.A., Aguiar, C., Carvalho, C.J. and Cabecinhas, R. (2008) Influence of

Podcasts Characteristics on Higher Students’ Acceptance. In C. J. Bonk, M. M. Lee & T.H.

Reynolds (eds), Proceedings of E-Learn, 3625–3633. Chesapeake: AACE




Cebeci, Z. and Tekdal, M. (2006) Using Podcasts as Audio Learning Objects. Interdisciplinary

Journal of Knowledge and Learning Objects, 2: 47–57. Available at v2p047-057Cebeci.pdf [accessed May 2008]

Clothey, R. and Schmitt, C. (2008) Education in Motion: Innovating with iPods. Proceedings of ED_MEDI, 629–639. Chesapeake, VA: AACE

Durbridge, N. (1984) Audio cassettes. In A. W. Bates (ed.), The Role of Technology in

Distance Education, 99–107. Kent, UK: Croom Helm

Edirisingha, P., Salmon, G. and Nie, M. (2008) Developing pedagogical podcasts. In G.

Samon and P. Edirisingha, eds., Podcasting for Learning in Universities. New York: Open

University Press: 153–168

Frydenberg, M. (2006) Principles and Pedagogy: The Two P’s of Podcasting in the

Information Technology Classroom. ISECON — EDSIG, 23: 1–10. Available at isecon/2006/3354/ISECON.2006.Frydenberg.pdf [accessed May 2008]

Geoghengan M. and Klass, D. (2005) Podcast Solutions: The Complete Guide to Podcasting.

Berkley, CA: Apress.

Harris, H. and Park, S. (2008) Educational usages of podcasting. British Journal of

Educational Technology 39, no. 3: 548–551

Hendron, J.G. (2008) RSS for Educators: blogs, newsfeeds, podcasts, and wikis in the classroom.

Washington, DC: ISTE

Lee, M., McLoughlin, C. and Chan, A. (2007) Talk the talk: Learner-generated podcasts as

catalysts for knowledge creation. British Journal of Educational Technology 39, no. 3: 501–521

Lee, M. and Tynan, B. (2008) Podcasts and distance learning. In G. Samon and P. Edirisingha, eds., Podcasting for Learning in Universities. New York: Open University Press: 92–102

Liu, Y. and McCombs, S. (2008) Why the students did not use podcasts? In K. McFerrin et al.

(Eds.), Proceedings of SITE, 2093–2094. Chesapeake, VA: AACE.

McLoughlin, C. and Lee, M. (2007) Listen and learn: A systematic review of the evidence

that podcasting supports learning in higher education. In C. Montgomerie & J. Seale (Eds.),

Proceedings of ED-MEDIA, 1669–1677. Chesapeake, VA: AACE

Ross, J., Gorra, A. and Finlay, J. (2008) Practical Tips for Creating Podcasts in Higher Education.

Proceedings of the 13th annual conference on Innovation and technology in computer science education 311–311.

Rosell-Aguilar, F. (2007) Top of the pods — In search of podcasting pedagogy for language

learning. Computer Assisted Language Learning 20, no. 5: 471–492

Salmon, G., Mobbs, R., Edirisingha, P. and Dennett, C. (2008) Podcasting technology. In

G. Samon and P. Edirisingha (eds.), Podcasting for Learning in Universities. New York: Open

University Press, 20–32

Young, D. (2007) iPods MP3 Players and Podcasts for FL Learning: Current Practices and Future

Considerations. NECTFL 60: 39–49

Schrock, K. (2009) What makes a good podcast? Kathy Schrock’s Guide for Educators. Available at [accessed February 2009]

Research funded by FCT, reference PTDC/CED/70751/2006; CIEd.



Preeti Shah

The Robert Darbishire Practice,

Rusholme Health Centre, Walmer Street,

Manchester M14 5NP [email protected]

Hilary Dexter

186 Waterloo Place, Oxford Road,

Manchester M14 5TB [email protected]

Tim Dornan

North Manchester General Hospital,

Delaunays Road, Crumpsall, Manchester

M8 5RB [email protected]

Neil Snowden

Salford Royal NHS Foundation Trust,

Stott Lane, Salford M6 8HD [email protected]


Professional trainees in the workplace are increasingly required to demonstrate speciic standards of competence. Yet, empirical evidence of how professionals acquire competence in practice is lacking. The danger, then, is that efforts to support learning processes may be misguided. We hypothesised that a systemic view of how expertise is acquired would support more timely and appropriate development of technology to support workplace learning. The aims of this study were to provide an empirically based understanding of workplace learning and explore how learning could be facilitated through suitable application of technology.

We have used the medical specialist trainee as an exemplar of how professionals acquire expertise within a complex working environment. We describe our methodological approach, based on the amalgam of systems analysis and qualitative research methods. We present the development of a framework for analysis and early indings from qualitative data analysis.

Based on our indings so far, we present a tentative schema representing how technology can support learning with suggestions for the types of technology that could be used.

0102 Can acquisition of expertise be supported by technology?


For many professions, the acquisition of expertise is expected to occur by learning from experience on the job. However, learning from experience is a phrase that has dominated both adult education and learning in the workplace without much critical attention. The medical profession is a prime case of where learning is expected to occur within a complex working environment. Adequate professional training is fundamental in the context of patient safety and quality of care, yet a look at the literature reveals that approaches to professional training are based on little examination or underpinning knowledge of how professional expertise is acquired (Eraut 2004). Although medical education has an expanding body of research in its own right, most of the emphasis has been placed on the formal training years.

Reforms within professional education have arisen in response to public concern over professional inadequacies and a need to increase the standard and structure of training. Accountability has increased and trainees are expected to demonstrate their capabilities more explicitly. Formal curricula now guide their learning. However, these do not take into account the complexity of the working environment. Amongst trainees, there is a perceived lack of connection between curriculum documentation and workplace experiences. This difference is yet to be articulated and trainees thus make little use of their formal curricula.

Despite such reforms, medical education has to cope with shortages of teaching physicians, increasing demands on service provision, shortening of training hours and reductions in funding. Consequently, conditions for adequate workplace learning are often poorly met. Trainees are faced with the challenge of making the most of the experiences they are exposed to. So far, methods to support training have not been based on evidence of where improvements are required.

There is widespread use of technology to support training and it has been recommended, amongst other things, as a way of improving workplace education. Although there are numerous accounts of technological innovations to enhance and support training, most technologies simply act as didactic tools, media of communication, sources of learning materials, or means of delivering assessments. The tools implemented are often limited by a narrow spectrum of facilities. There have been limited advances in technology to support experiential learning in medicine

(Greenhalgh 2001). Where technologies have been introduced, it has been done with little prior analysis of the system they are required to support

(Childs and Hall 2005).

The changing nature of work practice and increasing demands on professionals to carry out a quality service need to be offset by an increase in the quality of the learning experiences. There is a need to reconsider how technology can be used to scaffold learning with suitable and timely interventions. A clearer view of the how professionals develop expertise within a complex environment is indispensable if improvements are to be fostered appropriately.

In this paper we provide a background to this research area. We provide a rationale for the selection of research methods and analysis, followed



2 by a case study in progress. Preliminary findings are summarised and the approach evaluated. First steps are taken to identify the ways technology could intervene. We conclude the paper with a summary of limitations and strengths of this approach and outline potential avenues for the future.


The specialist medical trainee is a good exemplar of how complex expertise is acquired in workplace settings. After 4–5 years of formal undergraduate training and a year of satisfactory practice based training, a full registration to practice is obtained. The postgraduate years usually involve a few years of general training, followed by a period of specialist training in a particular field. These latter years of training are essential in preparing doctors for independent practice and are crucial in shaping their habits, behaviours and attitudes towards patients. The training programme amounts primarily to a form of apprenticeship, interspersed with formal didactic teaching.

Breadth of experience is gained by working within a variety of hospital organisations and specialist departments. As service demands increase, trainees are increasingly required to work under conditions of time pressure. Heavy workloads, long working hours, fatigue and perceived lack of support can lead to stress, dissatisfaction, inadequate job performance, and burnout amongst trainees (Veasey et al. 2002). Despite this challenging environment, it is necessary for doctors to acquire a wide and complex range of knowledge and skills and integrate them appropriately into their practice.

They do this through a process of progressive independence, requiring less supervision until they become expert practitioners (Kennedy et al. 2005).

Attempts to characterise workplace learning have been made by several researchers (Hager and Halliday 2006) and have resulted in increased awareness of the significance of informal learning processes. In contrast to

Figure 1: The system framework model

Learning Plan Reflection








Support for learning Barrier to learning




Support for work Barrier to work

Interaction with learning









Level of expertise


Interaction with work












Task Resource




formal learning, which tends to be structured and more institutionally based, informal learning is holistic, contextual, and activity / experience-based. It is often unplanned and unpredictable, arising in situations where learning is not the main aim. As a result, the learner is often unaware of the extent of their learning, even though they may be aware of their ability to perform the job. Studies in this area have alluded to the inseparable and implicit nature of learning at work and the importance of tacit knowledge learned in the workplace (Eraut 2004).

Most of our understanding of how professionals acquire expertise comes from the cognitive psychology literature. Cognitive processes based on the transformation of experiences, through reflection and action, have been highlighted within experience based learning (Kolb 1984; Schön 1983). Such theories have provided insights into how physicians learn to solve clinical problems, whereby previous clinical experiences prove fundamental in helping to solve future problems. Dreyfus and Dreyfus (1986) propose that practitioners acquire their skills in practice according to progression from novice to advanced beginner, competent, to proficient and finally to expert.

The novice may depend on concepts learnt through formal processes or the use of guidelines. The more experienced practitioner tends to learn through more informal self-initiated mechanisms, constructing their knowledge themselves in the context of their practice. The model emphasises pattern recognition, intuition, and reflection as critical to the development of professional skills. Critical to the acquisition of expertise is for the more expert practitioner to encounter less familiar or more complex problems.

These require adaptation of the learnt scripts and more critical thinking

(Schmidt, Norman, and Boshuizen 1990).

Social theories of learning have also provided some invaluable insights into workplace learning. Such models suggest an important social dimension to learning. Professionals in the workplace learn from interactions with others within the environment. Central to the learning process is participation in activities, which fosters and is fostered by socialisation within a community of practice (Lave and Wenger 1991).


We had to choose a method capable of describing how expertise is acquired.

It would therefore have to tackle an area of high complexity that is not well understood. The one chosen was an amalgam of systems analysis and qualitative analysis, aiming for synergy between the two fields. The dual approach would allow a systemic view to be taken and an interpretation within complexity, but also arrive at pragmatic, solutions in the face of that complexity. Qualitative methods enable exploration of phenomena in their natural settings, characterising the meanings, experiences, and views of all the participants through rich and detailed descriptions. Systems analysis generally uses semi-formal visual models as a way of filtering out the complexity of large systems. In abstracting complexity, it becomes possible to identify areas in the system that might be enhanced or better supported.

Representation of the workplace learning system through visual models

(created in the Unified Modelling Language (UML)



) allows development of a shared representation of how trainees are learning and an understanding of where learning could be facilitated. System modelling provides the basis for requirements specification for people-based and technology-based support of the workplace learning system.

Case study

This paper reports a case study that is partially completed but already yielding useful outputs. Doctors in rheumatology specialist training spanning

1 Uniied Modelling Language




3 regions of England (North-East, North-West and West Midlands) are being recruited into the study. Approval for this work has been granted by the

National Research Ethics Service and written informed consent has been obtained from each participant. Data are being obtained by two methods:

Observation and Audio Diary.


Observation of trainees in their workplaces was chosen to allow the workplace processes that lead to learning, to be captured for subsequent analysis. Abstracting out parts of the system which are important for understanding learning requires a broad analysis. A purposive sampling approach was therefore adopted to achieve maximum variation. A variety of trainees were sampled across a breadth of work based activities and settings.

Two researchers (P.S and H.D), whose different disciplinary backgrounds gave them different perspectives on the research field were chosen to carry out the observational work. P.S. is a doctor familiar with the research field and participants under study. H.D is a systems analyst with no prior knowledge of the research field. The final goal is to conduct four observation periods (3–4 hours) per month over 1 year. A non-participatory approach to observation is being used. Brief discussions in between activities, or short

Table 1: Qualitative analysis indings

Framework Category / Theme Examples of Findings

Q1 Domain


Q2 Role

Q3 Work

Q4 Artefact (Resources and outputs)

Q5 Support for work

Q6 Barriers to work

Q7 Support for Learning

Q8 Barriers to Learning

Q9 Competencies

Q10 Level of expertise

Hospital: Teaching, District General, Community.

Out-patient clinic, In-patient duties, Clinical meetings.

Specialist trainee, Patient, Medical students, Consultant, Junior doctors, Nurses, Allied health professionals, other medical teams sharing care.

Clinical: reviewing patients in many contexts, taking case histories, clinical examination, providing advice to patients, making clinical decisions, giving and requesting specialist advice, interpreting investigation results, carrying out clinical procedures.

Generic: preparing for patient reviews, giving and receiving updates of current work, requesting investigations, reports and appointments, attending meetings, teaching, clinical research, searching for appropriate clinical evidence, reading books and journals, paper work, audio records.

Resources: Patient appointment list, Patient case records, codiied information (e.g.

Google, British National Formulary), task lists, clinical guidelines, procedural and clinical investigation artefacts.

Outputs: Investigation results (e.g. radiographs), clinical specialist opinions, treatment plan.

Patient records system, computerised clinical workstation, translator services, laboratory services, other clinical specialists, clinical support staff.

Disorganised patient case records, non-integration of support systems, lack of resources e.g. appointments for patient review or clinical investigation

People: Consultants, peers, patients, allied health professionals, other specialists.

Artefacts: Patients case records, work-based assessments, codiied information (sourced from people, internet, books, journals), tacit knowledge shared between the team.

Resources: Case based discussions, formal teaching sessions, feedback on performance

Eficiently run outpatient clinics with appropriate allocation of cases, variation of work activities and case-mix.

Heavy workload, shortage of staff, insuficient time to pursue personal goals, insuficient communication with team, lack of relection in action, individual persona.

Generic skills: personal organisation, working as a team, supervision and teaching, negotiating the organisational system.

Clinical skills: Communication (e.g. rapport building, counselling, negotiating agenda), clinical decision and management, diagnostic reasoning, procedural skills.

Novice trainee demonstrates high level of competency in generic skills. Requires direct supervision in specialist clinical skills. Move to autonomous practice is dependent on complexity of competency being developed.

Expert trainee demonstrates high level of competency in a range of generic and specialist skills. Works autonomously in practice with arms length supervision.

Feedback mainly requested to conirm adequacy in decision making skills and competence.


Figure 2: Mapping support for workplace learning

Find learning support

Map support needs


Learning type

Tech affordances

Learning activity type

Access information

Log issues/ questions


Plan learning

Ask an expert

Discuss with peers

Ask supervisor

Get feedback on a case

During the working day

Immediate - real time


In own time - after work

Planned learning time

Position in curriculum

Find opportunities




Privacy and


Ease of use

Depth of treatment






Semantic search

Curriculum mapping

Reflective journal

Social network

Group of peers


Group of experts debriefing interviews with participants following the period of observation, are being carried out when possible. Observations are recorded as detailed contemporaneous field notes and researcher debriefings following each period of observation. Field notes are transcribed immediately afterwards.

Inter-observer concordance will be verified by comparing field notes of observations carried out in tandem.

So far, 29 hours of observational fieldwork in three hospitals have been completed including: General and specialist outpatient clinics (n=5), review of in-patients during ward rounds (n=4) and clinical meetings

(n=1). Participants have included six specialist trainees ranging in seniority from first to seventh year; they have included both UK (n=3) and non UK



2 graduates (n=3). Further purposive sampling of trainees will be carried out to observe a range of procedural skills, interactions between specialist teams, clinical meetings, and teaching activities across different hospital settings.

Audio diaries

Most studies attempting to characterise internal processes of learning have used interviews. A novel approach is to use audio-diaries. In contrast to interviews, diaries allow participants to record their experiences contemporaneously, resulting in a higher level of recall (Knight and

Sweeney 2007). Trainees are being asked to report salient experiences in their workplace activities, which they find more or less valuable as learning opportunities. In education research, this approach has been useful tool for exploring what people find difficult or stimulating in their work and for discovering experiences that are perceived as being particularly meaningful.

Participants are given verbal and written guiding prompts. Trainees have been asked to maintain their diaries over 7 clinical working days, which may or may not be consecutive. Analysis of the diaries will include an audioanalysis to capture points of emphasis and expression, followed by a detailed analysis of written transcripts. To date, eleven specialist trainees have completed audio-diaries.

Data analysis

A generic system model was developed to provide an initial set of nodes to code the observation notes and transcripts of the audio diaries. The model was based on an approach used to elicit system requirements from domain experts in which they engage in facilitated discussion or system modelling activities around a set of 5 key questions (Dexter 2007):

1. Where are we? (Our workplace domain and its boundaries)

2. Who is here? (The roles (areas of responsibility) of people here)

3. What are we doing? (The tasks given to the trainee to carry out)

4. What supports our work? (Needed systems and services around the domain)

5. What are we using and producing at work? (Resources and outputs of the tasks)

The questions are an expression of a working system but do not specifically address a workplace learning system. The model was therefore extended and documented as a UML Class Diagram (Figure 1).


From this model of the system, the set of coding nodes for the first iteration of qualitative analysis of the data was:

1. Domain (where the work is taking place)

2. Roles (people and their responsibilities)

3. Work (the set of tasks that the trainee is carrying out)

4. Artefacts (resources for doing tasks and outputs from tasks)

5. Support for work (external systems and services needed to complete


7. tasks)

Barriers to work (things that get in the way of efficient or effective work)

Support for learning (people and things that are around to help


9. workplace learning)

Barriers to learning (things that get in the way of workplace learning)

Competencies (knowledge, skills and personal qualities required for a task)

10. Level of expertise (advancement of the trainee towards expert standing)

We are currently applying the framework to the first sets of data collected.


All the data coded against each category (node) will be identified and examined to establish analytical subcategories. New themes which may be discovered during analysis will be back checked against earlier coding using the process of constant comparison. All four members of the research team will code a number of transcripts separately to check their degree of concordance. Data analysis and collection will be iterative to allow purposive sampling and exploration of new lines of inquiry.

A sample of the findings is presented in Table 1 in terms of the ten nodes that formed the basis for the analysis.

The acquisition of competency in the workplace

The combination of qualitative research methods and systems analysis is generating useful outputs. Our results are beginning to indicate how trainees are learning. Through their work, trainees acquire a complex range of competencies, with several different knowledge types being utilised within any one activity. Data suggests that codified knowledge assists them in their practice. A cultural knowledge of how to negotiate the organisational system is also a necessary pre-requisite to their performance at work.

Activities which involve participation within a team appear to be particularly associated with learning outcomes. Through social interactions and team dialogue, knowledge is frequently shared and contextualized within its narrative. This is usually within clinical case based scenarios or other forms of story telling. They perceive their learning to be more restricted during activities in which they are less socially supported. In some instances, trainees describe a hierarchical approach used by supervisors to tests their individual knowledge. In other instances, they describe how team dialogue and feedback proceed non-confrontationally. Through socialisation the trainee is given the opportunity to learn by modelling their behaviour on the activities, actions and knowledge of those around them.

‘I found it very useful going through the differential with the consultants as it gave an opportunity of understanding the thinking pattern which is how I feel that I best learn, that is to say, understanding how other people think and adapting that into my thinking strategy if I feel that it’s appropriate.’

Participation in authentic practice is fundamental to the acquisition of expertise. Within the clinical scenario, the patient acts as the primary learning resource. This might be through direct interaction with the patient or through discussion of a case with others. Trainees describe how they acquire expertise in assessing and managing a particular case by experiencing it repeatedly within different contexts. Some trainees reflect on how a change in practice can be stimulated by a recent discussion, feedback, or error based around a similar case. The busy schedules of most trainees, however, seem to leave little room for self-initiated reflection. There is some evidence that the study itself is acting as a form of intervention, by encouraging trainees to reflect on their learning.

How might this study feed into the design of technology or people systems?

Findings could potentially be fed into the design of technology-based or people-based systems in a number of ways. Design of the right kind of technology support for workplace learners needs to consider the types of activity taking place in the working day, the timeframe in which support is needed or in which the learner is able to use the technology and the particular affordances of the technologies. Figure 2 shows these three aspects and how they may be used to ‘map’ a support requirement to the most appropriate technologies.




There is a rich array of tools available in the web 2.0 world to support learning (Franklin 2008) but they have to be offered in a way that will serve a learner who has very little free time during the working day and already has to deal with a very complex environment. The different technologies may be used in combinations with each other and with bespoke software applications. In this case study, early findings suggest that one of the most useful technologies might be a social networking platform to support groups of peers and of experts. This platform would offer a range of tools supporting conversations, such as micro-blogging for obtaining immediate response to quick questions and obtaining feedback on clinical decision making. Discussion forums or wikis could be used for following opinion on particular topics. A reflective journal (either text or audio) which would allow selected parts to be shared with the community is also a candidate technology for enhancing workplace learning.

The theory of workplace learning and sharing of expertise in communities of practice as laid out by (Wenger, Mc Dermott, and Snyder 2002) has made a significant contribution to understanding but may not deal sufficiently with the community boundaries in complex institutions (Fuller et al. 2005). The learners observed here pass through a number of different but overlapping

‘communities’ in their work and any group management in a social network would need to reflect this.

The learners require a map of areas of expertise and the appropriate channels of access to experts in each. Their interactions with located experts will vary in response-time and degree of formality and any provided technology will be required to support this. It will be necessary to establish the most useful types of social interactions (Brouns et al. 2008; Walter, Battison, and

Schweitzer 2008) since it would be counter-productive to expose the learners to ‘noise’ in the form of connections to people who are not valued or trusted, or are not relevant to the area of expertise. Examples of design for systems to support the location and reuse of tacit expert knowledge can be found in the engineering sector (Collison and Parcell 2001;Woo et al. 2004) and provide a basis for part of the on-going research design.


The study so far has a number of limitations. The data source is confined to a single discipline and specialty within the medical profession. The number of participants observed thus far is limited and, as yet, we are still continuing to find new themes emerge from our data. Data collection will continue until no major themes emerge from new data collected. However, the study is still in its early stages and requires further purposive sampling before saturation of data is reached. Data collection in this kind of study relies on voluntary participation. The quality of data is heavily dependent on access to the research field for observation and on the motivation of the participant keeping the audio diary. To enhance transferability, sampling across other medical disciplines may be useful.

Although observational research is relatively time consuming, the effort has yielded rich and descriptive. The iterative process of data collection and analysis will allow us to see when new data is yielding no new major findings. This will ensure data collection is limited to that necessary to yield useful outputs. Using a framework approach to the analysis allows our findings to be interrogated for reproducibility by independent researchers, thereby increasing the construct validity of the findings.

The findings suggest that our approach is also taking us closer towards developing a better understanding of how expertise is acquired in the workplace. The use of qualitative research methods have provided us with the rich descriptive data required to analyse this area of study. The selection


of data sources have proved useful in providing different perspectives on learning. Triangulation of these data sources to gather internal and external perspectives on learning appears particularly pertinent to yielding rich findings. The observations carried out so far have provided a more systemic overview of the workplace environment, particularly in relation to the organisational factors, which are supportive and constraining to their work. The observations are providing useful insights into areas where learning occurs more implicitly. The use of two observers has been helpful in obtaining different perspectives of the learning environment. Sampling purposively across a broad number of settings and activities is allowing gradual construction of a systemic overview and comparison of the learning environment across different contexts. This has led to a decision to continue to develop this approach to direct the rest of the study. The methods developed and the approach used within this study, have the potential to be adopted usefully by others who are carrying out research in this field. It is expected that at least some of the findings will be generalisable to a wider arena, particularly within the health profession.


In conclusion, the chosen methods are allowing us to characterise how expertise is acquired in the workplace. The combination of qualitative research methods and systems analysis has led to a first iteration framework from which to analyse the data. The framework is leading us towards a systemic representation of trainees’ work-based learning environments and we have shared some of our early findings in this paper. Our analytical framework has provided a starting point from which to interrogate each aspect of the model further. Analysis of interrelationships between framework categories will hopefully lead to further conceptual development within our findings.

As the research develops iteratively, further categories for analysis are likely to be identified. They will be incorporated, leading to refinement of our first iteration framework. Analysis of workplace learning is also likely to benefit from further triangulation with additional participant interviews. This will allow us to explore their trainee views of where and how learning can be facilitated appropriately.

From our analysis of the workplace so far we have established design principles for workplace learning support. We suggest that any technology or other types of support need to take into consideration the types of activity taking place in the working day, the timeframe in which support is needed or in which the learner is able to use the technology and the particular affordances of the technologies. A major finding from the study so far has been that the acquisition of expertise is facilitated through the sharing of tacit knowledge and expertise, within a community of practitioners. We tentatively propose one useful technology might be some sort of social networking platform.

This could facilitate the support of groups of peers and experts in a shared learning practice and exposure of expert knowledge within a wider community of practitioners. We bear in mind that avenues other than technology might also appropriately support learning. Modified work practice, which facilitates the sharing of expertise on a broader level, might be a potential solution. This might, for example be through changing of dialogue between practitioners or alternative arrangement of meetings within a team.

In summary, the findings from his study will provide a useful representation of how expertise is acquired in the workplace and the curriculum as it is experienced by the trainee. The approach used will provide requirements specification to inform the development of technology to facilitate workplace learning. The study contributes to technology advances by basing new potential innovations on sound empirical evidence.





We thank Arthritis Research Campaign for funding this study and all the participant volunteers


Brouns, F., Berlanga, A., Bitter-Rijpkema, M., Sloep, P., Van Rosmalen, P., Kester,

L., Fetter, S., Nadeem, D. and Koper, R. (2008) Personal profiles: Facilitating participation

in Learning Networks. Available at, 1.TENC: Publications and


Childs, S.B.E. and Hall, A. (2005) Effective e-learning for health professionals and

studentsbarriers and their solutions. A systematic review of the literature — findings from the HeXL project. Health Info Libr J 22 (Suppl 2): 20–32

Collison, C. and Parcell, G. (2001) Learning to Fly. Practical Lessons from the world’s leading

knowledge companies. Capstone Publishing Mildford USA

Dexter, H. (2007) Workflows Activity & Implementation. Managing Requirements through

to Service Specification. Available at eframework/transition_models_working_systems_hilary_dexter.pdf

Dreyfus, H. and Dreyfus, S. (1986) Mind over Machine: The power of human intuition and

expertise in the era of the computer. Oxford; Basil Blackwell

Eraut, M. (2004) Informal learning in the workplace. Studies in Continuing Education 26 (2):


Franklin, T. (2008) Web 2.0 for Content for Learning and Teaching in Higher Education.

Available at

Fuller, A., Hodkinson, H., Hodkinson, P. and Unwin, L. (2005) Learning as peripheral

participation in communities of practice: a reassessment of key concepts in workplace learning.

British Educational Research Journal 31 (1): 49–68

Greenhalgh, T. (2001) Computer assisted learning in undergraduate medical education. BMJ.


Hager, P. and Halliday, J. (2006) Lifelong, informal and workplace learning. In Recovering informal learning, 15–45. Springer Netherlands

Kennedy, T.J., Regehr, G., Baker, G.R. and Lingard, L.A. (2005) Progressive independence in

clinical training: a tradition worth defending? Acad. Med. 80 (10 Suppl): S106-S111

Knight, L.V. and Sweeney, K. (2007) Revealing implicit understanding through enthymemes: a

rhetorical method for the analysis of talk. Med. Educ. 41 (3): 226–233

Kolb, D.A. (1984) Experiential Learning. Englewood Cliffs Prentice Hall, NJ.

Lave, J. and Wenger, E. (1991) Situated Learning: Legitimate peripheral participation.

Cambridge: Cambridge University Press

Schmidt, H.G., Norman, G.R. and Boshuizen, H.P. (1990) A cognitive perspective on medical

expertise: theory and implication. Acad. Med. 65 (10): 611–621

Schön, D. (1983) The Reflective Practitioner. New York: Basic Books.

Veasey, S., Rosen, R., Barzansky, B., Rosen, I. and Owens, J. (2002) Sleep loss and fatigue

in residency training: a reappraisal. JAMA: The Journal of the American Medical Association

288 (9): 1116–1124

Walter, F., Battison, S. and Schweitzer, S. (2008) A model of a trust-based recommendation

system on a social network. Autonomous Agents and Multi-Agent Systems 16 (1).

Wenger, E., McDermott, R. and Snyder, W. (2002) Cultivating Communities of Practice. A guide to manage knowledge. Harvard Business School Press

Woo, M., Clayton, R., Johnson, B. and Ellis, C. (2004) Dynamic Knowledge Map:

reusing experts’ tacit knowledge in the AEC industry. Automation in Construction design in

Architecture Volume 13, Issue 2, March 2004, Pages 203–207 (2): 203–207


Index of Authors

Author Paper number Page

Aguiar, Cristina . . . . . . . . . . . . . . . . . . . . . . . 0161 . . . . . . . . . . . . . . . . . . . .132

Axe, Jo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0123 . . . . . . . . . . . . . . . . . . . .104

Bates, Simon . . . . . . . . . . . . . . . . . . . . . . . . . . 0114 . . . . . . . . . . . . . . . . . . . . . . 1

Carvalho, Ana . . . . . . . . . . . . . . . . . . . . . . . . 0161 . . . . . . . . . . . . . . . . . . . .132

Cornelius, Sarah . . . . . . . . . . . . . . . . . . . . . . . 0055 . . . . . . . . . . . . . . . . . . . . . 31

Cross, Simon . . . . . . . . . . . . . . . . . . . . . . . . . 0299 . . . . . . . . . . . . . . . . . . . . . 10

Dexter, Hilary . . . . . . . . . . . . . . . . . . . . . . . . . 0102 . . . . . . . . . . . . . . . . . . . .141

Dickinson, John . . . . . . . . . . . . . . . . . . . . . . . 0031 . . . . . . . . . . . . . . . . . . . . . 51

Dornan, Tim . . . . . . . . . . . . . . . . . . . . . . . . . . 0102 . . . . . . . . . . . . . . . . . . . .141

Dunne, Liz . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0207 . . . . . . . . . . . . . . . . . . . . . 61

Edwards, Miranda. . . . . . . . . . . . . . . . . . . . . . 0084 . . . . . . . . . . . . . . . . . . . . . 40

Galpin, Fenella . . . . . . . . . . . . . . . . . . . . . . . . 0213 . . . . . . . . . . . . . . . . . . . .116

Gordon, Carole . . . . . . . . . . . . . . . . . . . . . . . . 0055 . . . . . . . . . . . . . . . . . . . . . 31

Hall, Graham . . . . . . . . . . . . . . . . . . . . . . . . . 0084 . . . . . . . . . . . . . . . . . . . . . 40

Hardy, Judy . . . . . . . . . . . . . . . . . . . . . . . . . . 0114 . . . . . . . . . . . . . . . . . . . . . . 1

Harris, Margaret . . . . . . . . . . . . . . . . . . . . . . 0055 . . . . . . . . . . . . . . . . . . . . . 31

Haywood, Jeff . . . . . . . . . . . . . . . . . . . . . . . . 0114 . . . . . . . . . . . . . . . . . . . . . . 1

Haywood, Denise . . . . . . . . . . . . . . . . . . . . . . 0114 . . . . . . . . . . . . . . . . . . . . . . 1

Johnson, Mark . . . . . . . . . . . . . . . . . . . . . . . . 0084 . . . . . . . . . . . . . . . . . . . . . 40

Jones, Chris . . . . . . . . . . . . . . . . . . . . . . . . . . 0299 . . . . . . . . . . . . . . . . . . . . . 10

Kay, David . . . . . . . . . . . . . . . . . . . . . . . . . . . 0291 . . . . . . . . . . . . . . . . . . . . . 82

Keat Khoo, Thean . . . . . . . . . . . . . . . . . . . . . 0315 . . . . . . . . . . . . . . . . . . . .124

Lager, Peter . . . . . . . . . . . . . . . . . . . . . . . . . . 0084 . . . . . . . . . . . . . . . . . . . . . 40

Maciel, Romana . . . . . . . . . . . . . . . . . . . . . . . 0161 . . . . . . . . . . . . . . . . . . . .132

Macleod, Hamish . . . . . . . . . . . . . . . . . . . . . . 0114 . . . . . . . . . . . . . . . . . . . . . . 1

March, Donna . . . . . . . . . . . . . . . . . . . . . . . . 0123 . . . . . . . . . . . . . . . . . . . .104

McGonigle, Bob . . . . . . . . . . . . . . . . . . . . . . . 0291 . . . . . . . . . . . . . . . . . . . . . 82

Newcombe, Matthew . . . . . . . . . . . . . . . . . . 0207 . . . . . . . . . . . . . . . . . . . . . 61

Paterson, Jessie . . . . . . . . . . . . . . . . . . . . . . . 0114 . . . . . . . . . . . . . . . . . . . . . . 1

Patterson, Walter . . . . . . . . . . . . . . . . . . . . . . 0291 . . . . . . . . . . . . . . . . . . . . . 82

Paull, Alan . . . . . . . . . . . . . . . . . . . . . . . . . . . 0081 . . . . . . . . . . . . . . . . . . . .111

Pollard, Bill . . . . . . . . . . . . . . . . . . . . . . . . . . . 0084 . . . . . . . . . . . . . . . . . . . . . 40

Prinsloo, Paul . . . . . . . . . . . . . . . . . . . . . . . . . 0213 . . . . . . . . . . . . . . . . . . . .116

Rhind, Susan . . . . . . . . . . . . . . . . . . . . . . . . . . 0114 . . . . . . . . . . . . . . . . . . . . . . 1

Rodway-Dyer, Sue . . . . . . . . . . . . . . . . . . . . . 0207 . . . . . . . . . . . . . . . . . . . . . 61

Romenska, Sandra . . . . . . . . . . . . . . . . . . . . . 0164 . . . . . . . . . . . . . . . . . . . . . 97

Russell, Carol . . . . . . . . . . . . . . . . . . . . . . . . . 0178 . . . . . . . . . . . . . . . . . . . . . 21

Shah, Preeti . . . . . . . . . . . . . . . . . . . . . . . . . . 0102 . . . . . . . . . . . . . . . . . . . .141

Slade, Sharon . . . . . . . . . . . . . . . . . . . . . . . . . 0213 . . . . . . . . . . . . . . . . . . . .116

Snowden, Neil . . . . . . . . . . . . . . . . . . . . . . . . 0102 . . . . . . . . . . . . . . . . . . . .141

Sutherland, Shane . . . . . . . . . . . . . . . . . . . . . 0081 . . . . . . . . . . . . . . . . . . . .111

Tabbiner, Barbara . . . . . . . . . . . . . . . . . . . . . 0291 . . . . . . . . . . . . . . . . . . . . . 82

Traxler, John . . . . . . . . . . . . . . . . . . . . . . . . . 0288 . . . . . . . . . . . . . . . . . . . . . 70

Ward, Rupert . . . . . . . . . . . . . . . . . . . . . . . . . 0084 . . . . . . . . . . . . . . . . . . . . . 40

Whitfield, Pete . . . . . . . . . . . . . . . . . . . . . . . . 0084 . . . . . . . . . . . . . . . . . . . . . 40

Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project