The Classroom of the Future: Enhancing Education through Augmented Reality Jeremy R. Cooperstock Centre for Intelligent Machines McGill University 3480 University Street, Montreal QC, H3A 2A7 Canada email@example.com ABSTRACT Electronic classrooms offer instructors a variety of multimedia presentation tools such as the VCR, document camera, and computer projection, allowing for the display of video clips, transparencies, and computer generated simulations and animations. Unfortunately, even the most elegant user interfaces still frustrate many would-be users. The technology tends to be underutilized because of the cognitive effort and time its use requires. Worse still, it often distracts the instructor from the primary pedagogical task. 1 BACKGROUND Information technology promised to empower us and simplify our lives. In reality, we can all attest to the fact that the opposite is true. Modern presentation technology, for example, has made teaching in today’s classrooms increasingly complex and daunting. Whereas fifty years ago, the only concern a teacher had was running out of chalk, faculty now struggle to perform relatively simple tasks, such as connecting their computer output to the projector, switching the display to show a video tape, and even turning on the lights! Technology’s capacity to improve the teaching and learning experience is evident, but so far, its potential remains largely untapped. A related concern in the pedagogical context is the effort required to exploit the technology for novel applications, for example, distance or on-line education. The desire to provide lecture content to students who are unable to attend the class in person, as well as to those who wish to review the material at a later time, has been a driving force behind the development of videoconferencing and web-based course delivery mechanisms. Although a number of universities now offer courses on-line, the cost involved in creating high-quality content is enormous. Both videoconferencing and simple videotaping of the lectures require the assistance of a camera operator, sound engineer, and editor. For asynchronous delivery, lecture material, including slides, video clips, and overheads, must be digitized, formatted and collated in the correct sequence before being transferred. Adding any material at a later date, for example, the results of follow-up discussion relating to the lecture, is equally complicated. The low-tech solution, which offers the lecture material by videotape alone, still involves considerable effort to produce and suffers further from a lack of modifiability, a single dimension of access (tape position), and a single camera angle. This prevents random accessibility (e.g. skip to the next slide) and view control (e.g. view the instructor and overhead transparency simultaneously at reasonable resolution), thus limiting the value to the students. 2 AUTOMATED CONTROL In response to our frustration with this situation, we have augmented our classroom technology with various sensors and computer processing . The room now activates and configures the appropriate equipment in response to instructor activity without the need for manual control (see Figure 1). For example, when an instructor logs on to the classroom computer, the system infers that a computer-based lecture will be given, automatically turns off the lights, lowers the screen, turns on the projector, and switches the projector to computer input. The simple act of placing an overhead transparency (or other object) on the document viewer causes the slide to be displayed and the room lights adjusted to an appropriate level. Similarly, picking up the electronic whiteboard marker causes a projector "swap." so that the whiteboard surface displays the current slide while the main screen shows the previous slide. Audiovisual sources such as the VCR or laptop computer output are also displayed automatically in response to activation cues (e.g., the play button pressed on the VCR; the laptop connected to a video port). Together, these Laptop Jack HC 12 P/2 DA2 SW/2 drape motors P/2 DA2 SW/2 RS-232 serial IR emitter composite video/audio component (HD15) video screen motor Figure 1. Architecture of our computer-augmented classroom connections. The large black module in the center of the image is the AMX Accent3 controller, which drives various devices under computer control and the HC 12 module is our button-panel unit with microcontroller, pictured separately in Figure 2. The SW/2 units are video switchers, one of them running in an auto-sense mode, such that an active signal on the laptop connection is automatically selected, while the second is driver by computer control. The P/2 DA2 units are video splitters, such that either video signal can be routed to both projectors. mechanisms assume the role of skilled operator, taking responsibility for the low-level control of the technology, thereby freeing the instructor to concentrate on the lecture itself, rather than the user interface. 2.1 Manual Override Along with such automation, the need for a seamless manual override mechanism becomes paramount. For example, if the instructor raises the lights, the technology must respect that preference. Furthermore, the ability to turn the lights on or off must not be dependent upon the automatic controller, as it was before this project began.1 As a default backup, manual controls for each device (lights, projector, VCR, etc.) should be accessible and functional at all times. Such manual controls serve as basic on/off switches as well as output enable/disable buttons. For example, a single toggle button on the VCR would allow the presenter to select whether or 1 This led to disastrous consequences when the controller became unresponsive, as was the case after any power fluctuations. On at least two occasions, the instructor was unable to control any of the room lights, as no manual override mechanism existed! Figure 2. Touch-screen interface involving a hierarchical menu structure not the video clip being played is projected to the class. By observing the use of these manual override mechanisms, the reactive classroom system can adapt to the preferences of individual users and remember these settings for future use by the same individual. At the end of each lecture, the system resets itself to a default configuration. Early interviews with instructors revealed that for most users, manual override functions were only required for the room lights and speaker volume, so these were made a top priority. The confusing multilayered touch-screen menu (Figure 2) was replaced by a simple physical button panel (Figure 3) consisting of six switches for the various banks of lights, another two switches for the projection screen and window blinds, and a volume control knob that adjusts VCR and microphone volume, depending on which is in use. 2.2 Usage Observations However, advanced users wanted greater control over the selection of projector outputs, for example, display the laptop output on the main screen and the primary computer display (current lecture slide) on the side screen. Unfortunately, our limited deployment of toggle buttons does not permit such flexibility. While the previously described (nondefault) configuration is possible, the mechanism by which it is invoked is hardly obvious: picking up the whiteboard marker.2 A second panel is presently being designed to permit manual input selection for the two projectors. 2 This apparently bizarre mechanism is, of course, related to the projector toggle function, needed when the instructor moves from the digital tablet to the electronic whiteboard. Figure 3. The replacement button-panel interface (right) for manual override, providing manual light switches, projector, screen, and drape controls, and a context-sensitive volume dial. Interestingly, the one aspect of the current buttonpanel that failed to achieve improved results over the touch-screen is the volume control. Although the single, context-sensitive control knob is far more accessible than the confusing choice of five independent volume panels (only two of which were actually useful) on the touch-screen, the physical interface of a rotating knob poses problems when used in conjunction with the AMX controller. Since a change in volume is dependent on the response time of the controller, which may be as much as one second when the difference between the current level and the newly selected level is fairly large, users often assume that the controller is not working and start turning the knob faster. The LED directly above the knob, which flashes when the volume is being adjusted, does not, unfortunately, convey sufficient information to the user regarding the state of the system. Once the controller catches up, the user may find that the new volume level is far too low or high. This was not a problem with the touch-screen system since the slow response of the AMX controller could be illustrated graphically by level meters. 3 AUTOMATED LECTURE CAPTURE In addition to automating device control, the classroom is wired to record a digital version of any presentation, including both the audio and video, as well as the instructor's slides and notes written during the lecture. This recording facility is based on Eclass, formerly known as Classroom 2000  a system developed at the Georgia Institute of Technology. Eclass provides a mechanism for the capture, collation, and synchronization of digital ink, written on an electronic whiteboard or tablet, with an audiovisual recording of the class (see Figure 4). Figure 4. A sample eClass lecture capture being viewed through a web browser and RealPlayer. At the end of a class, the recorded version of the lecture is then converted into a set of web pages automatically. Each ink stroke written by the professor is linked to the position in the video when that stroke was generated. Students can review the lecture any time after class, randomly accessing portions of the lecture as desired, either from networked university computers or home computers connected by modem. 4 PRESENTER-TRACKING In order to improve the quality of the video capture, we developed a presenter-tracking algorithm  which follows the instructor's movements, even when in front of a projected video screen, thereby obviating the need for a professional cameraman. Device activity, for example, the instructor’s use of a pen on the electronic whiteboard, provides additional tracking cues to the camera. Initialization, activation, and recovery from tracking errors are all handled automatically by computer augmentation, allowing the instructor to remain oblivious to the fact that a recording is being made. The only requirement is that the instructor enters a userid and password to confirm that the lecture should be recorded. 5 FUTURE DIRECTIONS AND CHALLENGES While interesting in its current implementation, our augmented reality approach to the classroom holds even greater potential when integrated with videoconference technology, in which some, or all of the students are in a remote location from the instructor. While current videoconference technology has proven to be grossly inadequate for the social demands of effective classroom teaching, we believe that augmented technology may play a role in overcoming this shortcoming. In our envisioned "classroom of the future" scenario, the teaching technology would respond to events in both locations so as to enhance the interaction between instructor and students. For example, a student’s hand raised might generate a spatialized background audio cue to draw the instructor’s attention toward that student, while a pointing gesture by the instructor toward a remote student could bring about a zoom-in on that student. Figure 5 illustrates a temporal-difference image processing algorithm, used for extraction of the direction of such a pointing gesture. This information, when correlated with the current display, can be used to determine where the remote camera should zoom. A key aspect to allowing for an engaging remote lecture is the use of high-fidelity and low-latency communication of audio and video information, complemented by multichannel, spatialized audio , allowing the instructor to capitalize on audio localization abilities for effective interaction with the students. Echo-cancellation, a constant source of headaches for the videoconference technician, becomes an even greater challenge in this context. ACKNOWLEDGEMENTS The author would like to thank Aoxiang Xu, Shawn Arseneau, Stephane Doutriaux, and Christian Côté, all of whom contributed valuable components to the technology described in this paper. Special thanks are due to Gregory Abowd and Jason Brotherton of Georgia Tech for their assistance with eClass and allowing us to benefit from their research. Support has come from the Natural Sciences and Engineering Research Council of Canada, Fonds pour la Formation de Chercheurs et l’Aide a la Recherche (FCAR), Petro-Canada, Canarie Inc., and the Canadian Foundation for Innovation. This support is gratefully acknowledged. Figure 5. Extraction of a pointing gesture using a temporal difference image processing algorithm. As a preliminary effort in extending the use of augmented reality to support such interaction, we are developing the infrastructure of a Shared Reality Environment, to provide physically distributed users the sensory experience of being in the same space (see Figure 6). Figure 6. Video insertion of a remote participant in the Shared Reality Environment. Note that the segmentation algorithm used in this image was unrefined and thus, results in a number of video errors. REFERENCES 1. Abowd, G., Atkeson, C., Brotherton, J., Enqvist, T., Gulley, P., and Lemon, J. (1998). Investigating the capture, integration and access problem of ubiquitous computing in an educational setting. In Proceedings of Human Factors in Computing Systems CHI '98. ACM Press, New York, pp. 440-447. 2. Arseneau, S. and Cooperstock, J.R. (1999). Presenter Tracking in a Classroom Environment. IEEE Industrial Electronics Conference (IECON’99), Session on Cooperative Environments, Vol. 1, pp. 145-148. 3. Cooperstock, J.R., Tanikoshi, K., Beirne, G., Narine, T., and Buxton, W. (1995). Evolution of a Reactive Environment. Proc. Human Factors in Computing Systems CHI '95, (May 7-11, Denver). ACM Press, New York, pp. 170-177. 4. Xu, A., Woszczyk, W., Settel, Z., Pennycook, B., Rowe, R., Galanter, P., Bary, J., Martin, G., Corey, J., and Cooperstock, J.R. (2000) "RealTime Streaming of Multichannel Audio Data over Internet." Journal of the Audio Engineering Society, July-August.
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project