Augmenting Reality with Projected Interactive Displays

Augmenting Reality with Projected Interactive Displays
Augmenting Reality with Projected
Interactive Displays
Claudio Pinhanez
IBM T.J. Watson Research Center, P.O. Box 218
Yorktown Heights, N.Y. 10598, USA
Abstract. This paper examines a steerable projection system, the everywhere
displays projector (ED-projector), which transforms surfaces into interactive
displays. In an ED-projector, the display image is directed onto a surface by a
rotating mirror. Oblique projection distortion is removed by a computer-graphics
reverse-distortion process and user interaction (pointing and clicking) is achieved
by detecting hand movements with a video camera. The ED-projector is a generic
input/output device to be used to provide computer access from different locations
of an environment or to overlay interactive graphics on any surface of a space,
providing a simpler, more comfortable, and more social solution for augmented
reality than goggles. We are investigating applications of ED-projectors that
provide computer access in public spaces, facilitate navigation in buildings,
localize resources in a physical space, bring computational resources to different
areas of an environment, and facilitate the reconfiguration of the workplace.
1 Introduction
Most augmented reality systems are based on the use of goggles that create
graphics on a transparent display positioned between the real world and the eyes of
the user. If the graphics need to be aligned to some aspect of the physical reality,
the position of the goggles has to be constantly determined in relation to the
environment, to assure the necessary alignment.
This paper describes a projection system that overcomes this problem by
directly projecting the graphics on the surfaces of the physical environment. To
provide flexibility in the projection we have developed a prototype that deflects the
image projected by an LCD projector using a rotating mirror (see Fig. 1).
Moreover, the system uses a video camera to detect user interaction (such as
pointing and clicking) on the projected image, allowing device-free interaction
with a computer system. With this apparatus, we allow not only access to computer
displays from any surface of a space but also enable the computer to directly label
and “act” upon objects and people in the real world through the projection of
images and light patterns.
Our approach of projecting information has clear advantages over the
goggles normally used in augmented reality systems. First, projection allows the
augmentation of real environments with information and graphics without
requiring the users to wear goggles and without installing multiple displays or
wiring sensors to surfaces. Also, it is not necessary to track the user’s head position
Fig. 1. Using the everywhere displays projector to create displays on different
surfaces in a room.
and attitude since the graphics are overlaid directly on the surfaces in the real
world. And finally, multiple users can easily share the information being projected,
making projection-based augmented reality systems much more adequate for
collaborative or social situations.
2 The Everywhere Displays Projector
The basic everywhere displays projector, or simply ED-projector, is composed of
an LCD projector and a computer-controlled pan/tilt mirror. The projector is
connected to the display output of a computer, which also controls the mirror.
Fig. 2 shows a prototype of an ED-projector built with an off-the-shelf rotating
mirror used in theatrical/disco lighting. In the configuration shown in Fig. 2, the
projector’s light can be directed in any direction within the range of approximately
70 degrees in the vertical and 120 degrees in the horizontal. When positioned in
the upper corner of a room, this prototype is able to project in most part of the four
walls, and almost all the floor.
Brightness and Contrast
Our prototype currently employs a 1200 lumens LCD projector that has proved to
have enough brightness and contrast to project images on the surfaces of a normal
office room with the lights on. Although we have not conducted experiments to
determine the perceived brightness and contrast, in typical home and office
conditions a white pattern projected by our prototype is approximately 10 times
Fig. 2. Prototype of the everywhere displays
brighter than its surroundings. With such a difference in brightness, viewers
perceive the white projected pattern as “white” and any neighboring area receiving
only the ambient light as “black.”
Correcting for Oblique Projection Distortion
As illustrated in Fig. 1, when the projection is not orthogonal to the surface, the
projected image can appear distorted. To correct the distortions caused by oblique
projection and by the shape of the projected surface, the image to be projected
must be inversely distorted prior to projection. In general, this distortion is nonlinear and is computationally expensive. However, we have developed a simple
scheme that uses standard computer graphics hardware (present now in most
computers) to speed up this process.
Our scheme relies on the fact that, geometrically speaking, cameras and
projectors with the same focal length are identical. Therefore, to project an image
obliquely without distortions it is sufficient to simulate the inverse process (i.e.,
viewing with a camera) in a virtual 3D computer graphics world. As show in
Fig. 3, we texture-map the image to be displayed onto a virtual computer graphics
3D surface identical (minus a scale factor) to the real surface. If the position and
attitude of this surface in the 3D virtual space in relation to the 3D virtual camera
virtual surface
texture map
image plane
virtual camera
virtual 3D world
Fig. 3. Using a virtual computer graphics 3D world to correct the distortion caused by
oblique projection by simulating the relationship between the projector and the
projected surface.
is identical (minus a scale factor) to the relation between the real surface and the
projector, and if the virtual camera has identical focal length to the projector, then
the view from the 3D virtual camera corresponds exactly to the “view” of the
projector (if the projector was camera).
In practice, we use a standard computer graphics board to render the virtual
camera’s view of the virtual surface and send the computed view to the projector.
It is easy to see that if the position and attitude of the virtual surface are correct,
the projection of this view compensates the distortion caused by oblique projection
or by the shape of the surface. Of course, an appropriate virtual 3D surface must be
uniquely used, and calibrated, for each surface where images are projected.
So far we have experimented only with projecting on planar surfaces. The
calibration parameters of the virtual 3D surface are determined manually by simply
projecting a special pattern and interactively adjusting the scale, rotation, and
position of the virtual surface in the 3D world, and the “lens angle” of the 3D
virtual camera.
A problem with oblique projection is that it is not possible to put all areas of the
projected image simultaneously in focus. Fortunately, current commercial
projectors have a reasonable depth of focus range, enough to maintain decent focus
conditions in most cases. We have succeeded in projecting on surfaces with more
than 60 degrees of inclination in relation to the projection axis without significant
degradation of focus. However, the problem becomes more severe as the distance
between the projected surface and the projector decreases.
Fig. 4. Interacting with projected display by hand.
3 Making the ED-Projector Interactive
Although a device that simply projects images on multiple surfaces has many
applications, we are particularly interested in making the projected surface
interactive like a touch screen. We are currently investigating the use of a pan/tilt
video camera that is controlled by the computer so it has a complete view of the
projected surface (see Fig. 2). The goal is to have the user interact by moving her
hand over the projected surface, as if the hand was a computer mouse; and by
moving the hand rapidly towards the surface, to generate a “click” event
(see Fig. 4). Techniques similar to the ones described in [1] can be used to detect
the hand’s position and the interaction gestures. It is important to notice that using
vision-based interaction with the projected display allows user interaction without
any contact with physical devices, making the ED-projector a system easily usable
in public spaces or in hazardous environments.
4 Examples of Applications
The ED-projector is a generic input/output device that has been designed for use in
multiple applications. We are just beginning to explore such applications as
illustrated in the examples below. These applications can be basically classified in
two types. The first class of applications deals with typical augmented reality
applications: the ED-projector can be used to point to physical objects, show
connections among them, attach information to objects, and to project dynamic
patterns to indicate movement or change in the real world. The second class of
Fig. 5. Augmented reality application: overlaying the database of the contents of a file
cabinet on the top of the cabinet.
applications corresponds to the creation of computer desktop-like interactive
displays that provide computer access from surfaces of objects, furniture, walls,
floor, etc.
Bringing Information to a Physical Location
Most applications of augmented reality are concerned with the virtual attachment
of information to places and objects in the real world. Among such applications,
we have been using the ED-projector to bring information to a physical location
where the information is used or needed. Fig. 5 shows a situation where a database
application accessing a list of files is projected on the top of the file cabinet that
contains those files. In this situation, the user can search the computer database
using any kind of complex query, obtain and refine answers, and use the results to
find the corresponding files in the cabinet.
ED-projectors are ideal devices to support user navigation through an
environment. For instance, by installing an ED-projector in a corridor it is possible
to project information on any wall, door, or area in the floor. Moreover, if the
intent is to provide directions for visitors, an ED-projector-based system has the
advantage that no device needs to be given or worn by the visitor. Plate 7, Fig. 1
shows an example application where the projector is used to signal the direction of
Fig. 6. Resource localization application: finding and checking-out a digital camera.
an emergency exit on the floor. Notice that the rotating mirror allows one single
ED-projector to switch among different areas and directions as needed.
Tagging and Localization of Resources
In a physical environment where the position of objects or components is known to
the computer, it is possible to use an ED-projector system to visually point to an
object’s position or to tag it with relevant information, without requiring the user
to wear or carry any kind of device. Fig. 6 shows an example where the EDprojector responds a verbal request for the position of an object (a digital camera)
by directly pointing to the location of the object in the room. The system follows
by displaying a checkout list near the object that allows the user to update the
information about the item just by touching the projected checklist. Similarly,
information about the digital camera or instructions for its use could have been
displayed on the same area.
Unlike systems based on goggles, it is possible to have a similar application
in a public space. For example, a ED-projector can be used in a store to provide
information about specific items, help customers to find products, and point to
special promotions. All that is required is a set of white or light-colored surfaces
where the information is to be projected such as walls, floors, or pieces of
Dynamic Configuration of Workplaces
Workplaces are increasingly becoming more versatile. As an example, a personal
office many times is also used for small meetings, and for collaborative work. The
ED-projector can be used as a tool to help the reconfiguration of a workplace for
different functions. Plate 7, Fig. 2 shows an example where a desktop application
is projected on a desk and then moved onto a whiteboard. Unlike the interactive
whiteboard described in [1], the ED-projector can move the application around the
room as needed; for instance, to the top of a desk for detailed reading or writing, to
a whiteboard for group discussion, and to a blank wall if the area of the whiteboard
is temporarily needed for writing or scribbling.
Computer Access in Public Spaces and for Disabled People
The ED-projector can also be used to provide computer and information access in
spaces where traditional displays can be broken or stolen, or create hazardous
conditions, such as in public spaces and areas subject to harsh environmental
conditions. The device also permits an interactive display to be brought to the
proximity of a user without requiring the user to move. In particular, the EDprojector can facilitate the access and use of computers by people with locomotive
disabilities. For instance, it can project an interactive display on the sheet of a
hospital bed without creating the risk of patient contact with any device. The
patient, in this case, can interact with the display by simply using his hand and use
it to search for information, call doctors and nurses, or to obtain access to
5 Discussion
Although there has been a substantial amount of research on interactive projected
surfaces [1, 2, 3], the concept of the everywhere displays projector is unique in its
combination of steerable projection and interactive display and interface. In
particular, unlike tangible object-based interfaces [4], it does not require the
construction or use of special, electronically enhanced objects. Instead, the EDprojector is a way to transform any surface or object, without modifications, into a
generic data I/O device by simply projecting and sensing light.
A major advantage of the ED-projector over computer graphics goggles is
that no user contact with any physical device is required. Moreover, since
information is projected directly over the physical objects, it is not necessary to
track the user’s head position in the environment, or to design mechanisms to cope
with the inevitable delays.
Although we have just started examining and prototyping specific
applications, we believe that this device can not only simplify the use of
augmented reality but also significantly expand its range of applications. We are
currently working on improving the performance of the ED-projector and on the
deployment of applications.
1. Crowley JL, Coutaz J, Berard F. Things that See, Communications of the ACM 2000;
43(3): 54-64.
2. Raskar R, Welch G, Cutts M et al. The Office of the Future: A Unified Approach to
Image-Based Modeling and Spatially Immersive Displays. In Proc. of SIGGRAPH'98,
Orlando, Florida, 1998, pp 179-188.
3. Underkoffler J, Ullmer B, Ishii H. Emancipated Pixels: Real-World Graphics in the
Luminous Room. In Proc. of SIGGRAPH'99, Los Angeles, California. 1999, pp 385392.
4. Ishii H, Ullmer B. Tangible Bits: Towards Seamless Interfaces between People, Bits,
and Atoms. In Proc. of CHI'97, Atlanta, Georgia. 1997, pp 234-241.
Plate 7 Fig. 1. Navigation application of an everywhere displays projector: pointing to the
emergency exit.
Plate 7 Fig. 2. An everywhere displays projector moving a desktop application front the
top of a table to a whiteboard.
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF