Magic Wako - User Interaction in a Projector-based

CONTENT 2011 : The Third International Conference on Creative Content Technologies
Magic Wako - User Interaction in a Projector-based Augmented Reality Game
Daniel Franzen, Ignacio Avellino, Fabio Mauri
Marc Jentsch, Andreas Zimmermann
RWTH Aachen University
Fraunhofer FIT
Aachen, Germany
Sankt Augustin, Germany
{daniel.franzen, ignacio.avellino, fabio.mauri} {marc.jentsch, andreas.zimmermann}
Abstract—Augmented reality games offer a new level of
player immersion into a game world. Upcoming pico projectors,
which, are integrated into mobile phones, provide a way to
make augmented reality games commonly available. This technologies aided by techniques such as image processing can serve
as a way to creatively enhance existing games. Unless there
is first work done on how projector based augmented reality
applications in general can be controlled by users, investigation
on augmented reality games control is little available. We
present a projector based augmented reality game which, can
be controlled by either real world or virtual world interaction
methods. In two qualitative user explorations, one using a lowlevel prototype and one using a first implementation of the
game, we identify user-related and technological challenges
regarding interaction with the augmented reality game.
Keywords-Projector phone; interaction metaphor; augmented
reality; mixed reality game.
Embedding pico projectors in mobile phones provides a
complete new way to display information and new interaction techniques. The first integrated projector phones are
now available. Such devices overcome the inherent display
limitations of mobile phones, since larger displays can now
be dynamically created on rather any surface. Projector
phones also open up the range of possible interactions, since
the user can interact with the mobile phone while looking
at a projection or even using a combination of the mobile
phone’s screen and the projection in parallel. For users, this
methodology eliminates the necessity to switch the focus of
attention between the real object and the augmented video
version on the mobile device’s screen, which was inherent
to the See-Through-Approach [1] for creating augmented
reality applications.
Recently, using the physical world as playing field is
becoming popular since this increases user immersion into
the game world. Examples are dance mats or Kinect [2].
But, in these cases, the actual game world is still behind the
screen. Hence, using augmented reality to play games in the
real world can further increase gamer immersion.
This paper researches the interaction of users with augmented reality games focussing on a congruent setup of
the mobile phone’s camera and projector, were the field of
view and the field of projection overlap [3]. This particular
spatial configuration enables the user to interact directly
Copyright (c) IARIA, 2011.
ISBN: 978-1-61208-157-1
with the projection without any limitations, and is open for
the different interaction concepts described by Rukzio and
Holleis [4]. However, the congruent setup also introduces
problems, since the projected image can have an immediate
effect on the processing of the captured camera image.
Furthermore, the physical process of pressing a button on
the device causes that it is shaken. This affects other input
modalities which, are considering position and orientation.
This paper elaborates on how users perceive the interrelation of output and input channel and how they utilize this
interrelation for their interaction in a game. For being able to
observe players’ behavior we created an augmented reality
game called Magic Wako. This game offers an easy to learn
gameplay associated with the necessity for fast, direct and
gesture-rich interactions with a mobile projector.
The remainder of this paper is organized as follows. After
discussing related work in the succeeding section, we present
findings of our first low-level prototype. After that, we
describe experiences from the first system implementation
and insights from further user tests. Finally, we conclude
and give an outlook on future work.
The Wear UR World prototype [5] shows many everyday
life examples how a portable projector can be used to
augment everyday objects with additional information. As
input modality, a wearable camera captures four-fingergestures. For easier recognition, the fingers are equipped
with colored markers. Baldauf and Froehlich [6] use the
same gesture recognition approach but process the image
on the mobile phone to make the setup more portable.
There have already been some games implemented which,
use mobile projectors to interact with the real world. Pinhanez et al. [7] use the Everywhere Display Projector to
let people build puzzle pictures with colored sweets. The
projection shows gamers where to put a sweet on the ground.
So, in this case the projector is controlling the user. In CoGAME, gamers project paths onto any ground were a realworld robot toy is being guided towards a defined destination
[8]. The robot is equipped with infrared LEDs that are
tracked by a camera and steered by a server component.
So, gamers control the game by moving themselves and the
projector in their hands. Flashlight jigsaw is a multiplayer
CONTENT 2011 : The Third International Conference on Creative Content Technologies
puzzle game made for handheld projectors [9]. Players have
to search on a public display for pieces of the puzzle by
exploratively pointing at the screen with handheld controllers. Interaction metaphors are similar to Magic Wako
without considering real-world objects. In LittleProjectedPlanet, Loechtefeld et al. [10] use paintings on real world
walls to let projected balls run through it. In this case, the
individual manipulation of the real world creates the game
world. At the same time, the information where the projector
is pointing at is steering the game.
The presented game approaches do not deeply investigate
user acceptance of the new input modality. However, Kawsar
et al. [11] let users compare the See-Through approach
against mobile projection in three different non-gaming
applications. Afterwards, they conduct a qualitative study
concerning user acceptance and usability issues of the interaction techniques. While preferring the larger displays of the
projection approach, users figure out a higher degree of cognitive load due to more demanding hand-eye coordination.
Blasko et al. [12] identify the stabilization of the projected
image as the major challenge in the interaction with their
wrist-worn projection display. However, we expect to find
out additional challenges of projector phone interaction
in the gaming context, as this requires faster and more
spontaneous movements.
users would intuitively control the game with the new
interaction paradigm and how they feel about it. Secondly,
we wanted to know what problems might be caused by
the interaction using a handheld projector for controlling
an augmented reality game. The low-level prototype does
not reflect the final system real conditions such as light,
since the main focus is on the interaction of the control and
not on the gameplay. This will be analyzed using the final
We presented the users the physical playing field, which,
contained nine different colored circles made of cloth and
explained the gaming rules. All users were familiar with
computer games but never played an augmented reality
game or used a mobile projector before. Users had to use
an electric torch to simulate the projector (cf. Figure 1).
Additionally, they were equipped with a mobile touchscreen
phone, which, presented only one big button labeled ”Hit”.
We simulated the Wakos by randomly putting and removing
a card with an X on the circles of the playing field. Without
further information on how the actual control of the game
should work, for example how to hit the X, we asked users
to play the game. We observed the players and afterwards
interviewed them about their feeling of the game.
We set up a first low-level prototype in order to find out
about user requirements how to control the game.
A. Gameplay
MagicWako explores a new paradigm of gaming that aims
at recreating the popular arcade game Whac-A-Mole [13].
In the original game, little moles (called Wakos) come out
randomly from holes in the game board and disappear again
after a short time. The aim of the game is to hit as many
Wakos as possible. In our augmented reality version of
that game, players have to search for the Wako with a
mobile projector in their hand. The game is played on a
physical playing field on which, virtual Wakos are projected
by the mobile projector. As the field of projection does not
cover the complete playing field, gamers have to explore the
playing field to search for Wakos by moving the projector.
Magic Wako differs from games presented in the previous
work since firstly the projector is controlled by the user and
not the other way around, and secondly only the controller
and the gameboard are required to play the game; there is
no need for additional world objects. Additionally, the game
board can be replaces with any available object due to the
initial color calibration.
B. Setup
We conducted a qualitative exploration with a low-level
prototype and seven users. The aim was to find out how
Copyright (c) IARIA, 2011.
ISBN: 978-1-61208-157-1
Figure 1.
Torch setup for user evaluation
C. Ergonomic Findings
In the interviews, all users claimed the game to be
comfortable and found the movement with the torch natural.
Hence, we assume that the general approach of controlling
the game is intuitive and was appropriate to be implemented
in the first prototype. It seems that searching for objects with
light in the real world is intuitively easy to understand even
for users without experience in augmented reality games.
Three out of seven users pressed the ”Hit” button for
hitting the X. Another three users tried to hit the physical X
with their hand. One person wanted to use the torch itself to
hit the X. The game setup is meant to be a hybrid between
CONTENT 2011 : The Third International Conference on Creative Content Technologies
a virtual and a physical game. According to that, it seems
that about half of the users favored a virtual way of hitting
- pressing a button. The other half of the users intuitively
preferred to act in the physical world, be it with the hand
or the torch itself. Hence, we plan to offer both possibilities
to hit the Wako in the prototype and compare again.
We observed that especially users who tried to hit the X
with their hand experienced problems to keep the torchlight
steady while touching a far away circle with the other
hand. When users leaned towards the playing field to hit
a projected X, they tended to move the torch thus pointing
to another position where the X was not present. Hitting the
X by pressing the button relieved the users from leaning
towards the playing field. Nevertheless, in an integrated
projector phone setup, also the physical process of pressing
the button would make the device shake a bit so that the
current aim of the projection would change in that moment.
Five users pointed out the problem of carrying around
the playing field and asked to be able to play the game on
arbitrary surfaces.
In this section, we present our first prototype of MagicWako. It was used as to find out more user-related and
technical challenges for augmented reality games control.
For this, we conducted another preliminary user study.
TV out port and high processor power for graphic computation. The ADPP-305 is a good trade-off between mobility
(battery-powered), brightness (45 lumens) and size (fits in
one hand). A wireless camera is attached to the projector to
guarantee the congruent setup. The wireless camera provides
colored images of 640x480 pixels. This quality is high
enough in order not to raise problems with color detection,
given regular indoor light conditions. The prototype’s weight
and size allow to operate with one hand.
One of the main problems encountered with this prototype
concerned the connecting cables between the projector and
the smarthpone. Firstly, there were several cables which
confused the users on how to handle the device comfortably.
Some of the users wore the cables around their neck.
Secondly, as the users moved the prototype some of the cable
jacks rotated making the projected image to be unavailable
for a short period of time until the projector detected the
input again. In a high-level prototype were the projector is
integrated into the smartphone this would not be a problem.
Figure 2 shows the system architecture: The wireless
camera collects an image of the game board and sends it
to a server PC. The server computes the position and size
of the Wako and sends this information to the smartphone,
which, creates the final image to be displayed by the mobile
projector using the stored current score and time left, too.
A. Decisions
As mentioned in the previous section, two possibilities to
hit the Wako were taken into account. We implemented a
first version of Magic Wako where the user had to hit the
smartphone’s screen laying in front of her. Providing this
modality, the input process promises to have fewer impact on
the position of the projector. Furthermore it is also possible
to hold the Smartphone in the other hand and do the hitting
with the thumb like it would be done with a button. However,
we will add a module that can determine if a Wako is hit
by a hand or another physical object in future work.
To detect the currently focussed real world object, we
use a color recognition engine which, can determine the
currently targeted circle. To meet the users’ requirement of
playing the game on other surfaces as our game board, a
color calibration phase was added to the game. In calibration
mode, users are asked to point at a color for 3 seconds. This
color is saved as one of the game colors. In this way, players
can initially specify the colors of the holes were the Wakos
appear. This also allows a new range of playing fields, for
example on people’s colored shirts in front of an unicolored
B. Hardware Prototype
Due to the lack of appropriate integrated projector phones
on the market, we connected an Adapt ADPP-305 projector
to a Samsung Galaxy S smartphone. The Galaxy S offers a
Copyright (c) IARIA, 2011.
ISBN: 978-1-61208-157-1
Figure 2.
Game setup
Figure 3 shows a user who is playing with the first
C. Software Concepts
An image recognition module on the server detects the
color of the circle where the user is pointing at by analyzing
the current camera frame. The colors of a certain amount of
pixels inside the shape are averaged. The actual algorithm
was kept as easy as possible to execute it on a smartphone in
CONTENT 2011 : The Third International Conference on Creative Content Technologies
Figure 3.
User plays with first prototype
later implementations. In the calibration phase, the images
are analyzed for 3 seconds and the average value is stored
for usage in the next game.
The server sends the position and size of the currently
aimed spot to the game logic module on the smartphone.
This module keeps track of the state of the game and passes
all needed information to the image projection module,
which, determines if, where and in which size the Wako
shall be displayed. After this, the image is generated and
displayed by the projector.
D. Lessons Learned
We conducted another qualitative exploration with the
first version of Magic Wako. The aim was to confirm the
observations of the low-level prototype testing. Additionally,
we wanted to identify further user-related and technological
challenges of this augmented reality game approach.
The exploration was conducted with six users which, did
not take part in the first user test. They were equipped
with the hardware prototype. After an introduction to the
gameplay, users were asked to perform four tasks and
provide feedback about the interaction: 1) Point at the top
rightmost spot in the game board. 2) Point at all the spots at
least once in the game. 3) Find the Wako during the game
at least twice. 4) Hit the Wako during the game at least
Copyright (c) IARIA, 2011.
ISBN: 978-1-61208-157-1
once. All the tasks were successfully performed by every
user. All users managed to hit the Wako with increasing
frequency towards the end of the game. It confirms that this
way of handling the projector is natural and easy to learn.
Additionally, every player told that the game is interesting
and fun.
Three users asked for a device that is lightweighter and
easier to handle. As pointed out in this section, the hardware
prototype is a tradeoff because sufficient integrated hardware
is currently not available. In an optimal setup, camera
and projector would be integrated in the smartphone. This
would reduce the size, weight and bulkiness of the hardware
components. Also the server would not be necessary because
all the computations could be performed on the smartphone.
With regard to the shaking problem, it would have to
be investigated if it is still a problem in this new setup.
Alternatively a wireless button could be provided so that
the user could hit it with her free hand. Although this time
we explained that the Wako has to be hit by touching the
mobile phone’s screen, two users still tried to hit the Wako
by hand. This confirms the need for a physical interaction
with the game board as an alternative to the one with the
In some cases when the Wako was projected on a circle,
the color of the latter was altered by the projected light and
the color detection did not work properly. This generated
an annoying flickering of the projection. The color recognition algorithm was improved by increasing the similarity
threshold between the detected color and the color stored
during calibration. Projectors with a darker light work better
because their projected image interferes less with the real
world objects. At the same time, if the projection is too dark,
the Wako is not properly visible. Unfortunately, background
subtraction is not possible in this setup. The gameplay
involves the user to navigate the field with the projector,
which, constantly changes the image detected by the camera.
This doesn’t allow to distinguish whether the change in the
detected image is due to the movement of the user or due
to light changes caused by the projector.
Three users had problems to find a Wako because they
were moving the camera and projector too fast thus preventing the image recognition module to work properly.
The color detection process of one camera frame takes a
few 100 ms. When color detection was completed, too fast
users already pointed at another color. This resulted in a oneframe blinking of the Wako. That problem can be solved by
optimizing the color recognition. Also a reduction of data
communication steps can increase performance, for example
by omitting the server between camera and smartphone.
Using augmented reality with a projector as a mean of
game control has potential to intensify current trends in
CONTENT 2011 : The Third International Conference on Creative Content Technologies
gaming. During two qualitative explorations with a lowlevel prototype and a first implemented version of the
augmented reality game Magic Wako, we could identify
user-related and technological challenges. We also observed
that our interaction approach of controlling the game through
projector movements was easy to learn and widely accepted
by users.
Our explorations indicate that both modalities, hitting the
Wako by button press and hitting the Wako by hand or
object, are requested by a significant group of users. There
is slight evidence that hitting the Wakos physically seems
to be more intuitive. We will further investigate the hitting
modality in the next iteration. It will have to be taken
into account the movement of the persons hand holding the
control when they try to hit the wako phisically with their
other hand. In future work, we are going to implement the
possibility to hit the Wako by hand or by using objects in
order to confirm this finding with a high-level prototype.
This would solve the problem of having to push a button
which caused the shaking of the device. Also in the next
iteration it will be investigated what kind of user group
prefers what modality. This input modality will probably
cause bigger usability problems because the physical input
process affects the projector handling a lot. We will try
to solve these problems, for example, by applying image
stabilization methods. The next iteration will include a larger
group of testers since the number used in the previous two
user studies is not sufficient to draw solid conclusions.
The ability to play the game on arbitrary surfaces is a
further matter of investigation. In this context, reachability
or mobility of objects offer new problems and possibilities.
Also, further investigation on this matter will possibly lead to
improvements or alternative implementations on the current
image processing algorithm. These might include a new
approach on distinguishing the colors where the Wakos come
out such as background substraction.
If more sufficient integrated projector phones will be
available in the future, this will meet one of our identified
user requirements. However, it will also change the nature
of a button press compared to our current prototype. By
now, the physical process of touching the phone’s screen
does not influence the position of the projector, but this is
different in an integrated projector phone. We will examine
how strong this limits the user acceptance. A solution could
be to compensate the projector movement recalculating the
position of the projected image.
Further work will concern technological issues. The color
recognition module will be optimized to deal with the
problem of too fast user interaction. Also, the initial color
calibration allows for the game to be played in an arbitrary
surface that contains a variety of nine different colors. Test
will be made in another setup than the one presented with
the physical board with nine dots.
Copyright (c) IARIA, 2011.
ISBN: 978-1-61208-157-1
This research was supported by the European Commission
within the BRIDGE project (project No. 261817).
[1] E. Bier, M. Stone, K. Pier, W. Buxton, and T. DeRose,
“Toolglass and magic lenses: the see-through interface,” in
Proceedings of the 20th annual conference on Computer
graphics and interactive techniques, 1993, p. 80.
[2] Microsoft, “Kinect,,” 2011, available online at http:
//; visited on June 28th 2011.
[3] J. Schoening, M. Loechtefeld, M. Rohs, and A. Krueger,
“Projector Phones: A New Class of Interfaces for Augmented
Reality,” International Journal of Mobile HCI, pp. 1–14,
[4] E. Rukzio and P. Holleis, “Projector Phone Interactions:
Design Space and Survey,” in Workshop on Coupled Display
Visual Interfaces at AVI 2010, 2010, pp. 408–410.
[5] P. Mistry, P. Maes, and L. Chang, “WUW-wear Ur world:
a wearable gestural interface,” in Proceedings of the 27th
international conference extended abstracts on Human factors
in computing systems, 2009, pp. 4111–4116.
[6] M. Baldauf and P. Froehlich, “Supporting Hand Gesture
Manipulation of Projected Content with Mobile Phones,” in
Proceedings of the Workshop on Mobile Interaction with the
Real World (MIRW), 2009.
[7] C. Pinhanez, F. Kjeldsen, A. Levas, G. Pingali, J. Hartman,
M. Podlaseck, V. Kwatra, and P. Chou, “Transforming Surfaces into Touch-Screens,” IBM Research Report - RC22273
(W0112-0216), 2001.
[8] K. Hosoi, V. Dao, A. Mori, and M. Sugimoto, “CoGAME:
manipulation using a handheld projector,” in ACM SIGGRAPH 2007 emerging technologies, 2007.
[9] X. Cao, M. Massimi, and R. Balakrishnan, “Flashlight jigsaw:
an exploratory study of an ad-hoc multi-player game on public
displays,” in Proceedings of the ACM 2008 conference on
Computer supported cooperative work, 2008, pp. 77–86.
[10] M. Loechtefeld, J. Schoening, M. Rohs, and A. Krueger, “LittleProjectedPlanet: An Augmented Reality Game for Camera
Projector Phones,” Proc. of MRIW, 2009.
[11] F. Kawsar, E. Rukzio, and G. Kortuem, “An explorative
comparison of magic lens and personal projection for interacting with smart objects,” in Proceedings of the 12th
international conference on Human computer interaction with
mobile devices and services, 2010, pp. 157–160.
[12] G. Blasko, S. Feiner, and F. Coriand, “Exploring interaction
with a simulated wrist-worn projection display,” in Proceedings of the Ninth IEEE International Symposium on Wearable
Computers, 2005, pp. 2–9.
[13] B. S. Racers, “Whac-A-Mole,” 2011, available online at http:
//; visited on June 28th 2011.