Using Cellular Phones to Interact with Virtual Environments Bent Dalgaard Larsen Jakob Andreas Bærentzen Niels Jørgen Christensen Technical University of Denmark∗ Distributed VR promises to change the way we work and collaborate [Singhal and Zyda 1999]. In this sketch we will extend the accessibility of the virtual world originally developed in [Larsen and Eriksen 1998] by introducing the use of the modern cellular phone as a platform for primitive interfaces to VR applications. We believe that our use of a cellular phone has led to the first completely pocketable platform for VR user interfaces. While a cellular phone will offer only primitive access to the virtual environment, we expect that being able to access the VE at all times will spawn new and enhance existing applications of VR. For instance, existing online multi-user games may provide limited access through a cellular phone enabling people who either in transit or just not near a PC to participate through a more limited interface. An important question is “how limited?”. We believe that a “cellular interaction mode” need not be very restrictive as long as careful use is made of the technology available. Unfortunately, existing cellular phones have no hardware support for 3D graphics and the screen is much smaller (Typically 176 × 208 or 208 × 320 pixels). Furthermore, the only input is from a limited number of push buttons. But a cellular phone has a high-speed TCP/IP connection to any server on the Internet using the current GPRS (General Packet Radio Service) protocol (40.2 kb/s) or the upcoming and much faster UMTS (Universal Mobile Telecommunications System) protocol. Currently, the producers of cellular phones have chosen no common operating system standards. However, many of these mobile devices support a subset of Java (J2ME—Java 2 Micro Edition). This contains no floating-point math and the only supported network protocol is HTTP which implies that the only way to obtain duplex data streams on the cellular phone is if it continuously polls the server. Figure 1: Interface on the workstation and the cellular device. We have chosen to construct a LEGOTM building application and our task was to enable people to build LEGOTM structures in collaboration using either a traditional workstation or a cellular phone. The system is based on a client-server model. The server contains the 3D models and maintains connections to clients. In our case, the clients may be either workstations or cellular devices. In the following discussion, we focus on the case where the client is a cellular device. ∗ e-mail:(bdl|jab|njc)@imm.dtu.dk Figure 2: Local modifications on the cellular phone. The lack of support for a 3D graphics API and floating point arithmetic makes it very important to divide work between the server and the cellular device in an intelligent way. In our system, we have chosen to perform high quality rendering on the server which uploads images to the device on demand. Since LEGOTM bricks manipulated interactively cannot be rendered locally in realtime, they are rendered in wire-frame. In the absence of floating point numbers, we have implemented a simple facility for fixed point arithmetic. As the user moves in the environment new images on the server are generated and transferred to the cellular phone. When the user selects a LEGOTM brick by pointing at it, the server sends a subimage (the changed part of the image) to the user without the selected brick, which is then drawn in real-time wire-frame while the user alters its location and orientation. As the user deselects the brick, another subimage is generated on the server and the high quality image on the mobile phone is then update again. The user interface is implemented as follows: • A locator is shown on the display of the phone, which is controlled by the arrow keys. • When the selection key is pressed, the brick under the locator is selected using a ray-box intersection algorithm. • The brick is moved using the arrow keys. The arrow keys are used to move the brick in the ground-plane. The height value is found by moving the brick the smallest possible distance from its previous height value, given the constraint that the brick does not intersect other bricks. Users located in the same VE and connected using a workstation have the same posibilities of manipulating LEGOTM bricks. Modifications made either through the cellular phone or the workstation are instantly reflected on all devices connected to the virtual environment. Although this is still work in progress, our conclusion is that mixing locally rendered images on the cellular phone with server rendered images is an appropriate solution for applications such as ours. This work was supported in part by the STVF project DMM and the Nordunit2 project NETGL. References L ARSEN , B. D., AND E RIKSEN , M. A. 1998. Multi-User Interactive Virtual Environments. Master’s thesis, Technical University of Denmark. S INGHAL , S., AND Z YDA , M. 1999. Networked Virtual Environments. Addison Wesley.