Developing a remote control application for Windows CE

Developing a remote control application
for Windows CE
Bachelor Thesis performed in Computer Engineering
at ABB Robotics
Mälardalen University,
Department of Computer Science and Electronics
by
Olof Kjellgren
Västerås 2007
Developing a remote control application
for Windows CE
Bachelor Thesis performed in Computer Engineering
at ABB Robotics
Mälardalen University
Department of Computer Science and Electronics
Olof Kjellgren
Supervisor: Ralph Sjöberg
Danny Rusén
Frank Lüders
Examiner: Frank Lüders
Västerås May 30, 2007.
Abstract
An ABB robot system consists of a mechanical unit, a drive module and a controller. To let the operator interact with the system a handheld device called a
FlexPendant is used. The FlexPendant has a touch screen, some hard buttons and
a joystick. It runs on Microsoft Windows CE. When developing new software for
the FlexPendant, a lot of testing needs to be done. One part of this is the manual
testing by pushing the touch screen. Recently, a robotized system was developed
where a robot is used to do the manual testing.
The main purpose of this thesis was to develop a remote control tool so the
testing can be done without the need of a human or robot physically pressing the
touch screen. This has been done by improving the existing systems for debugging
and diagnostics of the FlexPendant.
Another request from ABB Robotics was to investigate the possibility to capture
the coordinates when the touch screen is pressed, and highlight the spot in some
way on a remote display tool on a connected computer. This proved to be very
difficult, no solution that is suitable for the FlexPendant was found.
Keywords: GUI testing, Automated Testing, Remote Control, Remote View,
Capturing Coordinates, Robotics
i
ii
Sammanfattning
Ett robotsystem utvecklat på ABB Robotics består av en mekanisk enhet, en drivmodul och en controller. För att användaren ska kunna interagera med systemet
används en speciell handdator kallad FlexPendant. Den har en pekskärm, några
knappar och en joystick. På FlexPendanten körs Windows CE. När ny mjukvara
utvecklas för FlexPendanten måste omfattande testning genomföras. Ett steg i
testningen består i att manuellt trycka på pekskärmen. Nyligen utvecklades ett
mekaniserat testverktyg där en robot används för att utföra den manuella testningen.
Huvudsyftet med det här examensarbetet var att utvecka ett verktyg för fjärrstyrning av FlexPendanten så att testningen kan utföras utan att en människa eller robot
behöver trycka fysiskt på pekskärmen. Detta har genomförts genom att utveckla
befintliga system för debuggning och diagnostik av FlexPendanten.
Ett annat önskemål från ABB Robotics var att undersöka möjligheten att fånga
upp skärmkoordinaterna när man trycker på pekskärmen. Positionen skulle sen
presenteras tillsammans med en avbildning av pekskärmen på en ansluten datorskärm. Detta visade sig vara mycket svårt att genomföra, ingen lösning som
passar FlexPendanten hittades.
Nyckelord: GUI Testning, Automatiserad Testning, Fjärrstyrning, Fånga Skärmkoordinater
iii
iv
Acknowledgment
I would like to thank my supervisors Danny Rusén, Ralph Sjöberg and Frank
Lüders for all help during the realization of this thesis.
I would also like to dedicate a special thank to Henrik Forsberg for sharing his
expertise considering GTPU Services.
v
vi
Contents
1 Introduction
1.1 Background .
1.2 Purpose . . .
1.3 Goal . . . . .
1.4 Delimitation .
1.5 Thesis Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1
1
1
2
2
2
2 Definitions
2.1 ABB Robotics . . . .
2.2 Cell . . . . . . . . . .
2.3 Controller . . . . . . .
2.4 FlexPendant . . . . . .
2.5 FlexPendant Viewer .
2.6 FPCommand . . . . .
2.7 GTPU Services . . . .
2.8 RAPID . . . . . . . .
2.9 Robot / Robot system
2.10 Stylus . . . . . . . . .
2.11 Windows CE . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
3
3
3
3
4
4
4
4
5
5
3 Pre-existing Systems
3.1 User interaction on the FlexPendant . . . . . .
3.1.1 Teach Pendant Application Framework
3.1.2 GTPU Services / FPCommand . . . . .
3.2 Remote view of the FlexPendant’s GUI . . . .
3.2.1 CE Remote Display Control . . . . . . .
3.2.2 FlexPendant Viewer . . . . . . . . . . .
3.3 The Robotized Testing-system . . . . . . . . .
3.3.1 Overview of the Capture & Replay tool
3.3.2 Capture . . . . . . . . . . . . . . . . . .
3.3.3 Replay . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
7
7
7
9
9
9
10
10
10
10
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
vii
viii
Contents
4 Problem Formulation
13
4.1 Remote control of the FlexPendant . . . . . . . . . . . . . . . . . . . 13
4.2 Capture mouse coordinates on the touch screen . . . . . . . . . . . . 13
5 Analysis of Problem
15
5.1 Remote Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5.2 Capture mouse coordinates . . . . . . . . . . . . . . . . . . . . . . . 15
5.2.1 Hooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
6 Introducing Remote Control
17
6.1 Improving the FlexPendant Viewer . . . . . . . . . . . . . . . . . . . 17
6.2 Adding new features to GTPU Services . . . . . . . . . . . . . . . . 17
7 Capturing mouse clicks on the
7.1 System performance . . . . .
7.2 API functions . . . . . . . . .
7.3 Using a mousehook . . . . . .
7.4 Result . . . . . . . . . . . . .
FlexPendant
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
19
19
19
20
21
8 Remote control applied to the testing system
23
8.1 The FlexPendant TestMaker . . . . . . . . . . . . . . . . . . . . . . 23
8.2 The RobotTest application with remote control . . . . . . . . . . . . 24
9 Summary and Conclusions
27
A User’s Guide to the Program
31
Chapter 1
Introduction
An ABB robot system consists of a mechanical unit, a drive module and a controller. To let the operator interact with the system, a hand held device called a
FlexPendant is used (see 2.4). The FlexPendant has a touch screen panel and it
runs on Microsoft Windows CE (see 2.11). The software is developed with .NET
technology.
1.1
Background
Today, testing on the GUI of the FlexPendant is done in two ways. Firstly, class
tests are run, testing the public interface. Secondly, a tester manually tests numerous test cases by pressing the FlexPendant’s touch screen.
Recently, a robotized testing system was developed, where a mechanical unit is
used to perform the manual testing.
The robotized test programs are created by recording actions from users on the
Virtual Controller (see 2.3) (a virtual controller and FlexPendant on a PC). These
actions are captured to make the robot’s actions as human like as possible.
As the tests are running, the system compares the results with an anticipated
result and logs any misbehaviour in the software and continues with the next test
case.
1.2
Purpose
This work’s purpose is to create a testing system where the mechanical unit isn’t
needed. Instead, coordinates representing the pressed position will be sent to the
FlexPendant, and it will act as if the touch screen was physically pressed.
A GUI run on a desktop computer will be created to send the coordinates. Also,
if the FlexPendant’s touch screen is pressed, the GUI will show the position that
was pressed.
1
2
1.3
Introduction
Goal
The goal of this thesis is to improve an existing automated testing system by
creating a remote control application for Windows CE devices.
1.4
Delimitation
This thesis will only look into remote control of the Windows CE device. It will
not consider any theories of testing etc.
1.5
Thesis Guide
Chapter 1 and 2 is to get the reader acquainted with the terminology and theories
throughout this thesis.
Chapter 3 describes the user interaction applications that are running on the FlexPendant and also contains a description of the Robotized Testing System.
Chapter 4 describes the problems to be solved.
In chapter 5, the problems are being analyzed and different approaches are discussed.
Chapter 6 through 8 describes the introduction of remote control and how to capture mouse events on the FlexPendant.
Chapter 9 summarizes this thesis.
Chapter 2
Definitions
2.1
ABB Robotics
ABB Automation Technologies Robotics and Manufacturing.
ABB Robotics is a company that develops industrial robots and related products.
It has about 800 employees and offices in Västerås, Stockholm and Göteborg.
2.2
Cell
The area in which the mechanical unit operates. It is enclosed for safety reasons.
2.3
Controller
The controller is the heart of the robot system (see 2.9). It contains a computer
with several ports, power supply, drive units and an operator panel. It is located
in a cabinet.
2.4
FlexPendant
The FlexPendant is a hand held device for controlling the robots movements. It is
operated by a touch screen, a joystick and a few hard buttons. It operates on the
Windows CE operating system (see figure 2.1).
2.5
FlexPendant Viewer
The FlexPendant Viewer is an application that runs on a desktop or laptop computer. It simply shows the same image as the GUI on the FlexPendant.
3
4
Definitions
Figure 2.1. The FlexPendant.[2]
2.6
FPCommand
FPCommand runs on a desktop or laptop computer that is connected to the FlexPendant by a network cable and makes it possible to send commands and receive
debugging information from GTPU Services (see 2.7).
2.7
GTPU Services
Graphical Teach Pendant Unit Services.
A process that runs on the FlexPendant and deals with diagnostics and debugging.
GTPU Services makes it possible to communicate with the FlexPendant via a
command prompt on a connected computer.
2.8
RAPID
RAPID is a high level programming language for ABB robots. It contains a number
of instructions such as if-statements, loops, and robot-specific instructions such as
Move. The Move-instruction is the most commonly used, and is obviously used to
move the mechanical unit in various modes.
2.9
Robot / Robot system
An ABB Robot system consists of four parts.
1. The mechanical unit, sometimes referred to as the actual ’robot’.
2. The controller, which controls and calculates movements.
3. The drive module, which powers the mechanical unit.
4. The FlexPendant
2.10 Stylus
5
Figure 2.2. Overview of a robot system.
2.10
Stylus
A pen-like thing to be used to interact with the touch screen of a hand held device
instead of using for example a finger.
2.11
Windows CE
Windows CE is a version of Microsoft’s Windows operating system. It is specially
designed for hand-held devices. It is an open, scalable, 32-bit operating system.
[3]
6
Definitions
Chapter 3
Pre-existing Systems
This chapter describes the systems running on the FlexPendant for user interaction
with the robot system. It also describes the robotized testing system developed at
ABB Robotics.
3.1
User interaction on the FlexPendant
There are two main processes that are running on the FlexPendant; TAF and
GTPU Services.
3.1.1
Teach Pendant Application Framework
TAF is short for Teach Pendant Application Framework and is the main application
which the operator uses to interact with the robot system. It’s graphical interface
looks very similar to Microsoft Windows, but with an ABB-button instead of a
Start-button etc. (see figure 3.1)
3.1.2
GTPU Services / FPCommand
The Graphical Teach Pendant Unit Services is a system that was originally designed
to be used to locate bugs during run-time. It is also commonly used for debugging
when developing new features in the robot system. One of the most frequently
used functions is the opportunity to restart the FlexPendant in case the TAF locks
up. The easiest way to do this is to send a command to GTPU Services. A restart
can also be done by making a particular series of joystick moves, or if needed, to
disconnect the cable, and thereby causing the emergency stop to go off. That would
also make the controller to enter emergency stop state which stops the mechanical
unit.
7
8
Pre-existing Systems
Figure 3.1. The TAF user interface.
GTPU Services
The main GTPU Services program is constantly running on the FlexPendant. It can
provide the user with a wide range of data, such as debugging information, system
diagnostics and screen shots from the touch screen. It responds to a number of
commands that can be sent from a command-line-application that runs on a pc.
GTPU Services is constantly collecting run-time information and saving it in a log,
which can be read from at any time.
FPCommand
A laptop or desktop computer is connected to the FlexPendant via a network cable.
On the computer, a command prompt application is run that interacts with the
corresponding side on the FlexPendant. A number of commands can be sent, a few
of them are:
-restart Restart the FlexPendant (not the controller)
-ps Take a screen shot and save it on a desired location.
-d Create a log file on a desired location.
-s Start a specified application
3.2 Remote view of the FlexPendant’s GUI
9
-memshow Display available and used memory.
-fileput Transfer a file from the computer to the FlexPendant.
Communication between GTPU Services and FPCommand
The communication between the FlexPendant and the pc is a simple request-reply
system. GTPU Services is constantly listening on a socket. When a command is
sent from FPCommand, it is received and interpreted by GTPU Services, and then
the action according to the command is performed.
3.2
Remote view of the FlexPendant’s GUI
For educational and demonstrational purposes, it is useful to be able to show the
FlexPendant’s user interface on a laptop or desktop personal computer. From
there, it can be shown on a much larger screen. There are a few applications of
this kind available, but none of them is really suited for this system. For this
reason, the FlexPendant Viewer has been developed at ABB Robotics.
3.2.1
CE Remote Display Control
One of the pre-existing applications of this type is Microsoft’s CE Remote Display
Control application, Cerdisp. Like many other applications of this type, the main
disadvantage that makes it unsuitable is that it needs to run a process on the
server device, in our case the FlexPendant. This is unwanted for several reasons,
for example:
• The memory resources on the FlexPendant is very limited, e.g. there is no
room for applications of this kind.
• The application that has to be run on the server device needs to be set up
each time it shall be run.
A good thing about Cerdisp is that it also can be used to interact with the FlexPendant. When clicking the mouse in the application window, the FlexPendant will
receive the click and act as if its touch screen was pressed on the same position.
3.2.2
FlexPendant Viewer
The FlexPendant Viewer is a remote display tool developed at ABB Robotics. At
a first glance, it looks very similar to Cerdisp. It opens a window on the target
computer, and shows the same image as on the FlexPendant’s touch screen. The
big difference between the two is that the FlexPendant Viewer doesn’t need to
run any extra software on the server device. Instead, it uses the print screen
feature in GTPU Services. The application constantly sends the command ”-ps”
via FPCommand, and receives screen shots from the FlexPendant. If the received
10
Pre-existing Systems
image has changed, FlexPendant Viewer redraws its window. The application can
also be run in full screen mode.
A disadvantage to this model, compared with Cerdisp, is that if the appearance of
the touch screen is rapidly changed, the FlexPendant Viewer will have a hard time
to keep up the pace. Although this might cause the viewing audience to experience
a bit of lagging, it shouldn’t a big problem since it’s up to the operator of the touch
screen to press it slow enough. Unlike Cerdisp, the FlexPendant Viewer can only
be used for viewing and not for remote control.
3.3
The Robotized Testing-system
When developing new features in a robot system, a lot of testing needs to be done.
One part of this is to test the graphical user interface of the FlexPendant. This is
done by a tester, who performs a number of test cases by manually pressing the
touch screen. Recently, a robotized testing system was developed, where a robot
is used to do the manual testing. [4]
3.3.1
Overview of the Capture & Replay tool
The Robot Capture & Replay tool consists of two separate robot systems. One
system with the moving mechanical unit which presses the FlexPendant in the
other system. The system that is being tested doesn’t need any mechanical unit;
it can be a test rack, i.e. a FlexPendant connected to a controller that simulates
the mechanical unit. The tests are recorded by a capture tool and then carried out
by a replay tool. Figure (3.2) shows an overview of the testing system.
3.3.2
Capture
The test cases to be performed needs to be recorded in some way in advance.
This is done by a capture tool. The tool uses the CE Remote Display Control
application described in section (3.2.1). The test maker records the test by clicking
in the application window. The capture tool is configured so that it captures the
coordinates of all clicks that take place within that window. Checkpoints are added
to facilitate insurance that the system is in the right state. When a checkpoint is
ordered, a screen shot of the touch screen is taken and saved on the connected
computer for later comparison.
3.3.3
Replay
The replay tool, called RobotTester, continuously reads data from the created test
script and converts it into RAPID-instructions. Every position to be clicked on
the touch screen generates three move instructions, two to hover above the screen
and one to make the click. After each click has been done, the controller notifies
the testing system which updates the Move-instructions with new positions for the
next click in the script.
3.3 The Robotized Testing-system
11
Figure 3.2. Overview of the Robot Tester.[4]
When getting to a Compare-instruction in the script, the Robot Tester orders a
screen shot from the FlexPendant. The picture is compared with the one taken
during the capture-sequence. If the bitmaps match, everything is ok and the test
continues. If not, the execution will stop after the actual test case and a restart
command will be sent to the FlexPendant to put it in it’s original state. Information
about the error will be saved in the log. Then the test continues with the next test
case.
The user interface has three views; Run, Execution and Settings.
Run Shows a list with test files and buttons to run the tests.
Execution Shows debugging information when running a test.
Settings Has controls to set ip addresses, path to the test files etc.
Figure (3.3) shows the Run-view of the user interface.
12
Pre-existing Systems
Figure 3.3. The Run-view of the Robot Tester.[4]
Chapter 4
Problem Formulation
The background of this thesis is to enhance the robotized testing system. Although
it works, it is not very practical to use a whole robot system while testing the
FlexPendant. Is there any way to test it without the physical interaction? Another
problem occurs when demonstrating the FlexPendant. Is it possible to capture the
coordinates and in some way show exactly where the user presses the touch screen?
4.1
Remote control of the FlexPendant
Instead of physically interact with the touch screen of the FlexPendant, it should
be possible to send the coordinates and make it act as if it was pressed. A problem
lies in how to receive the data and make the click. Since there is no space to
install and run any extra applications on the FlexPendant, this has to be solved
using the already running applications and the predefined Windows CE functions
in some way. To send the coordinates, an application showing the FlexPendant’s
user interface on a laptop or desktop computer should be used, where the user
clicks the mouse where he/she want’s the touch screen to be pressed.
The problem can be summarized as: Is it possible to receive coordinates and
make a virtual mouse click without installing any extra software on the
FlexPendant?
4.2
Capture mouse coordinates on the touch screen
A request from ABB Robotics was to investigate the possibility to capture the
coordinates when the touch screen is pressed, and show them on a GUI on a connected computer. This might come in handy when demonstrating the robot system
by using the FlexPendant. The touch screen doesn’t have any pointer to show the
mouse’s current position, and this will make it difficult for the audience to follow
when the user clicks through menus etc on the FlexPendant. The FlexPendant
Viewer already exists to show the interface on a connected PC, but since the touch
13
14
Problem Formulation
screen doesn’t show in any way where the click takes place, neither does the FlexPendant Viewer. It would be desirable to highlight the position where the click
takes place with a red dot or some other kind of marker. The big problem lies
in how to capture the coordinates. Like the problem described above, no extra
software can be installed and run on the FlexPendant to perform this task.
Summary of the problem: Is it possible to capture the coordinates of mouse
events on the FlexPendant and send them to an application on a connected PC?
Chapter 5
Analysis of Problem
This chapter will analyze different approaches to solve the problems. For implementation of remote control, there is only one solution that is really feasible for
this system. To capture the mouse coordinates on the FlexPendant is a much more
complex problem.
5.1
Remote Control
Since there is no room for extra applications due to disk space limitations on
the FlexPendant, this narrows down the number of possible solutions of remote
control. This is also the background of this problem; if there were room for extra
applications, a program like Cerdisp could be used and this problem wouldn’t exist.
The obvious solution is to use GTPU Services to send the data of each click.
The communication system handles text strings, and new commands can be added
to the code to contain the data necessary to perform the mouse action. When
the coordinates have been received, GTPU Services can call a windows function to
execute the mouse click.
The FlexPendant Viewer shows the same image as the touch screen, and with
a few enhancements, it should be perfectly suited to be used to capture the coordinates to be sent.
This is the solution that will be implemented.
5.2
Capture mouse coordinates
The application that is running on the FlexPendant when the mouse clicks are to
be captured is TAF. The TAF user interface is constructed of numerous controls.
When the touch screen is pressed, a WM LBUTTONDOWN message is handled
by the control in which the click takes place. It’s impossible to know in advance
which control will get the next mouse message, and therefore some kind of global
event-handler needs to be used.
15
16
Analysis of Problem
A solution that would seem applicable to this problem is to use a global mousehook.
5.2.1
Hooks
Using hooks is a way to monitor events by capturing system messages. [1] By
”hooking”, you tell windows to call a specified function each time a specified event
occurs. Hooks can be used locally (in our own process) or remote (in other processes). Remote hooks can be thread-specific or system wide. Thread-specific
hooks are used to trap events in specific threads in processes other than our own.
System wide hooks traps all events of a predefined kind destined for all threads in
all processes in the system. System wide hooks are also called global hooks.
When a hook is created, Windows puts a data structure in the memory, containing information about the hook, and adds it to a linked list of existing hooks.
If it’s a global hook, the system has to inject the code for the hook procedure into
the address space of all other processes. This can only be done if the function is
located in a DLL-file which the whole system has access to. However, there are two
exceptions to this rule; the journal record and the journal playback hooks. These
hook procedures must reside in the thread that installs the hook to work properly. The reason is that they deal with the low-level interception of hardware input
events. The events must be recorded and played back in the order they occur, and
this can not be guaranteed if the code is in a DLL. In that case, the events may
scatter among several other threads and it will be impossible to know the order in
which they occurred. The hook procedure of those two hooks must be in a single
thread only; the thread that installs the hooks.
The use of a global mousehook seems to be a possible solution to capture mouse
clicks on the FlexPendant.
Chapter 6
Introducing Remote Control
The main purpose of this thesis was to improve the robotized testing system so
that the touch screen doesn’t need to be physically pressed during the tests. This
was done by improving the already existing communication systems. A few new
commands to FPCommand/GTPU Services have been introduced, and also some
changes to the FlexPendant Viewer have been made.
6.1
Improving the FlexPendant Viewer
To be able to perform the tests without physically pressing the touch screen, an
application like Cerdisp is needed. But this application is not very suitable for
this system, as described in chapter 3. Instead, the FlexPendant Viewer has been
improved so that it can control the FlexPendant, similarly to Cerdisp.
Previously, the FlexPendant Viewer window was non-clickable; the mouse was disabled when hovering above it. After the enhancements, it is possible to click in the
window, and thereby also affect the FlexPendant. The FlexPendant Viewer application is written in C-Sharp, using .net. New event-handlers were added to the code,
to capture the events MouseDown, MouseUp and MouseMove. MouseMove only
keeps track of the position of the mouse. When MouseDown or MouseUp occurs,
the program uses FPCommand to send the current coordinates to the FlexPendant,
along with a newly introduced command. Also, if the application is running in full
screen mode, the coordinates need to be re-scaled before being sent.
When demonstrating the application in front of an audience, it might be useful to
ensure that everyone see when the mouse is clicked. Therefore, a red dot is drawn
when the left mouse button is down.
6.2
Adding new features to GTPU Services
The new commands that have been introduced to the system is -modo, -moup and
-click. They implement the features of mouse down, mouse up and click (i.e. a
17
18
Introducing Remote Control
Figure 6.1. The FlexPendant Viewer running on a PC. (Note the red dot over ”ABB”
where the left mouse button is currently pushed down.)
mouse down and a mouse up instruction on the same position). The reason why all
of them are needed is that some menus and buttons on the FlexPendant respond
directly to a mouse down event, and some respond to a click. If the stylus is
put down at one place, moved, and then released, the system might act different
depending on which menu or button is being clicked. The command string that is
sent for mouse down looks like this: fpcmd ”-modo -x [xpos] -y [ypos]”, where [xpos]
and [ypos] are the coordinates of the position. When receiving the string, GTPU
Services interprets it and identifies the command and the coordinates. It then calls
a system function to execute the mouse event. The function that is being called is
mouse event(...). It takes a number of flags/parameters, the ones most used are:
event Specifies the event, for example MOUSEEVENTF LEFTDOWN or
MOUSEEVENTF LEFTUP.
position Specifies the position in x and y coordinates.
If the coordinates are out of screen, an error message will be returned to FPCommand.
Chapter 7
Capturing mouse clicks on
the FlexPendant
As discussed in chapter 5.2, it seems possible to use a mousehook to capture the
coordinates on the FlexPendant. However, when looking further into this, a number of problems occurred.
7.1
System performance
A set-back with hooks is that they negatively affect system performance. Hooks
should only be used when really needed, and otherwise turned off. When using
a global hook, all system messages have to go through the filter function, and
therefore the system might slow down noticeably. Since the use of remote control
by the FlexPendant Viewer already is making the system slower, one can only
imagine how the use of hooks would affect the system. Global hooks might also
cause conflicts with other applications that implement the same type of global
hook. These facts make it rather unsuitable to implement global hooks on the
FlexPendant.
7.2
API functions
The API-functions SetWindowsHookEx, UnhookWindowsHookEx and CallNextHookEx
that are needed when using hooks, are placed in the dll-file user32.dll. Unfortunately, this dll is not one that is included in the package placed on the FlexPendant.
Since one of the restrictions is that no extra files can be put on the FlexPendant,
this makes the task difficult to complete. It is possible to write a new dll-file
containing these functions, which would require less disc space than user32.dll.
However, the problem would still persist.
19
20
7.3
Capturing mouse clicks on the FlexPendant
Using a mousehook
This section describes how to install a mousehook on the FlexPendant, if it could
have been done.
To install a hook, the SetWindowsHookEx function is called. It takes a number of
parameters:
int idhook Specifies the type of hook procedure. This parameter can have one of
numerous values; the one to be used in our case is WH MOUSE.
HOOKPROC lpfn A pointer to a hook procedure in a DLL.
HINSTANCE hMod Handle to the DLL containing the hook procedure pointed
to by the lpfn parameter.
DWORD dwThreadId Specifies the identifier of the thread with which the hook
procedure is to be associated. If this value is zero, all running threads are
associated, and it is a global hook.
The return value of the function is a handle to the hook procedure. When the
hook has been installed, every time a mouse event occurs, the hook procedure will
be called. This procedure will then perform what we tell it to do when we capture
the event. In our case, it would contain code to draw a red dot on the spot where
the touch screen has been pressed. The syntax of the hook procedure must always
look like this:
LRESULT CALLBACK HookProc(
int nCode,
WPARAM wParam,
LPARAM lParam
);
The nCode parameter is used to determine which action to perform. The value
depends on the type of the hook; each type has its own characteristic set of hook
codes. The parameters wParam and lParam contains information about the message that was sent.
Hooks are always chained in a linked list, with the most recently installed hook first.
When an event occurs, Windows will call only the first hook in the list. Therefore,
it’s important to use the CallNextHookEx function in case there are other hooks
installed by other applications. If this function isn’t called, no other application
will catch the event.
Uninstalling a hook is done by calling the UnhookWindowsHookEx function,
which only takes one parameter; the handle of the hook.
7.4 Result
7.4
21
Result
As described in sections 7.1 and 7.2, the solution with a mousehook do not meet
the requirements of our system. Therefore, this feature has not been implemented
on the FlexPendant. The FlexPendant TestMaker (see 8.2) uses hooking when
recording tests, but that application runs on a pc with the proper dll’s provided,
and without limitations regarding lagging etc.
No other solution than to use a mousehook to capture the coordinates has been
found, so this problem is left unsolved, open for future work. There might be some
other way to go around this, and to find a solution without hooks. However, this
is to extensive to be done within the time span of this thesis.
22
Capturing mouse clicks on the FlexPendant
Chapter 8
Remote control applied to
the testing system
The robotized testing system uses two applications to record and run the tests.
These applications have now been modified to handle the improved FlexPendant
Viewer and to use remote control.
8.1
The FlexPendant TestMaker
The FlexPendant TestMaker is an application built to create tests and save them
to files. It contains a number of predefined actions, for example openEventLog.
When this action is selected, the program loads the path to a file containing the
proper coordinates to be clicked on the FlexPendant for the event log to be opened.
These predefined test cases are very useful, since the layout of the menus on the
FlexPendant sometimes are changed. For example, if a button is moved, all test
files with instructions to press that button would have to be recreated. When using
a predefined action, only the file containing the data to click that button needs to
be changed. Wait instructions can be put into the test by just choosing the time
and clicking a button. Also restart of the FlexPendant and the ordering of a screen
shot can be put into the file by a single click.
Mouse clicks can also be manually recorded. This is done by clicking in a remote
control application showing the FlexPendant’s touch screen. Previously, this was
done by capturing mouse clicks in the Cerdisp window. Now, when the FlexPendant Viewer manages to produce remote control, the FlexPendant TestMaker is set
to capture all clicks that take place within the FlexPendant Viewer window. The
mouse clicks are captured regardless of where on the screen the window is located
and the coordinates are recalculated to fit the FlexPendant.
Predefined test cases, wait-instructions, restarts, screen shots and manually recorded
mouse clicks can all be put into the test file in any desired order. Figure 8.1 shows
the user interface of the FlexPendant TestMaker.
23
24
Remote control applied to the testing system
Figure 8.1. The user interface of the FlexPendant TestMaker. Top left side contains
buttons to manage the test. Bottom left shows the actions put into the test. Right side
contains the menu with predefined test cases to choose from.
8.2
The RobotTest application with remote control
The RobotTest application was originally designed to send instructions to the mechanical unit performing the clicks in the robotized testing system. Based on this
software, a new application called Remote FlexPendant Tester has been developed.
The underlying software has been modified to use remote control instead of sending
instructions to a robot. The graphical user interface also looks a bit different, since
all features are not longer needed.
When a test is to be run, the user can choose from a list of test created with
the FlexPendant TestMaker. One or several files can be chosen. Previously, every
8.2 The RobotTest application with remote control
25
click instruction in the test file led to three move instructions for the mechanical
unit to perform the action. Now, the application instead uses remote control and
just sends the click to the FlexPendant via GTPU Services. When a predefined
action such as openEventLog is to be performed, the application loads the file
placed on the location provided in the path that is put in the test file. This file is
an ordinary test file that is merged the same way as the already running test file.
The application has windows to display debugging information when the tests are
run, and to list failed tests, if there are any. Figure 8.2 shows the user interface of
the Remote FlexPendant Tester.
Figure 8.2. The Remote FlexPendant Tester user interface.
26
Remote control applied to the testing system
Chapter 9
Summary and Conclusions
The main purpose of this thesis was to develop a tool to remotely control the
FlexPendant in order to perform tests of the touch screen. This has been done
by adding new features to a few of the already existing systems for debugging
and diagnostics, and to link them together. The end product is called Remote
FlexPendant Tester and will replace the previous RobotTester, which uses a robot
to test the touch screen. In comparison, the new system seems to work as well as
the old one, and also has many advantages. The biggest advantage with the new
testing is that no extra robot system needs to be set up. Setting up and calibrating
such a system takes a lot of time, resources and space. With remote testing, only
a test rack is needed, which doesn’t take more space than an ordinary workplace
with a pc.
A feature on the FlexPendant that have shown to cause some problems is the
screen saver. If the screen saver is active when a test is started, the first click will
only deactivate the screen saver, and not affect the system in any other way. This
can be avoided by manually controlling the screen before a test is run, but it would
be preferable to automate that test in some way.
A question that has come up and needs to be answered is how fast the tests
can be run. With the robotized testing, the speed was limited by how fast the
mechanical unit moved. By just sending coordinates, the test could theoretically
be performed at a very high speed, as long as the FlexPendant can keep up the
pace. It would be useful to find the optimal speed in which the coordinates can be
sent.
Both the Robot Tester and the Remote FlexPendant Tester only tests the touch
screen of the FlexPendant. But the device also has hard buttons and a joystick to
control movements etc. A suggestion to future work is to look into how these can
be tested, and if it can be done by the remote testing system.
Unfortunately, the problem regarding capturing mouse events on the FlexPendant wasn’t solved. This was due to the affection of the system performance it
would have caused, and the limitation of memory resources. The solution with a
27
28
Summary and Conclusions
mouse hook wasn’t suitable, but it isn’t impossible that there is another solution
that is applicable.
Bibliography
[1] MSDN
Library:
Hooks.
http://msdn2.microsoft.com/enus/library/ms632589.aspx. Web page. Fetched 2007-03-26.
[2] ABB Imagebank. http://inside.abb.com/imagebank. Web page. Fetched 200702-28.
[3] Microsoft Windows CE Home Page.
http://msdn2.microsoft.com/enus/embedded/aa731407.aspx. Web page. Fetched 2007-02-23.
[4] D. Rusén. Capture & replay testing a touch screen device with an abb robot
system. Master’s thesis, Department of Computer Science and Electronics,
Mälardalen University, Västerås, Sweden, December 2006.
29
30
Summary and Conclusions
Appendix A
User’s Guide to the Program
See ”APPENDIX A - User’s guide to the program - Remote FlexPendant Tester.pdf”
31

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project